A Systematic Review on the Use of Artificial Intelligence Techniques in the Diagnosis of COVID-19 from Chest X-Ray Images

level. Higher layers or deep layers enhance the information parts which are significant for segregation and smother unimportant attributes. Because of the unlimited mined parameters in this process, several methods, such as pooling, are used for achieving dimension reduction. Utilizing the AI in diagnosis from the x-ray images can encourage the experts to use these images for the detection of the Coronavirus (15). Several studies have been done on diagnostic of COVID-19 using the radiological images, based on AI and deep neural networks. The assessment of the use of AI in the diagnostic imaging of patients with COVID-19 could be a test case for the type of AI selected in the early stages of future emerging diseases, and also for providing information on the size of the dataset to be used and the appropriate use of AI. In this paper, we reviewed the diagnostic performance of the recently published AI on chest x-ray imaging of COVID-19. Materials and Methods The databases were searched up to July 21, 2020, to extract articles on chest x-ray imaging and AI used for COVID-19. The databases were extracted from PubMed, arXiv, and Google Scholar. The search terms “COVID-19”, “SARSCoV-2”, “Artificial intelligence”, “deep learning”, and “Chest X-ray image”. We looked for articles that involved studies of AI for chest x-ray images and extracted the number of datasets and results from these studies. The datasets include train and test sets. The results contained area under the curve (AUC), sensitivity, accuracy, and specificity. We introduced AI-based systems for detecting COVID-19 used in each article and also investigated the publication of the datasets and the source codes. Results All the extracted articles can be found on Google Scholar. Duplications were removed, and only papers related to chest x-ray imaging were extracted. A total of 20 papers were extracted from Google Scholar, 14 from arXiv, and 5 from PubMed. Nine papers were about creating a novel AI software and the accuracy of AI for predicting COVID-19, eight papers reported on the modification of existing AI software to COVID-19, and 3 papers compared the existing AI software programs for predicting COVID-19. One AI software program predicted the severity of COVID-19; one AI software program measured uncertainty and interpretability for COVID-19 detection, and one paper used the existing AI software program for the classification of pulmonary diseases (Table 1). One article was only on Google Scholar. Khan et al (16) proposed a novel COVID-RENet model to classify COVID-19 patients automatically, using chest x-ray images. COVID-RENet architecture exploits regionbased classification and concept of edge in a deep CNN. To improve the performance of the model, they applied custom VGG-16 nets to the model architecture and finally used SVM classifier for the discrimination of non-COVID patients from COVID-19 ones. Images for this study were obtained from publicly available dataset. The used dataset contained 150 COVID-19 and 150 non-COVID images. Data augmentation was applied to increase data points. Non-COVID-19 class in this study contained abnormal cases, normal ones, and other disease types. They also exploited pre-trained systems and transfer learning approach. This technique was evaluated using 5-fold crossvalidation. The proposed technique achieved an accuracy of 98.3%, a sensitivity of 96.67%, a specificity of 100%, and an AUC value of 0.98. The dataset and proposed code are disclosed and publicly available. The article by Khan et al (16) was published on April 14, 2020. Studies Published on PubMed Five papers were extracted from PubMed. The earliest publication belonged to Apostolopoulos et al (8) which was published on April 3, 2020. They examined a deep learning program using data from 224 published COVID-19 cases, 700 pneumonia cases, and 504 normal cases. The network and database are available for validation and refinement worldwide. They compared known CNNs (Inception ResNet, MobileNet, Xception, Inception, and VGG19) that used transfer learning, evaluated by 10-fold cross-validation, and found that Mobile Net showed the best results, obtaining a sensitivity and specificity of 99.1% and 97.1%, respectively in the detection of COVID-19. Ucar and Korkmaz (17) introduced a novel deep learning model, COVIDiagnosis-Net, based on deep SqueezeNet with Bayes optimization for COVID-19 detection. Their network achieved a specificity of 99.1%, for the COVID class, with an overall accuracy of 98.3%. The sensitivity has not been calculated. The network was trained by 66 images from patients with COVID-19, 3895 images from patients with pneumonia, and 1349 images of normal cases and tested by 10 images from patients with COVID-19, 395 with pneumonia, and 234 images of normal cases. The source code was not disclosed, but the dataset was a public dataset. The paper by Ucar and Korkmaz (17) was accepted on April 21, 2020. Ozturk et al (18) presented a new deep learning model (DarkCovidNet) for diagnosis of Coronavirus disease using chest x-ray images. The model was developed for accurate binary classification (normal vs. COVID) and multi-class classification (COVID-19 vs pneumonia vs normal). The proposed model was evaluated by 5-fold cross validation with an average accuracy of 98.08%, an average specificity of 95.30%, and a sensitivity of 95.13% for detecting COVID-19. The DarkNet (19) model was used as a classifier for the you only look once (YOLO) real time object detection algorithm (19). The source code was disclosed. Chest X-ray images were obtained from two different available datasets. They used 127 x-ray images of

abstract level. Higher layers or deep layers enhance the information parts which are significant for segregation and smother unimportant attributes. Because of the unlimited mined parameters in this process, several methods, such as pooling, are used for achieving dimension reduction. Utilizing the AI in diagnosis from the x-ray images can encourage the experts to use these images for the detection of the Coronavirus (15).
Several studies have been done on diagnostic of COVID-19 using the radiological images, based on AI and deep neural networks. The assessment of the use of AI in the diagnostic imaging of patients with COVID-19 could be a test case for the type of AI selected in the early stages of future emerging diseases, and also for providing information on the size of the dataset to be used and the appropriate use of AI. In this paper, we reviewed the diagnostic performance of the recently published AI on chest x-ray imaging of COVID-19.

Materials and Methods
The databases were searched up to July 21, 2020, to extract articles on chest x-ray imaging and AI used for COVID-19. The databases were extracted from PubMed, arXiv, and Google Scholar. The search terms "COVID-19", "SARS-CoV-2", "Artificial intelligence", "deep learning", and "Chest X-ray image". We looked for articles that involved studies of AI for chest x-ray images and extracted the number of datasets and results from these studies. The datasets include train and test sets. The results contained area under the curve (AUC), sensitivity, accuracy, and specificity. We introduced AI-based systems for detecting COVID-19 used in each article and also investigated the publication of the datasets and the source codes.

Results
All the extracted articles can be found on Google Scholar. Duplications were removed, and only papers related to chest x-ray imaging were extracted. A total of 20 papers were extracted from Google Scholar, 14 from arXiv, and 5 from PubMed. Nine papers were about creating a novel AI software and the accuracy of AI for predicting COVID-19, eight papers reported on the modification of existing AI software to COVID-19, and 3 papers compared the existing AI software programs for predicting COVID-19. One AI software program predicted the severity of COVID-19; one AI software program measured uncertainty and interpretability for COVID-19 detection, and one paper used the existing AI software program for the classification of pulmonary diseases (Table 1).
One article was only on Google Scholar. Khan et al (16) proposed a novel COVID-RENet model to classify COVID-19 patients automatically, using chest x-ray images. COVID-RENet architecture exploits regionbased classification and concept of edge in a deep CNN. To improve the performance of the model, they applied custom VGG-16 nets to the model architecture and finally used SVM classifier for the discrimination of non-COVID patients from COVID-19 ones. Images for this study were obtained from publicly available dataset. The used dataset contained 150 COVID-19 and 150 non-COVID images. Data augmentation was applied to increase data points. Non-COVID-19 class in this study contained abnormal cases, normal ones, and other disease types. They also exploited pre-trained systems and transfer learning approach. This technique was evaluated using 5-fold crossvalidation. The proposed technique achieved an accuracy of 98.3%, a sensitivity of 96.67%, a specificity of 100%, and an AUC value of 0.98. The dataset and proposed code are disclosed and publicly available. The article by Khan et al (16) was published on April 14, 2020.

Studies Published on PubMed
Five papers were extracted from PubMed. The earliest publication belonged to Apostolopoulos et al (8) which was published on April 3, 2020. They examined a deep learning program using data from 224 published COVID-19 cases, 700 pneumonia cases, and 504 normal cases. The network and database are available for validation and refinement worldwide. They compared known CNNs (Inception ResNet, MobileNet, Xception, Inception, and VGG19) that used transfer learning, evaluated by 10-fold cross-validation, and found that Mobile Net showed the best results, obtaining a sensitivity and specificity of 99.1% and 97.1%, respectively in the detection of COVID-19. Ucar and Korkmaz (17) introduced a novel deep learning model, COVIDiagnosis-Net, based on deep SqueezeNet with Bayes optimization for COVID-19 detection. Their network achieved a specificity of 99.1%, for the COVID class, with an overall accuracy of 98.3%. The sensitivity has not been calculated. The network was trained by 66 images from patients with COVID-19, 3895 images from patients with pneumonia, and 1349 images of normal cases and tested by 10 images from patients with COVID-19, 395 with pneumonia, and 234 images of normal cases. The source code was not disclosed, but the dataset was a public dataset. The paper by Ucar and Korkmaz (17) was accepted on April 21, 2020.
Ozturk et al (18) presented a new deep learning model (DarkCovidNet) for diagnosis of Coronavirus disease using chest x-ray images. The model was developed for accurate binary classification (normal vs. COVID) and multi-class classification (COVID-19 vs pneumonia vs normal). The proposed model was evaluated by 5-fold cross validation with an average accuracy of 98.08%, an average specificity of 95.30%, and a sensitivity of 95.13% for detecting COVID-19. The DarkNet (19) model was used as a classifier for the you only look once (YOLO) real time object detection algorithm (19). The source code was disclosed. Chest X-ray images were obtained from two different available datasets. They used 127 x-ray images of  (21) proposed CoroNet, a convolutional neural network model for automatic detection of COVID-19 infection from chest x-rays. Their model was based on Xception architecture pre-trained on ImageNet dataset and trained end-to-end by X-ray images prepared by 2 publicly available datasets. Their program was trained and tested on the prepared dataset and the results for 3-class classification (COVID-19 vs pneumonia vs normal) showed that the specificity and sensitivity rates were 97.5% and 96.9% for COVID-19 cases. They used 4-fold cross-validation and calculated an overall accuracy of 89.6%. The network was trained with data that included 284 COVID-19 images, 657 pneumonia images, and 310 normal images. The network was evaluated by testing it with 157 COVID-19, 500 pneumonia, and 500 normal chest x-ray images. The source code and dataset are available. The paper by Khan et al (21) was published on May 30, 2020.

Studies Published on arXiv
Fourteen papers were extracted from arXiv. Hemdan et al (22) proposed COVIDX-Net framework to classify COVID-19 in chest x-ray images. This framework identifies COVID-19 status in conventional 2D Chest X-ray images. COVIDX-Net architecture was based on seven different architectures of CNN models such as MobileNetV2, DenseNet201, ResNetV2, InceptionV3, Xception, InceptionResNetV2, and VGG19. Among them, they recommended the DenseNet201and VGG19 models because these models achieved the best value of accuracy (90%). The sensitivity of both VGG19 and DenseNet201 models for COVID-19 patients was 100% and the AUC value was 0.9. The public dataset was used in this study. Images were divided into 2 classes: 25 positive COVID-19 and 25 normal images. 80% (40 images) including normal and diseased cases were randomly chosen for network training and 20% (10 images) of them were used for testing phase. They presented a detailed description of the developed framework in their paper. The article was published on March 24, 2020.
Narin et al (7) in their study which was published on March 24, 2020, proposed deep learning model which consisted of 3 CNN-based models (ResNet50, Inception-ResNetV2, and InceptionV3) for the detection of patients with COVID-19 using Chest X-ray images. They used 5-fold cross-validation. Their proposed model had an endto-end structure without selection methods and manual feature extraction. They also demonstrated that pre-trained ResNet50 was more effective than the other two models and Chest X-ray images were the best tool for the detection of COVID-19. Pre-trained models achieved very good results using a small dataset (50 normal vs. 50 COVID-19) from two different publically available datasets. Transfer learning technique was applied to overcome the low number of data and the insufficient training time. In this study, 80% (80 chest x-ray images) of the images were used for training and 20% (20 chest x-ray images) of them were used for testing. ResNet50 pre-trained model obtained the best proficiency with a specificity of 100%, an accuracy of 98%, and sensitivity of 96%. Inception-ResNetV2 showed the lowest performance with a sensitivity of 84%, an accuracy of 87%, and a specificity of 90%.
On March 27, 2020, Ghoshal et al (23) published an article about the uncertainty and interpretability of deep learning for COVID-19 detection. They showed that the drop-weights based Bayesian Convolutional Neural Networks (BCNNs) can estimate the uncertainty in deep learning approaches to improve the human-machine combination performance in diagnosing COVID-19. They showed that the uncertainty in prediction and the accuracy of the prediction are highly correlated with each other. About 5941 chest x-ray images of 4 classes, 1583 normal, 2786 bacterial pneumonia, 1504 non-COVID viral pneumonia, and 68 COVID-19 images, were used in this study. They used a public dataset. The experiment was run in a transfer learning setting using pretrained ResNet50V2 model and acquiring data only to fine-tune the original model. Eighty percent of the images were used in the training set and 20% of them were used for testing set. For enhancing the learning capability and preventing overfitting, they also applied real-time data augmentation, using Keras ImageDataGenerator during training. The analysis included a comparison of two variational dropweights based uncertainty measures, Bayesian Active Learning by Disagreement (BALD) and Predictive Entropy (PH).
On  (26) used deep learning and transfer learning algorithms to diagnose COVID-19 from x-ray and CT images. They proposed a modified CNN model and also used a pre-trained modified network namely AlexNet for accurate and fast detection of COVID-19. This model showed an accuracy of up to 98% using a pre-trained network and 94.1% accuracy via modified CNN. The achieved sensitivity using modified CNN was 100% for COVID-19 X-ray images and 90% for CT images. Pre-trained AlexNet identified all COVID-19 X-ray images and 97% of the normal x-ray images. However, AlexNet identified only 72% of COVID-19 CT images. The used data were publicly available and collected from different sources. 120 x-ray images (60 COVID-19, 60 normal) and 339 CT images (192 COVID-19 and 147 normal) were used to train the proposed CNN model. To test the proposed CNN models, 50 X-ray images (25 normal versus 25 COVID-19) were used and for CT images, 17 images were used in the testing phase, six of which were normal CT images and 11 of them were COVID-19 images.
In March 2020, Farooq and Hafeez (27) (29) to diagnose COVID-19 from chest x-ray images based on a small dataset. These networks were trained on images from ImageNet. Transfer learning was applied to reduce the required data. The used dataset was obtained from publicly available sources which contained 135 COVID-19 and 320 other pneumonia chest x-ray images. ResNet50 was trained on 102 non-COVID pneumonia images, and 102 COVID-19 cases in 10-fold cross-validation. Additionally, 33 COVID-19 images and 218 pneumonia cases were used to test the network. The source code was disclosed. This method achieved an overall accuracy of 89.2% for COVID-19, a specificity of 98%, and an AUC of 0.94. The results of this study were published on April 5, 2020.
On April 16, 2020, the results of a study by Afshar et al (30) about COVID-19 identification were published. They presented a modeling framework based on capsule networks, namely COVID-CAPS, to identify COVID-19 from chest x-ray images based on a small dataset. COVID-CAPS model is publicly available. The model showed an accuracy of 95.7%, a specificity of 95.8%, a sensitivity of 90%, and an AUC of 0.97. In order to improve the accuracy, they also used transfer learning and pre-training approaches using an external dataset of 94 323 chest x-ray images of common thorax diseases, which were obtained from a public dataset. Using the pre-trained external dataset improved accuracy, specificity, and AUC to 98.3%, 98.6%, and 0.97, respectively; however, sensitivity decreased to 80%. The COVID-CAPS model consisted of three capsule layers and four convolutional layers. The inputs of this network were 3D X-ray images.
Minaee et al (31) used deep learning for COVID-19 detection from 2D Chest images from the publicly available dataset, by fine-tuning four pre-trained CNN models, ResNet50, ResNet18, DenseNet-121, and SqueezeNet. The results showed a sensitivity rate of 97% (± 5%) and a specificity rate of about 90%. The AUC value of these four models was around 0.99. They found that SqueezeNet had a much better performance in distinguishing COVID and non-COVID images. Non-COVID-19 class in this study contained both normal cases and other disease types. The source codes were disclosed. The network was trained by data that included 31 images (496 after augmentation) with COVID-19, 2000 images that were non-COVID. The network was evaluated by testing it with 40 COVID-19 images and 3000 non-COVID images. Minaee et al used transfer learning because the images that were labeled as COVID-19 were very limited. The paper by Minaee et al (31) was published on April 20, 2020.
Oh et al (32) applied a patch-based CNN with few trainable parameters for COVID-19 detection. Their proposed model was composed of several components. The first part was the segmentation network. They utilized fully convolutional (FC)-DenseNet103 for segmentation and extracting the heart and lung contour from Chest images. The second part was the classification network with the aim of classifying chest images based on the type of disease. They used ResNet-18 as a classification network. In the third step, they investigated model interpretability by Grad-CAM map visualization method. To compose a total dataset, they combined COVID-19 image data collection with RSNA Pneumonia Detection Challenge dataset (9). Among them, 8651 normal, 5812 pneumonia, and 160 COVID-19 images were randomly selected for training. Then, 100 normal cases, 100 pneumonia cases, and 10 COVID-19 cases used as the test set. In this study, the resulted accuracy, sensitivity, and specificity for COVID-19 were 88.9%, 92.5%, and 96.4%, respectively. COVID-Net achieved 93.3% test accuracy and 91.0% sensitivity for COVID-19 cases. They also introduced COVIDx, an open-access benchmark dataset that they generated. To generate the COVIDx dataset, they used different available datasets like COVID-19 ActualMed COVID-19 Chest X-ray Dataset Initiative, Image Data Collection, RSNA Pneumonia Detection Challenge dataset, and COVID-19 radiography database. The dataset they used to train and evaluate the proposed network included 13 975 Chest X-ray images of 13 870 patient cases. It contained a total of 8066 patient cases who had no pneumonia (i.e., normal), 5538 patient cases who had other types of pneumonia, and 358 chest x-ray images from 266 COVID-19 patient cases. In order to test the network, they used 95 normal, 94 pneumonia, and 91 COVID-19 images. Wang et al (9) published their results on May 11, 2020.
Abbas et al (33) adapted their previous CNN model, called DeTraC model, to improve the performance of pretrained models in COVID-19 detection from chest x-rays. They disclosed the source code. They added an additional class decomposition layer to the pre-trained models. ResNet18, as an ImageNet pre-trained CNN model, was used for knowledge transformation. This model could deal with image dataset irregularities by investigating its class boundaries. An accuracy of 95.12% (with the specificity of 91.87% and the sensitivity of 97.91%) was achieved for this model in the detection of COVID-19 from normal and other pneumonia cases. They collected x-ray image datasets from several hospitals. The used datasets contained 80 samples of normal chest x-rays, 105 COVID-19, and 11 other pneumonia samples. Using data augmentation techniques, a total of 1764 samples were achieved. In this study, 70% of images were used for training and 30% of them were used for the evaluation of the classification performance. Their article was published on May 17, 2020.
Rahimzadeh and Attar (34) used a concatenated neural network based on ResNet50 and Xception networks for classification of the x-ray images of the chest, i.e. COVID-19, pneumonia, and normal. They introduced several techniques for improving the training process when the dataset was unbalanced, i.e., fewer cases of COVID-19 relative to more cases from other classes. Their network achieved a sensitivity of 80.53% and a specificity of 99.56% for the COVID-19 detection, while the overall accuracy was found to be 91.4%. They trained the network using data that included 149 images of patients with COVID-19, 1634 images of patients with pneumonia, and 2000 images of normal patients. They evaluated the network with test data from 31 images of COVID-19 patients, 4420 images of patients with pneumonia, and 6851 normal images, using 2 open-source datasets. The source code is publicly available. The paper by Rahimzadeh and Attar (34) was published on May 26, 2020.
The last one is Signoroni et al (35) that presented deep network architecture named Brixia-score Network (BS-Net), an end-to-end deep learning architecture, for prediction of the degree of lung compromise in COVID patients from chest x-ray images. The network showed good fine-tuning capability and good direct generalization of BS-Net. Images including 4707 clinical chest x-ray images from confirmed COVID-19 patients collected in the same hospital, corresponding to the entire amount of images taken during both triage and patient monitoring in sub-intensive and intensive care units of the hospital during one month. The network was trained and tested with 3313 and 449 images, respectively. The whole architecture source code was released but Brixia-dataset has been submitted to Ethics Committee and is currently pending. Signoroni et al introduced an end-to-end image analysis system for the estimation of a semiquantitative rating based on a difficult visual task. Semiquantitative scoring system was used to monitor the patients, having high prognostic importance. They adopted a weakly supervised learning strategy trained with a "from part to whole" procedure involving various different datasets for performing score estimation, spatial alignment, and segmentation. All image reports included the Brixia-score as a string of six digits indicating the scores assigned to each region, A to C for the right lung and D to F for the left lung. Scores were assigned by a team of about 15 radiologists operating in different radiology units of the hospital. The paper by Signoroni et al (35) was published on Jun 8, 2020.

Discussion
The coronavirus outbreak continues to surprise the world. The RT-PCR test is known as the standard method of COVID-19 detection, which is a time-consuming process. The patients must be isolated for a long duration until receiving the test results. Besides these disadvantages, the detection rate of such tests is low, i.e. between 30% and 50%. Hence, most of the time they need to be repeated to make a confirmation (36).
AI-based solutions with chest x-ray images have the potential to diagnose COVID-19 as a fast and accessible diagnostic modality. In our review, the papers were arranged according to their publication date. All studies used chest x-ray images. Seventeen studies used publicly available datasets and 10 studies disclosed source codes. Three studies used independent datasets, one was disclosed (9), one was not disclosed (33), and one of them was pending (35) until July 21, 2020. One of issues that need to be considered when releasing datasets is ethical considerations, and most of the studies done by the datasets have been cleared of these problems. Most of the studies used publicly available datasets. It is expected that the research will be further accelerated with published source codes and open datasets. All papers used deep learning as AI technique. Most papers have used novel (9,(16)(17)(18)21,22,27,30,35) or modified (20,23,24,26,28,(32)(33)(34) AI software based on deep learning-based neural networks. The COVID-19 prediction is the main topic of most papers and issues such as the severity of COVID-19 (35), uncertainty for COVID-19 detection (23), and classification of pulmonary diseases (20) have also been considered.

Conclusion
COVID-19 is currently an important issue for humans. Because of the rapid spread of this disease, chest x-ray images and AI can be used effectively in the rapid diagnosis of this disease. This review summarizes the currently published AI researches using chest x-ray for COVID-19 diagnosis. The studies revealed that AI can help humans in fighting the new coronavirus.