1. Top
  2. News
  3. Unveiling New Risks Lurking in the High Performance of Artificial Intelligence: Anticipating Enhanced Reliability in Medical Image Diagnosis

Press Release

2023.11.27

Press Release

Unveiling New Risks Lurking in the High Performance of Artificial Intelligence: Anticipating Enhanced Reliability in Medical Image Diagnosis

With the advancement of artificial intelligence (AI) technology, particularly in the realm of deep learning, there is growing interest in its application for medical tasks. Deep learning requires a substantial amount of training data, presenting challenges in data acquisition. Specifically, the characteristics of training data significantly influence the performance of AI. For example, an issue named class imbalance, which means an imbalance in data number between disease and normal groups, can lead to decreased performance. For avoiding this issue, in situations where urgent cases like the diagnosis of COVID-19 make it difficult to secure sufficient data, efforts have been made to balance the number of disease and normal cases by collecting data from several hospitals. While this approach may seem to alleviate the class imbalance issue, it has been observed that a new problem, that developed AI can appear to be high-performing but may actually be poor-performing (Image 1).

The research group from the Department of Radiological Imaging and Informatics, Tohoku University Graduate School of Medicine, which includes Researcher Zhang Zhang, Associate Professor Xiaoyong Zhang (Current Affiliation: The Department of General Engineering, National Institute of Technology, Sendai college), and Professor Noriyasu Homma, have newly identified the cause of this performance decline, termed “intra-source imbalance”. Intra-source imbalance occurs when the AI inadvertently learns differences in data sources, such as the hospitals or devices used for imaging, which are unrelated to the medical differences between disease and normal groups. Using the example of COVID-19 diagnosis with chest X-ray images, our research looked into the impact of intra-source imbalance on the performance of deep learning. To confirm whether the model is learning medical differences, they performed image processing by hiding or cropping important parts of the lungs for diagnosis on an intra-source balanced dataset and an intra-source imbalanced dataset separately, to create separate datasets for each processing step (Image 2). The results of our experiments demonstrated a decrease in reliability when models were trained on intra-source imbalanced data. Additionally, when comparing the regions of interest highlighted by the model using visualization methods, the intra-source balanced data-trained model focused on more medically relevant areas (Image 3). This finding underscores the significance of addressing intra-source imbalance in training data of AI models, which will lead to the development of trustworthy AI in the field of healthcare in the future.

Details of their research were published in the journal Scientific Reports on November 3, 2023

Publication Details:
Title: How intra-source imbalanced datasets impact the performance of deep learning for COVID-19 diagnosis using chest X-ray images
Authors: Zhang Zhang, Xiaoyong Zhang, Kei Ichiji, Ivo Bukovský and Noriyasu Homma
Journal: Scientific Reports
DOI: doi.org/10.1038/s41598-023-45368-w
https://www.nature.com/articles/s41598-023-45368-w

Contact: 
Name: Zhang ZHANG, Noriyasu HOMMA
Affiliation: Department of Radiological Imaging and Informatics, Tohoku University Graduate School of Medicine
Email: zhangzhang*tohoku.ac.jp Please replace “*” with “@”.
homma*med.tohoku.ac.jp Please replace “*” with “@”.
Website:  https://www.rii.med.tohoku.ac.jp/

[Image 1]This is an example where a deep learning model accurately diagnosed COVID-19 infection even in images where the medically significant region, the lungs (blacked out), was removed. However, the basis for this, such as the focus being outside the lung area, was medically inappropriate. Thus, the evidence acquired by deep learning models may have lower reliability compared to medical observations, but the reasons have not been clarified.
[Image 2]To investigate the features on which a deep learning model bases its diagnosis of COVID-19 infection in chest X-ray images, we applied image processing techniques to extract or remove specific regions such as the lungs from the original images. In this study, we introduced two new datasets: one with intra-source balance and the other with intra-source imbalance. These datasets were utilized for both training and evaluating the performance of the deep learning model. By comparing the results between the balanced and imbalanced datasets, we examined the impact of the intra-source imbalance.
[Image 3]The regions focused by the two models in diagnosing COVID-19 infection: (A) Regions of interest for the model trained on the intrinsically imbalanced dataset. (B) Regions of interest for the model trained on the intrinsically balanced dataset. Arrows in the images indicate the areas that contributed most to the infection diagnosis. The model trained on the intra-source imbalanced dataset (A) is directing attention to text ‘D’ in the medically improper area outside the lungs. In contrast, the model trained on the intra-source balanced dataset (B) is appropriately focusing on the medically relevant lung region, demonstrating higher reliability in the basis of diagnosis.