Leveraging Zero-Shot Learning to Detect Pathologies with Unannotated Data

Artificial intelligence and machine learning have revolutionized a wide range of industries and fields of study, resulting in significant advancements and improvements. In particular, the healthcare industry has seen a significant impact from these technologies. Deep learning has enabled the automation of complex medical image interpretation tasks, such as disease diagnosis and tumour segmentation, end-to-end.

This has not only increased the efficiency and accuracy of these processes, but also has the potential to improve patient outcomes by allowing for earlier and more accurate diagnosis. More specifically, the integration of AI and ML in healthcare is not limited to image interpretation tasks, but also encompasses areas such as personalized medicine, drug discovery, and clinical decision-making. These technologies have the potential to improve the overall quality of healthcare by enabling more accurate and personalized treatments, reducing costs, and increasing access to healthcare.

If you want to check out an example, over this month, I built a Nuclei Segmentation model and a COVID-19 diagnosis model (from CT scans). I hyper-linked my Github, check it out!

Getting back on track; there have been significant advancements made in diagnostic efficiency and automated deep learning models. As mentioned earlier, AI and Machine Learning has made its way into the field of biomedical imaging. Machine Learning models have been utilized to analyze powerful statistical models with a diverse range of applications.

One of the more commonly known models are Convolutional Neural Networks to assist radiologists in making accurate and efficient diagnostic decisions. However, these deep learning models often require large labelled datasets during the training phase. There are large scale efforts being made to label data, but they’re often expensive and very time-consuming. They require a lot of domain knowledge and technical expertise to execute and implement this for a very specific medical task.

Before getting into the very specifics of this article, for context, this article will discuss a new approach to detection pathologies from unannotated chest X-ray images with self-supervised learning.

To get around this issue, many approaches have been taken. For example, pre-trained models and self-supervision has been proposed to reduce the models’ reliance on large labelled datasets. These attempts have shown promise, self-supervised pre-trained models have increased the label efficiency across a multitude of medical tasks, but they are still in need of a ‘supervised fine-tuning step after pre-training’. This step in particular requires manually labelled data for the model to predict relevant pathologies. From this, it becomes evident that this approach is only able to predict diseases that were annotated in the dataset. They fall short when it comes to predicting pathologies that aren’t explicitly annotated for training. To obtain these high-quality annotations of certain pathologies is very costly and time-consuming. Additionally, they lead to significant operational inefficiencies within the clinical AI process.

Quick TL;DR 📑

Throughout this article, I’m going to highlight a recent paper that addressed these limitations by applying a machine-learning paradigm where a model can classify samples that aren’t explicitly annotated during training. They proposed a zero-shot method that uses a fully self-supervised learning process that doesn’t need any explicit manual or annotated labels for chest X-ray image interpretation. The method used contrastive learning; a type of self-supervised learning to enable zero-shot multi-label classification. Comparing this to other self-supervised approaches, this new model/method requires no labels except for testing and is able to identify pathologies with a high accuracy. To develop this method, they ‘leveraged the fact that radiology images are naturally labelled through corresponding clinical reports and that these reports can offer a natural source of supervision.’ Throughout this paper, it was seen that the performance of this method is able to compete with the results from both expert radiologists and fully supervised methods.

Understanding Zero-Shot Learning ☄️

Before jumping into the thick of the results, I’ve name dropped zero-shot learning a couple times, let me get into the specifics of what zero-shot learning actually is. Zero-shot learning is a technique that allows a model to classify new objects or classes without any prior training examples. This is achieved by training the model on a set of classes and then using a semantic representation of the classes, such as class attributes or word vectors, to generalize to new classes. For example, if a model is trained to recognize animals like dogs, cats, and birds, it can be used to classify a new animal like a zebra without ever having seen a zebra before. The model can do this because it has learned the semantic attributes of animals, such as “four legs” and “stripes,” and can use this knowledge to infer that a zebra is an animal with four legs and stripes.

One of the key advantages of zero-shot learning is that it allows a model to learn from a smaller amount of labeled data. This is particularly useful in real-world applications where data is scarce or expensive to collect. For example, in the medical field, it may be difficult to obtain labeled data for rare diseases, so a zero-shot learning model can be trained on common diseases and then used to classify rare diseases based on their semantic attributes.

Another advantage of zero-shot learning is that it can be used to classify new classes that have not been seen before. This is important in dynamic real-world applications where new classes may emerge over time. For example, in the field of computer vision, new object classes may be introduced, such as new types of vehicles or new types of animals. A zero-shot learning model can be trained on existing classes and then used to classify new classes based on their semantic attributes.

However, zero-shot learning also has its limitations and challenges. One of the main challenges is the need for accurate semantic representation. If the semantic representation is not accurate, the model may not be able to generalize to new classes. Additionally, zero-shot learning models may have difficulty generalizing to unseen instances of seen classes. This is because the model has not seen any examples of these instances during training and may not be able to infer the correct class based on semantic attributes alone.

Now that we have a general understanding of what zero-shot learning is, let’s dive into the results of the paper.

Results 📁

Zero-shot learning was leveraged to classify pathologies in chest X-ray without training on explicit labels. In order to do this, image-text pairs of chest X-rays and radiology reports were utilized. By using these 2 data sources, the model learned to predict which chest X-ray corresponds to which radiology report. The model was trained with over 377,000 pairs of image-text pairs of chest X-rays. This data was retrieved from the MIMIC-CXR dataset.

After training and testing the model, the results were quite promising, in fact, they were comparable to 3 benchmark radiologists classifying the 5 CheXpert competition pathologies that were evaluated on the dataset. To evaluate the model, Matthew’s correlation coefficient (MCC) was used. The results on the MCC metric was so promising that there was no statistical significant difference between the models performance and that of the benchmark radiologists. This was just an example of some of the many statistics that shows that the self-supervised model can compete against the best of the best. Other statistics include F1 performance and receiver operating characteristic (ROC) curve. Here are the results visualized:

The self-supervised model was then tested against 3 previous label-efficient models as well, MoCo-CXR, MedAug, and ConVIRT. Before diving into the test results, let’s get a better understanding of what these models are and how they work.

The MoCo-CXR and MedAug model uses self-supervision using chest X-ray images in the CheXpert dataset. More specifically, the MoCo-CXR model modifies the contrastive learning framework, Momentum Contrast (MoCo), to interpret chest X-ray images. MedAug on the other hand builds on the pre-training on the MoCo framework; patient metadata is used to select positive chest X-ray image pairs for contrastive pre-training. ConVIRT uses the image-text pairs to conduct self-supervision. It uses a ResNet-50 and a Transformer by taking advantage of randomly sampled text from paired chest X-ray and radiology-report data. The key difference between these models and the CheXzero model is that these models require some labelled data, even if it is a small fraction. Here are the percentages of how much of the data was labelled:

The results of testing the CheXzero model against three other models were highly encouraging, showcasing the potential of this innovative approach. Despite utilizing 0% labeled data, the CheXzero model achieved the highest mean AUC of all self-supervised models, demonstrating its ability to perform at a level comparable to models that rely on labeled data. This is a significant achievement, as labeled data can be costly and time-consuming to acquire, making the zero-shot learning approach highly efficient and cost-effective.

The CheXzero model’s performance in comparison to the other three models was analyzed in detail, and the results were impressive. The model outperformed the other models in several key areas, particularly in its ability to accurately identify and classify specific conditions. These results demonstrate the potential of the CheXzero model to revolutionize the field of biomedical-imaging and pave the way for further research and development in this area.For a detailed analysis, the following are the results of the CheXzero model in comparison to the other three models:

Extrapolating CheXzero to PadChest Data 📈

The flexibility of zero-shot learning enables the self-supervised model to perform a wide range of auxiliary tasks related to the content found in radiology reports. In addition to its previous applications, the model was further tested by tackling tasks such as differential diagnosis utilizing the PadChest dataset, predicting patient gender, and identifying chest radiograph projections. The results of applying the self-supervised model to differential diagnosis on the PadChest dataset were particularly noteworthy. Despite utilizing 0% labeled data, the model outperformed the approach used in the CheXNet study on three out of the eight selected pathologies; atelectasis, consolidation, and oedema. This highlights the potential of zero-shot learning models to achieve comparable or even superior performance compared to models that rely on labeled data. The results of this experiment are shown below:

Final Thoughts 💭

Overall, it can be seen that the self-supervised method achieved very promising results in the CheXpert competition. The performance of the zero-shot model was comparable to radiologists and there was no statistical significant difference between the F1 and MCC performances. Moreover, the self-supervised model out-performed the previous label-efficient models for chest X-ray pathology classification. This goes to show that putting time and money towards explicit labels isn’t a pre-requisite for well performing medical-image models. These results were attained from a deep learning model that uses available radiology reports as a natural signal.

The fascinating part is that the results were externally validated on the MIMIC-CXR dataset, and on 2 other independent datasets; the CheXpert test dataset and the human-annotated subset of the PadChest dataset 🤯

History has shown that models trained with small amounts of labelled data have shown promise, but they still required some explicit labels and annotations. The scary part is, these annotations might not be easy to obtain either. For example, one of the models achieved a mean AUC (area under ROC curve) of 0.870 on the CheXpert dataset using 1% labelled date. The downside is that most large-scale models use large amounts of data; 1% of the data can turn out to be a very large number. Another major downside is that these previous models can only predict the pathologies that were labelled during training. This is a huge limitation and prevents the model from being extrapolated further.

“The self-supervised method has the potential to alleviate the labelling bottleneck in the machine-learning pipeline for a range of medical-imaging tasks by leveraging easily accessible unstructured text data without domain-specific pre-processing effort”

However, this model is still a work in progress and is far from perfect. First of all, while the self-supervised method eliminates the need for labeled data, it still necessitates the ongoing evaluation of performance using a labeled validation set for the selection of hyper-parameters and the determination of condition-specific probability thresholds for the calculation of MCC and F1 statistics.

The second drawback is that the CheXzero model is limited to image data, it can’t be extrapolated to other types of data. However, in most cases, it is possible to extrapolate non-image data. “For instance, magnetic resonance imaging and computed tomography produce three-dimensional data that have been used to train other machine-learning pipelines.”

Overall, the CheXzero model has a big future ahead of it as it has shown lots of promise in the biomedical-imaging field. The results discussed in this article emphasize the potential that these zero-shot models hold. Additionally, the CheXzero model’s ability to perform well with zero labeled data highlights the efficiency and cost-effectiveness of this approach. The CheXzero model has the potential to revolutionize the way biomedical-imaging tasks are performed, and I’m looking forward to further exploring and developing this technology in the future.

If you’d like to check out the github for this model, it is linked here.

If you enjoyed this article, feel free to give it claps and share it! You can catch me on LinkedIn and if you want to check out some of my other work, here’s my personal website :)

References

1. “Classification: Roc Curve and AUC | Machine Learning | Google Developers.” Google, Google, developers.google.com/machine-learning/crash-course/classification/roc-and-auc#:~:text=AUC%20(Area%20under%20the%20ROC,than%20a%20random%20negative%20example.
2. Shah, Deval. “Self-Supervised Learning and Its Applications.” Neptune.ai, 14 Nov. 2022, neptune.ai/blog/self-supervised-learning#:~:text=Self%2Dsupervised%20learning%20is%20a,by%20auto%2Dgenerating%20the%20labels.
3. Tiu, Ekin, et al. “Expert-Level Detection of Pathologies from Unannotated Chest X-Ray Images via Self-Supervised Learning.” Nature News, Nature Publishing Group, 15 Sept. 2022, www.nature.com/articles/s41551-022-00936-9.
4. Tiu, Ekin. “Understanding Zero-Shot Learning — Making ML More Human.” Medium, Towards Data Science, 7 July 2021, towardsdatascience.com/understanding-zero-shot-learning-making-ml-more-human-4653ac35ccab.