Annotation Quality versus Quantity in Medical AI

How much data do you need?

If there is anything we’ve learned as ChatGPT and its sisters have begun their conquest of the human race, it’s that if you want to improve your model, throw more data at it. The LLMs and image generation models were trained on the complete corpus of human knowledge, plus Reddit and Facebook. 

This is likely the reason one of the first questions a clinical researcher asks when wading into the medical AI research pond is how many images they’ll need. And why the first response to an underperforming model is to think about where they can find another few hundred MRIs to train on. 

But firing the data cannon at a poorly-performing model doesn’t work as well in medical imaging AI. It can help, but, counter-intuitively, medical imaging data quality is usually a bigger lever than data quantity. Understanding that can help you better allocate your resources and even help you get started faster and make quicker progress once you’re underway.

Ground truth in medical AI

Ground truth is the most important asset you have in a medical AI project. If you are training a neural network to find truffles in the mud, then “ground truth” is absolute. There is a truffle right there or there isn’t. Humans assigning these labels might make mistakes, but the “truth” exists. The errors are noise in the signal.

In medicine, though, ground truth is considerably less comfortable. In the best case, ground truth is the diagnostic reality of the patient’s condition: your AI is trying to detect a tumor from an ultrasound, and ground truth is the result of a biopsy. If that kind of data is available, you are truly blessed by the Machine Learning Gods.

In 99% of situations, though, ground truth is the clinical judgement of an experienced practitioner, usually more than one. For medical AI data annotation, that means your labels are often based on expert interpretation rather than an absolute truth. And those experienced practitioners rarely agree with each other completely. We generally measure inter-rater agreement with a “Dice score”, which ranges from 0 (no overlap) to 1 (perfect agreement). In practical AI projects, a Dice of 0.75 - 0.85 is typical and perfectly acceptable.

Note, though, that the individual readers will be fairly consistent in their own readings, even though they disagree with other doctors. The errors are systematic rather than random. With random noise, the errors fluctuate around zero and cancel each other out as more data is collected, while the true signal gets stronger. With systematic error, the error remains constant when more data is added.

All is not lost, however. Although the error is systematic, it is also typically localized within the data: that is, the readers will typically largely agree with each other, and diverge mostly on a consistent subset of the images. These are the cases that are ambiguous or borderline, and it gives us a strategy to reduce the effect of noise in medical data.

How improving quality improves model training

Imagine that you have a dataset where the dice score is 0.8. About 15% of the cases could be considered borderline or ambiguous, and the majority of the disagreement between the experts is concentrated in that 15%. As you collect more and more images, the disagreement in the ambiguous portion will remain more or less constant, and the disagreements between the experts will become more strongly memorized by the neural network. Your model memorizes the error, and becomes more and more confident about the borderline cases. It becomes arrogantly wrong.

That’s in the case where the extra data is added to your training set. Data added to your test set is even more pernicious. Including significant systematic error in your evaluation data degrades your ability to know anything about the models you are developing. While error in the training set caps your model performance, the test set will tell you your model is only as good as the systematic error even if your model is perfect. As you modify architectures, hyperparameters and so on, you cannot tell if they are improving your results. It will reward a model that reproduces the systematic error more than a model that improves accuracy.

It can even reduce your ability to tell two models apart. Noise in the test labels adds variance to the metric. Once the real difference between two models is smaller than the noise-induced variation, you can't distinguish them. Even worse, you start "improving" your metrics by finding the architecture or training method that happens to fit the error.

However, locating and reducing that systematic error has the opposite effect. Better data labeling quality in the training set allows your model to achieve better performance. With less systematic error in the test set, you’ll be able to detect that improvement. It has been demonstrated that models trained with a large dataset can be greatly outperformed by models of identical architecture trained with a smaller, but higher quality set of annotations.

How to improve annotation quality

There are a few strategies to improve annotation quality. 

1. Carefully design your medical image annotation strategy. Document annotation protocols and give guidance to readers.

2. Annotate a small subset of images and measure inter-reader agreement on that subset before you proceed with the full test set. For tasks such as medical image segmentation, this can help identify areas where readers consistently interpret boundaries differently.

3. Assemble the readers and manually go over the images where disagreement was highest. Adjudicate those images and add the results to your annotation protocol.

4. When performance starts to stall, prioritize improving labels (through more group adjudication) over adding data.

Improving medical imaging data quality is not necessarily easy, cheap, or fast. But it is usually at least as easy as locating and annotating more images, and the point is to allocate effort where it has the highest payoff. 

Takeaways

I don’t want to overstate the “quality versus quantity” argument. Quantity is important in medical image AI. You need diversity of population and condition severity, and although systemic error dominates there is still substantial random error that can be averaged by adding data. So yes, get as much data as you can.

However, I want to shift your first question from “how do I get more data?” to “how good are my labels?”. The nature of medical imaging AI is that improving the medical image quality and overall data labeling quality will likely have outsized effect on the quality of the output.

1 Wong J, Baine M, Wisnoskie S, Bennion N, Zheng D, Yu L, Dalal V, Hollingsworth MA, Lin C, Zheng D. Effects of interobserver and interdisciplinary segmentation variabilities on CT-based radiomics for pancreatic cancer. Sci Rep. 2021 Aug 11;11(1):16328. doi: 10.1038/s41598-021-95152-x. PMID: 34381070; PMCID: PMC8357939.

2 Elise Covert, Kellen Fitzpatrick, Justin Mikell, Ravi Kaza, John Millet, Daniel Barkmeier, Joseph Gemmete, Jared Christensen, Matthew Schipper and Yuni Dewaraja, Journal of Nuclear Medicine August 2022, 63 (supplement 2) 2808;

3 Henderson EGA, Vasquez Osorio EM, van Herk M, Brouwer CL, Steenbakkers RJHM, Green AF. Accurate segmentation of head and neck radiotherapy CT scans with 3D CNNs: consistency is key. Phys Med Biol. 2023 Apr 3;68(8). doi: 10.1088/1361-6560/acc309. PMID: 36893469.