🎯 Objective
Learn how IADX, a AI-focused startup I co-founded, in developing a generalist AI model for x-ray images, and learn how the AI sausage is made.
👨💻 Back story
I am a computer scientist who specializes in medical AI research. I got my PhD in 2021 and since then have worked on a variety of projects helping medical researchers explore the possibility of AI to help clinicians diagnose and treat various conditions. But things started to get a bit more intense in 2021 when my brother, an orthopedist, started talking to me about fractures that were being missed by doctors with alarming frequency.
According to him, up to one third of fractures were being misdiagnosed as healthy by overworked physicians reviewing X-rays.
Can you build an AI that catches the fractures these doctors are missing?
This was the challenge my brother ended up giving to me.
Casual dares like this were nothing new from him.
He had this habit of hitting me with problems he faced in his daily life - from automating repetitive tasks in linux/windows/games to building an AI for playing video games.
But this was different - this was a real problem he was facing in his daily work that could have a huge impact on other people's lives since he, as an orthopedist, was noticing a lot of false negatives from the physicians.
I was intrigued by the idea, so we decided to work on it together on our spare time: he would annotate the images while I would collect data and build the AI model. Our startup company, IADX, was born.
I expected the AI modeling to be our biggest hurdle. I was wrong. Data engineering consumed 80% of our time - collecting, cleaning, and organizing dozen of thousands of X-rays.
Our data pipeline became increasingly complex as we tracked new annotations, managed versions, and maintained data quality. The process of emailing images back and forth, along with complex files pinpointing the fracture location quickly became a huge mess of file versions and spreadsheets with long names.
With a mix of effort and minor panic attacks, we finally managed to build out a first version of this AI model and now it is working pretty well. You can even check it out for free at https://iadx.com.br/try (the website is currently in portuguese only).
With lessons learned I began looking for a tool to help with organizing the research, and settled on Datamint, who is hosting this article. (This series isn't about Datamint though, it's about the process of developing a medical AI algorithm).
Go Bigger
Fortunately or unfortunately, this success only led us to expand our ambitions.
With our appetite whetted, we are moving on to a way bigger project, a few-shot learning model for x-ray images and we are facing the same problem - the data organization - but in a larger scale.
This few-shot learning model would be able to detect an unseen pathology in a x-ray image with only a few examples (e.g. 5 images) of that pathology. It is a huge challenge, but a very promising one for the medical community, since many studies have shown that ~30% of x-ray images are misdiagnosed with a false negative by physicians. [1] This model would also be able to help many clinics and hospitals in Brazil where there is a shortage of specialized doctors.
The real challenge here is getting a diverse dataset to train on, one that includes thousands of images from different sources and different pathologies, conditions, and body parts.
Yep, this is a huge challenge, but we are up for it.
Step 1: Get the data
Fortunately, we live in 2025 when there are many open source datasets that I told my brother we could use to jumpstart our efforts. Step One was therefore to choose the best open source dataset to base our research on. I went looking... and looking... and looking...
And found no dataset that satisfies our need.
All are in some way incomplete or lacking in diversity. 😞
For instance, many datasets are focused on a single pathology, like pneumonia or tuberculosis, or a single body part, like the chest. Others are too small, with only a few dozen of images.
So Step 1 will actually be to assemble and annotate this dataset so we can get underway with the research. As usual, 80% of the work is in the data.
Our intention is:
Annotate a dataset with a diverse set of pathologies/conditions, such as fractures, dislocations, tumors, osteopenia, pneumonia, tuberculosis, and more.
Build a dataset with a diverse type of annotations, including bounding boxes, segmentation masks and classifications.
Complete the incompleted datasets with all possible objects that we can find in x-ray images, such as pathologies (e.g, fractures and pneumonia), specific body parts (e.g, each meta-carpal bone segmentation), and other objects (e.g, metals and texts in the image).
We have long way to go, but when we are complete, we will make this data freely available to the research community for their own research, or to extend ours, so nobody else has to go through the same pain we did.
We are grateful to Datamint for sponsoring the development of this dataset.
Conclusion
The conclusion is the same as it always is in AI research: it's all about the data. We had hoped to get started by extending our previous algorithms to a larger and more diverse dataset, but instead we need to retreat to the stage of collecting and marking up raw data. We hope we'll be rewarded with a solid foundation to explore our ideas.
In Part 2, I'll detail the requirements we have for our collection of x-rays, how we go about finding them, and our strategy for getting them annotated by orthopedists.
References
[1] [Improving Radiographic Fracture Recognition Performance and Efficiency Using Artificial Intelligence](https://pubs.rsna.org/doi/pdf/10.1148/radiol.210937)
Comments