Don't wander into an AI project and try to manage it like a normal software project. You'll make a mess.
The Awakening
A few years ago I was asked to manage the software development process at Medo.AI, a company working on AI for ultrasound scans. The team was small, enthusiastic, and hardworking, but progress was unpredicable and the company needed to be able to plan better. I dove in, but it became pretty clear to me that a ton of the overall uncertainty was coming from the rapid-but-random progress of the data science team. When I proposed some structure to track their work, the brilliant scientist-founder of the company gently but firmly put me back in my lane: I would only end up slowing them down for no benefit.
After Medo.AI was acquired, I joined an ultrasound AI research lab at the local university as a research assistant. The principal investigator was not only a leading AI computer scientist, but a medical doctor and practicing radiologist. Every Monday he would come into the lab and apply the state-of-the-art in AI project management: He'd go around the table asking his postdocs "How's it going?". To which they would reply, "great!"
Over this time, the truth gradually revealed itself to me. AI researchers are unmanageble.
But they're not unmanageable because of their personality, but the nature of the work. Data science projects are inherently unsuited to traditional project management tools, and trying to use them is counterproductive.
But why?
AI projects involve writing code, but they're not normal software projects. (The question of whether AI programmers are normal is left for another day.)
Probabilistic versus Deterministic. Although a programmer staring down a bug that only happens at high tide in Brisbane while Venus is transiting Uranus, traditional programming is deterministic. Given the same inputs, a regular program will produce the same inputs, whereas a machine learning model could well produce different inputs for the same inputs. This means that every management question that wants an answer to "is it working yet" will always get the same answer - "it depends what you mean by working". Milestones on Gantt charts will be mocked mercilessly.
Exploratory versus goal based. Software projects are usually analogous to pointing to a destination on a map and telling your team to get you there. With AI projects, you often not only don't know exactly where the destination is, you don't even know if the destination exists. If it does exist, you don't know if it's possible to get there from here. It's more like a computer role-playing game where you start with a completely obscured map that has to be uncovered over time. Management techniques that try to track where you are on the Road to Rome will not be very useful. Burndown charts will burn up your team.
Step-function progress versus incremental improvements. While a traditional software project isn't exactly a linear progression from start to finish - proven by the established fact that the first 90% of the work takes 90% of the time, and the last 10% of the work takes the other 90% of the time - the work at least breaks down into a series of dependent tasks that lead toward a goal. In contrast, AI work often involves fiddling with various hyperparameters, model variant selection, and learning parameters until something suddenly pays dividends. A project can go from zero progress to significant progress overnight. Trying to impose sprints on this reality will lead to a lot of resume polishing.
Data dependance. The data in a software project is generally well understood - the static data from a database will be defined by a schema and the quantity of data fairly well known, and the dynamic data like user inputs can be constrained by the user interface. In a machine learning project, though, the quantity and characteristics of the data is fundamentally intertwined with the success of the models being developed. Furthermore, the amount of data is typically changing as the project proceeds. An approach can suddenly start working because a critical amount of data has been obtained, or an approach that seemed really promising can fail when released into the wild. All this contributes to a radical uncertainty about how successful an approach actually is, even if the test set shows promise.
So now what?
Developing TikTok for Cats or a fluid mechanics simulator are challenging projects, but they have some level of certainty of behaviour and a progression from start to end. Software management techniques like Scrum, XP, waterfall, or any combination of those are attempts to track where the project is in that journey. An AI or machine learning project is more an oil painting that has been painted over: a restorer gradually exploring the canvas and removing the outer layer hoping to find something valuable underneath.
It's hard to see how you can manage a process like that. But at some point, the people paying the bills need an answer to the professor's question - "How's it Going?" In my next instalment I'll talk about some of the ways a manager can clear away a bit of the fog.
Comments