Why Most 'AI Ideas' Die at Data Collection

You come up with a new startup idea that's going to make you a Zuckerberg or a Steve Jobs. You want to make an ML model to solve X, or an ML model to help X.

Turns out building the infra, coming up with the idea, and coding the frontend were all pretty simple. And guess what-training the model using a CNN or SVM wasn't that hard either. You sit down, play Rocket League, have some boba while your model is training… oh, and I forgot to mention: you found a dataset on Kaggle. Seems legit, right?

You win a couple of games, come back to your overheating laptop, and see this:

Epoch 1/10

loss: 2.9876 - accuracy: 0.1123 - val_loss: 3.4567 - val_accuracy: 0.0987

Epoch 10/10

loss: 2.0765 - accuracy: 0.2190 - val_loss: 9.0023 - val_accuracy: 0.0495

Classification Report:

precision recall f1-score support

Class 0 0.12 0.15 0.13 120

Class 1 0.09 0.08 0.08 115

Class 2 0.11 0.10 0.10 130

Class 3 0.10 0.06 0.08 110

accuracy 0.12 475

macro avg 0.10 0.10 0.10 475

weighted avg 0.11 0.12 0.11 475

If this looks like linear algebra, or you think I wrote a bunch of random stuff, don't worry. This is the summary of you training an ML model, and this example shows how bad the model is at predicting an outcome-meaning the model hasn't learned anything.

You may go back to reevaluate whether there are any compute errors, any ways you can increase precision, whether you used the correct model, or whether there's some error unrelated to the data. But most of the time, it is the data.

The Alzheimer's Detection Problem

One example of this happening is when I had to make an ML model using a CNN and Bayesian modeling to detect Alzheimer's using MRI brain scan images. There was even a good Kaggle dataset. The dataset was sorted into four different categories of brain scan images: mild impairment, no impairment, moderate impairment, and very mild impairment, containing 2,560 images in total.

So the dataset seems legit-why wouldn't the model work? We implemented a two-hidden-layer CNN approach and used the feature map results, then plugged them into a Bayesian regression model, and the model gave very bad accuracy and couldn't accurately identify brain scans.

Our approach was correct, and we even double-checked it with ML experts. It turns out the dataset wasn't good enough. You need tons and tons of data. You need bad data. You need outliers. You need ~100,000 images of impaired patients to accurately identify Alzheimer's with high accuracy.

And it makes sense-and it's important that we do.

Why This Matters

If a healthcare professional uses this model with an accuracy of 63% and a recall for mild impairment of 0.4, there's a good chance that the prediction will be wrong.

This is why most ML startups die-not because the people making the product aren't talented or don't have the right approach, but because they don't have accurate enough data.

The Path Forward

There are many ways we can combat this, but it's expensive and time-consuming. Open-source data archives like Kaggle are great for training smaller models to prove a point. With my model, I can say that I'm able to find a positive correlation between impairment and no impairment with Alzheimer's.

However, to make production-grade software, data collection requires quantity and quality data. That might mean paying companies that have this data, or even running your own trials. Data is the main pain point of making and training an ML model.