AutoML Poll results: if you try it, you’ll like it more

The results of latest KDnuggets Poll on AutoML are quite interesting. While most respondents were not happy with AutoML performance, the opinions of those who tried it were higher than those who did not.

Recent KDnuggets poll asked:
How well do current AutoML solutions work, in your opinion?
with options
  1. Badly: Do not really work
  2. Minimally: Help business users solve their problems with minimal help from Data Scientists
  3. Good enough: Allow business users to solve their problems by themselves
  4. Very good: Help Data Scientists to be a lot more productive
  5. Super-human: automatically find solutions that are better than those created by most Data Scientists
We also asked whether a person tried AutoML on a realistic data problem, and their employment type.

About 500 respondents took part in this poll. Overall results (Fig. 1 below) show that the majority of respondents were not very happy with AutoML.

Automl Poll Quality
Fig. 1: AutoML quality (KDnuggets Poll)

If we give a quality rating 1 to "Badly", 2 to "Minimally", and so on until 5 to "Super-human", then the average AutoML quality rating was 2.4.

However, a more interesting picture emerges if we examine the responses depending on whether they tried AutoML (57%) or not (43%)

Poll Automl Tried Vs Not
Fig. 2: AutoML quality vs Tried AutoML or Not.
Tried: yes is shown in gold, no in blue. Percentages are relative to the total number of those who either tried or not tried.

We see that a much smaller fraction (34.0% vs 45.4%) thought that AutoML performed minimally, and twice as many (22.8% vs 10.2%) thought that AutoML was very good! There was even a small increase in the fraction that said AutoML performance was superhuman - from 2.4% to 3.7%. Percentages for those who thought AutoML performed badly remained the same, and the change for "Good Enough" was not significant.

The average AutoML quality for who did not try it was 2.29, while for those who did try was 2.56.


The employment type of respondents was:
  • Industry/Self employed, 75.3%
  • Student, 9.1%
  • Academia, 7.4%
  • Government/non-profit, 3.8%
  • Other, 4.4%
Fig. 3 shows perceived quality by employment type. Bar length and color correspond to the average quality, while bar height to the number of respondents in that group.

KDnuggets Poll: AutoML Quality by Employment
Fig. 3: AutoML quality by Employment.

We see that "Other" reports the highest quality, "Industry" is in the middle, and "Government" reports the lowest quality.

However, if we break down by "Tried" vs "Not Tried", we see a similar pattern where the perceived quality is higher among those who tried AutoML, across almost all employment types.

Poll Automl Employment Tried
Fig. 4: AutoML quality by Employment and Tried AutoML or Not.

Finally, we looked at geography. Poll participation by region was
  • Europe, 33.8%
  • US/Canada, 32.8%
  • Asia, 20.9%
  • Latin America, 5.9%
  • Africa/Middle East, 3.6%
  • Australia/NZ, 3.0%
Breakdown by region shows a similar pattern for US and Europe: those who tried AutoML report higher quality. Surprising, this is not true for Asia and other regions.

Poll Automl Region Tried
Fig. 5: AutoML quality by Employment and Tried AutoML or Not.

The implication of this poll seems to be - AutoML quality, while not perfect, is actually higher than many think. Try it - you may like it!

Note: We did NOT ask in this poll which AutoML solution was tried. Why? Because there are many AutoML solutions now available and we did not want to turn this poll into a vendor contest.