AutoML Poll results: if you try it, you’ll like it more
The results of latest KDnuggets Poll on AutoML are quite interesting. While most respondents were not happy with AutoML performance, the opinions of those who tried it were higher than those who did not.
Recent KDnuggets poll asked:
About 500 respondents took part in this poll. Overall results (Fig. 1 below) show that the majority of respondents were not very happy with AutoML.
Fig. 1: AutoML quality (KDnuggets Poll)
If we give a quality rating 1 to "Badly", 2 to "Minimally", and so on until 5 to "Super-human", then the average AutoML quality rating was 2.4.
However, a more interesting picture emerges if we examine the responses depending on whether they tried AutoML (57%) or not (43%)
Fig. 2: AutoML quality vs Tried AutoML or Not.
Tried: yes is shown in gold, no in blue. Percentages are relative to the total number of those who either tried or not tried.
We see that a much smaller fraction (34.0% vs 45.4%) thought that AutoML performed minimally, and twice as many (22.8% vs 10.2%) thought that AutoML was very good! There was even a small increase in the fraction that said AutoML performance was superhuman - from 2.4% to 3.7%. Percentages for those who thought AutoML performed badly remained the same, and the change for "Good Enough" was not significant.
The average AutoML quality for who did not try it was 2.29, while for those who did try was 2.56.
The employment type of respondents was:
Fig. 3: AutoML quality by Employment.
We see that "Other" reports the highest quality, "Industry" is in the middle, and "Government" reports the lowest quality.
However, if we break down by "Tried" vs "Not Tried", we see a similar pattern where the perceived quality is higher among those who tried AutoML, across almost all employment types.
Fig. 4: AutoML quality by Employment and Tried AutoML or Not.
Finally, we looked at geography. Poll participation by region was
Fig. 5: AutoML quality by Employment and Tried AutoML or Not.
The implication of this poll seems to be - AutoML quality, while not perfect, is actually higher than many think. Try it - you may like it!
Note: We did NOT ask in this poll which AutoML solution was tried. Why? Because there are many AutoML solutions now available and we did not want to turn this poll into a vendor contest.
Related:
How well do current AutoML solutions work, in your opinion?with options
- Badly: Do not really work
- Minimally: Help business users solve their problems with minimal help from Data Scientists
- Good enough: Allow business users to solve their problems by themselves
- Very good: Help Data Scientists to be a lot more productive
- Super-human: automatically find solutions that are better than those created by most Data Scientists
About 500 respondents took part in this poll. Overall results (Fig. 1 below) show that the majority of respondents were not very happy with AutoML.
Fig. 1: AutoML quality (KDnuggets Poll)
If we give a quality rating 1 to "Badly", 2 to "Minimally", and so on until 5 to "Super-human", then the average AutoML quality rating was 2.4.
However, a more interesting picture emerges if we examine the responses depending on whether they tried AutoML (57%) or not (43%)
Fig. 2: AutoML quality vs Tried AutoML or Not.
Tried: yes is shown in gold, no in blue. Percentages are relative to the total number of those who either tried or not tried.
We see that a much smaller fraction (34.0% vs 45.4%) thought that AutoML performed minimally, and twice as many (22.8% vs 10.2%) thought that AutoML was very good! There was even a small increase in the fraction that said AutoML performance was superhuman - from 2.4% to 3.7%. Percentages for those who thought AutoML performed badly remained the same, and the change for "Good Enough" was not significant.
The average AutoML quality for who did not try it was 2.29, while for those who did try was 2.56.
Employment
The employment type of respondents was:
- Industry/Self employed, 75.3%
- Student, 9.1%
- Academia, 7.4%
- Government/non-profit, 3.8%
- Other, 4.4%
Fig. 3: AutoML quality by Employment.
We see that "Other" reports the highest quality, "Industry" is in the middle, and "Government" reports the lowest quality.
However, if we break down by "Tried" vs "Not Tried", we see a similar pattern where the perceived quality is higher among those who tried AutoML, across almost all employment types.
Fig. 4: AutoML quality by Employment and Tried AutoML or Not.
Finally, we looked at geography. Poll participation by region was
- Europe, 33.8%
- US/Canada, 32.8%
- Asia, 20.9%
- Latin America, 5.9%
- Africa/Middle East, 3.6%
- Australia/NZ, 3.0%
Fig. 5: AutoML quality by Employment and Tried AutoML or Not.
The implication of this poll seems to be - AutoML quality, while not perfect, is actually higher than many think. Try it - you may like it!
Note: We did NOT ask in this poll which AutoML solution was tried. Why? Because there are many AutoML solutions now available and we did not want to turn this poll into a vendor contest.
Related:
- Top 10 Technology Trends for 2020
- Automated Machine Learning: How do teams work together on an AutoML project?
- AutoML for Temporal Relational Data: A New Frontier