When Will Demand for Data Scientists/Machine Learning Experts Peak?
We analyze the results of Data Science / Machine Learning peak demand poll, examine the split between optimists and pessimists, and try to explain why predictions look so similar regardless of experience, affiliation, and region?
Latest KDnuggets Poll asked
The results, based on nearly 1,200 voters, are that KDnuggets readers expect this to happen between 4 and 10 years from now. The most common answer was 4-6 years and the median answer was in 8-9 year range - see Fig. 1.
Fig. 1: KDnuggets Poll: When Demand for Data Scientists/Machine Learning Experts will begin to decline?
This is a very interesting and important chart, and here are my observations.
Note: Beginning of decline starts right after peak demand, so we use these 2 terms interchangeably.
The demand for Data Scientists has began to grow rapidly soon after it was proclaimed the sexiest job of 21st century in 2012. This demand is being satisfied in 2 ways:
The current demand has a feel of a bubble and at some point the bubble will burst and demand will begin to decline, but as with many bubbles, it is hard to predict when. Peak demand (start of decline) somewhere between 4 and 9 years seems reasonable to me. This does not mean, of course, that Data Scientists will become unemployed, but the job will become a regular one, and salaries will stabilize, as happened, for example, with web developers - it was a very hot job at the beginning of the internet, but is a regular job now.
The poll also asked about voter experience in Data Science and Machine Learning and their affiliation (Industry / Student / Research / Govt / Other) - see Fig. 2.
Fig. 2: Poll respondents Experience in Data Science/Machine Learning and Affiliation
We note that the about 60% of respondents had 2 years experience or less, showing how young is this field now. 62% of respondents were from industry, 19.5% - students, 9.2% - researchers, and 4.5% - government/non-profit.
When I examined the peak demand vs experience and affiliation, surprisingly I got similar curves, with one peak at 4-6 or 7-10 years and a second, smaller peak at 25 years or more. To simplify the chart, I grouped the lines for "none" and "<1 year" experience, and for "1-2" and "2-4" years, which in both cases looked very similar.
Fig. 3: Peak Demand for Data Scientists/Machine Learning Experts vs Experience in Data Science/Machine Learning
Let me call people who expects the demand for Data Scientists to start declining only in 25 years or more "optimists", and the other group - "pessimists".
We note that the group with the most experience (over 16 years) had the highest share of optimists - over 35% from that group thought that demand will not decline for the next 25 years.
Next, we examined the peak demand curve vs Affiliation - Fig. 4.
Fig. 4: Peak Demand for Data Scientists/Machine Learning Experts vs Affiliation
Again, the most surprising finding is that the curves again are similar.
Students are the most optimistic, while government people are the most pessimistic, with industry in the middle.
Finally, we examine the distribution by regions. Regional participation was:
Fig. 5: Peak Demand for Data Scientists/Machine Learning Experts vs Region
Again, the most surprising finding is that the curves again are similar. The one notable difference is that US/Canada respondents were more optimistic - their most common choice was peak demand in 7-10 years, while in other regions it was 4-6 years. The percent of people saying 25 years or more was amazingly similar (18.3% to 19.8%) in all regions.
What can explain such a similar distribution?
One hypothesis is that people by nature are frequently optimists or pessimists, and that affects their predictions. Perhaps when asking about predictions about things like social trends that cannot be determined from physics but depend on people's feelings and intuition, polls should somehow measure the trait of optimism vs pessimism.
If this is true, we note that percentage of optimists is relatively similar across different affiliations and regions, and does not change much with experience.
Related:
When will Demand for Data Scientists/Machine Learning experts will begin to decline?
The results, based on nearly 1,200 voters, are that KDnuggets readers expect this to happen between 4 and 10 years from now. The most common answer was 4-6 years and the median answer was in 8-9 year range - see Fig. 1.
Fig. 1: KDnuggets Poll: When Demand for Data Scientists/Machine Learning Experts will begin to decline?
This is a very interesting and important chart, and here are my observations.
Note: Beginning of decline starts right after peak demand, so we use these 2 terms interchangeably.
The demand for Data Scientists has began to grow rapidly soon after it was proclaimed the sexiest job of 21st century in 2012. This demand is being satisfied in 2 ways:
- Education: many universities and other educational platforms like Coursera, edX, and Udemy stepping up to offer Education in Data Science and Machine Learning
- Automation: companies like DataRobot and others offer Automated Data Science/Machine Learning platforms that either let business users do modeling, or fully automate Data Science by being embedded in a business process (eg automated bidding on ads).
For a less serious look at the automation, see KDnuggets cartoon Data Scientist - the sexiest job of the 21st century until ...
The current demand has a feel of a bubble and at some point the bubble will burst and demand will begin to decline, but as with many bubbles, it is hard to predict when. Peak demand (start of decline) somewhere between 4 and 9 years seems reasonable to me. This does not mean, of course, that Data Scientists will become unemployed, but the job will become a regular one, and salaries will stabilize, as happened, for example, with web developers - it was a very hot job at the beginning of the internet, but is a regular job now.
The poll also asked about voter experience in Data Science and Machine Learning and their affiliation (Industry / Student / Research / Govt / Other) - see Fig. 2.
Fig. 2: Poll respondents Experience in Data Science/Machine Learning and Affiliation
We note that the about 60% of respondents had 2 years experience or less, showing how young is this field now. 62% of respondents were from industry, 19.5% - students, 9.2% - researchers, and 4.5% - government/non-profit.
When I examined the peak demand vs experience and affiliation, surprisingly I got similar curves, with one peak at 4-6 or 7-10 years and a second, smaller peak at 25 years or more. To simplify the chart, I grouped the lines for "none" and "<1 year" experience, and for "1-2" and "2-4" years, which in both cases looked very similar.
Fig. 3: Peak Demand for Data Scientists/Machine Learning Experts vs Experience in Data Science/Machine Learning
Let me call people who expects the demand for Data Scientists to start declining only in 25 years or more "optimists", and the other group - "pessimists".
We note that the group with the most experience (over 16 years) had the highest share of optimists - over 35% from that group thought that demand will not decline for the next 25 years.
Next, we examined the peak demand curve vs Affiliation - Fig. 4.
Fig. 4: Peak Demand for Data Scientists/Machine Learning Experts vs Affiliation
Again, the most surprising finding is that the curves again are similar.
Students are the most optimistic, while government people are the most pessimistic, with industry in the middle.
Finally, we examine the distribution by regions. Regional participation was:
- US/Canada, 40%
- Europe, 33%
- Asia, 16%
- other, 11%
Fig. 5: Peak Demand for Data Scientists/Machine Learning Experts vs Region
Again, the most surprising finding is that the curves again are similar. The one notable difference is that US/Canada respondents were more optimistic - their most common choice was peak demand in 7-10 years, while in other regions it was 4-6 years. The percent of people saying 25 years or more was amazingly similar (18.3% to 19.8%) in all regions.
What can explain such a similar distribution?
One hypothesis is that people by nature are frequently optimists or pessimists, and that affects their predictions. Perhaps when asking about predictions about things like social trends that cannot be determined from physics but depend on people's feelings and intuition, polls should somehow measure the trait of optimism vs pessimism.
If this is true, we note that percentage of optimists is relatively similar across different affiliations and regions, and does not change much with experience.
Related:
- Optimism about AI improving society is high, but drops with experience developing AI systems
- How to Job Interview a Data Scientist
- Getting Started with Machine Learning in One Hour!