Why So Many Data Scientists Quit Good Jobs at Great Companies

By Adam Sroka, Head of Machine Learning Engineering at Origami.

Driven to tears — when dream jobs turn bad (photo by Andrea Bertozzini on Unsplash).

 

Are you happy with your job? Not happy? Either way, take part in a short

KDnuggets Job Satisfaction Poll

Dream job?

 

We’re often bombarded with how popular data science is as a career. It’s all too common to read things like data science being the “Sexiest Job of the 21st Century” or yearly comparisons of high salary expectations.

Data science has a lot to offer. It’s a challenging role with plenty to learn and keep you occupied. Compared to many other roles, data scientists can be given a lot of autonomy to explore and solve interesting problems. And, in many cases, you’ll get the opportunity to work with talented and skilled people in a variety of domains.

Despite this, many data scientists spend several hours a week looking for new roles, according to this Kaggle survey. In fact, those working in machine learning regularly top lists of developers looking for new roles — 20.5% in the 2020 Developer Survey from Stack Overflow, second only to academics!

If data science is such a dream job, it begs the question…

Why are so many data scientists looking for another job?

Now I’ve been there and, hopefully, my experience can shed some light on the situation.

I was a data scientist for several years before taking the leap up to a director position at a startup. I now work in more of a managerial/leadership role. Having both been a data scientist and managing teams of them (as well as broader developer and data teams), I have an uncommon perspective from both sides of the fence.

In my time as a data scientist, I felt the pain too. I’ve worked mostly in startups and jumped ships a couple of times early in my career. The reasons were combinations of several factors, but with many elements that I have heard across other companies.

What I’ll aim to do here is outline some of the more general reasons below and what I think can help the situation. This advice will be useful for:

• Data scientists that are currently dissatisfied and unsure of leaving
• Managers and organisations that can’t seem to hang on to good data science talent
• Aspiring professionals considering stepping into the role that may be unaware (it’s a big investment if any of these are red flags for you)

Before we begin, it must be said I still love data science as a role. It can be a very rewarding career — if you know how to get the most out of it.

 

Reality vs Expectations

 

Sometimes working in a business environment isn’t quite what one expects (photo by Sebastian Herrmann on Unsplash).

Getting to work with cutting-edge technology, solving difficult problems in interesting ways, using new algorithms, and developing machine learning solutions that’ll have a big impact on your organisation — data science sounds great!

For many, however, this is too good to be true.

I’ve experienced first-hand and heard from many people in the industry that, all too often, the reality does not match expectations. I’d go so far as to say this is the number one reason data scientists get frustrated and ultimately leave.

Now, there are many reasons why this might be the case. We should remember, too, that this is two-sided.

Unrealistic expectations

Many data scientists early in their careers simply don’t have the experience of working in real organisations. Much in the same way that social media projects an unrealistic view of other people’s lives — it’s easy to take in all the exciting stories and assume that’s the norm.

I’ve found this can be common for those that have come fresh out of education or from a research position in academia. It’s easy to fall into an infinite timescale, infinite budget mindset. All too often, I’ve heard protests from data scientists saying they can’t put a timeline on when their work will be finished, and it’ll take as long as it takes. This simply isn’t true and won’t fit well with the culture at most organisations.

You either fix the scope of what you’re trying to achieve and vary the timescales to fit, or fix the timescale and vary the scope.

The other major contributing factor is the crushing realisation that a great deal of the job isn’t terribly exciting. In most organisations, you’ll have to spread your time between doing technical work and the other, less exciting stuff. This can be an issue if you’re not keen on reporting, writing, and delivering presentations, repeatedly explaining the basics of your models or approaches, project management, and administrative overhead, or trying to get buy-in from other parts of the organisation.

Harsh reality

It’s also very common that most of the infrastructure and data processing you might expect to be in place simply doesn’t exist.

I previously worked at a start-up as data scientist number two. My colleague who’d been there 18 months had spent the entire time building some fundamental data pipelines. Luckily for me, they’d been through all of the pain of convincing the relevant people to approve budgets, working through the security and IT headaches that come with adopting new cloud technology, explained thousands of times what the point of it all was.

In some cases, you’ll be utilised as a clever technical person that can deal well with loose requirements to get stuff done. The fact that you can also do data science might just come second.

These issues can be made worse when coupled with a lack of experienced data scientists in the team or a lack of experience in managing data scientists among the management of an organisation. If you’re the only person, it can be difficult to get your arguments across in a way that resonates.

This is a recipe that often leads to unhappy working environments and unfulfilled expectations.

As a data scientist, you may join thinking you’re there to build smart models and derive as much value from the data as possible. You get held up as your first few months require you to build the necessary infrastructure and pipelines to even get the data.

Senior stakeholders in the company then see a lot of time go by with little result. In reality, they’d be quite satisfied with getting some simple charting for the regular board meeting. They start to see an expensive resource not delivering value quickly enough.

This disconnect can ultimately frustrate both parties.

If you get the chance, ask questions around this at the interview stage:

• Who’s bought into data science at the most senior level of the organisation?
• Do they have experience with it, or are they hiring you because of the hype?
• How many other people are in the data team?
• Are there data engineers/analysts/DevOps engineers, or will you be expected to do all of that yourself?

Now, this might all seem doom and gloom — it isn’t. Many organisations do just fine here, but it’s a balance between managing your expectations and landing in a position that has the right backing and is set up for success.

 

The Wonderful World of Politics

 

Sometimes you’ll feel like a politician (photo by Deniz Fuchidzhiev on Unsplash).

Office politics. Oh, dear.

I’ve heard time and again about talented teams with good management completely starve, dwindle, and die because of politics. I’ve heard a first-hand account when the only senior leader in an organisation that is bought into data science is forced out, and the team is swiftly repurposed to mundane work that doesn’t make the best use of their skills.

Unfortunately, politics are a part of many careers. Now, you don’t have to ‘play the game’ as it were — you can always take your rare and in-demand skillset and go elsewhere.

If you’re not familiar with the Exit, Voice, Loyalty, Neglect model, I’d urge you to take a read. Stemming from the work of Albert Hirschman, it outlines an abstract model for how individuals respond to unacceptable situations. Originally published in 1970, the book has been discussed at length and expanded upon.

When things aren’t right, it boils down to four options:

• Exit — you quit the job and find another, leaving the company in a worse state as the problem will persist, and it’s lost your skills and experience.
• Persist — you ride it out and see if it gets better. This, in time, often leads to one of the other options if things don’t right themselves before your patience runs out.
• Neglect — you rebel against what’s occurring by neglecting your responsibilities, which will either result in your getting an easy ride for a while or getting fired.
• Voice — you take a stand and try to make a change.

Of the four options, Voice is the only one in which you actively try to improve things. In this case, that means navigating office politics.

In many circumstances, the political situation in your organisation might seem well above your pay grade. This can be very uncomfortable as you may feel powerless to affect the huge budget cuts or sweeping changes. This might be a good time to weigh up your options, but sending a well-written communication to someone in the leadership might actually be the catalyst for real change.

If you’re in a smaller organisation where the decision-makers are more reachable, I’d strongly urge you to try and build a relationship with them. Contrary to what many people might think — people generally want to do right by the organisation and the people in it. It’s rare that a company hires someone genuinely evil and out to get you.

Often senior stakeholders might not have had the exposure to understand the needs of a data science team. Taking the time to show them how you can add value and building a strong relationship with them will help them get the most value from your skills. It may also help you better understand what the real concerns of the business are at the highest levels.

In talks I do for early career data scientists, I jokingly give the advice to automate some of the CFO or Finance Director’s workflow as early as you can in your new job. This will demonstrate your value directly to the person holding the budget and win you an ally. This is only a half-joke as, to be honest, these people are usually some of the busiest in any company and are often trapped in Excel hell.

People in the business with influence need to have a good opinion of you. Most of them won’t care one bit how much you know about algorithms or statistics. You’ll win favour by doing mundane tasks and basic data retrieval, automation, or reporting work for them. If you can do this with a smile and build a strong reputation, you’ll find things get easier for you in the long run.

 

Data Science == Data EVERYTHING

 

It can get overwhelming when you’re the go-to person for all things data (photo by Usman Yousaf on Unsplash).

If you manage to navigate office politics, there’s a strong chance you’re going to build a good reputation. This can be a double-edged sword.

Many people won’t understand (or care about) what it means to be a data scientist. As mentioned earlier, you’ll be seen as a clever techie person that can get stuff done. Having access to all the data and being armed with a broad arsenal of technical tools, you can quickly become the go-to person for fixing their issues.

If you can handle this, then great, but it can become a burden, and when people start to rely on you and apply pressure, it can get uncomfortable. You might quickly find yourself spending 80% of your time doing something an early-career DBA might be better suited to do.

I often tell organisations that data scientists can do anything, usually slower and more expensively than everyone else — but the stress is on the anything.

Having a wide skillset and a loosely defined role can be fun but don’t fall into the trap of doing work better suited to other jobs just because your organisation isn’t aware. Reach out to the senior stakeholders and offer to help hire that DBA or BI developer that’d love to be doing the work that’s keeping you away from what you really want.

This can also help tackle the problem of isolation. If you’re in a small isolated team of data scientists, your expertise in all things data can actually keep you isolated. Data becomes your domain, and people don’t want to take ownership. Helping your organisation better structure itself and expand its data roles can better integrate you with the wider team.

 

Final word

 

Unfortunately, just knowing all the latest tools and algorithms won’t be enough to get the most out of the majority of data science roles. If you go in with a level set of expectations and an understanding that you might need to educate your organisation a little, you’re likely to succeed.

Original. Reposted with permission.

 

Bio: Dr Adam Sroka, Head of Machine Learning Engineering at Origami Energy, is an experienced data and AI leader helping organisations unlock value from data by delivering enterprise-scale solutions and building high-performing data and analytics teams from the ground up. Adam shares his thoughts and ideas through public speaking, tech community events, on his blog, and in his podcast.

Related:

• 15 Habits I Learned from Highly Effective Data Scientists
• Who is fit to lead data science?
• Want to Be a Data Scientist? Don’t Start With Machine Learning