Photo by Anete Lusina from Pexels

When I first started learning data science, I assumed that landing a job in the field meant that the hard part was over. After a few years of working in the industry, however, I have come to realize that I couldn’t have been more wrong.

Many data scientists I know have left their jobs in just months after landing the position. I quit a data science internship one week after I joined, since I felt as though the tasks I was assigned had nothing to do with all the skills I’d painstakingly learnt.

After speaking to co-workers in the data industry who like me, had left their jobs at a very early stage in their career, I’ve come to realize that there are two main reasons the data science field has such a high employee attrition rate:

Reason #1: Mismatch in Employer Expectations

Imagine this:

 
You spend thousands of hours learning statistics and the nuances of different machine learning algorithms. Then, you apply to dozens of different data science job listings, go through extensive interview processes, and finally get hired by a mid-sized organization. 

You’re excited to finally start working on real-world machine learning problems, and put all the skills you’ve acquired over the years in practice. However, on your first day at the job, you realize that the company has massive amounts of unstructured data entering the system that hasn’t been formatted or processed in any way.

Your employer sees you as the go to “data guy,” and puts you in charge of helping him increase sales with the help of large amounts of data they’re collecting everyday.

At the end of the day, you’re no longer building complex algorithms and predictive models like you imagined. You now spend all your time brushing up on SQL and data preparation skills to pull data out of the system into different formats, and present this data to stakeholders so they can use it to make business decisions.

Although your job title has the word “data science” in it, you aren’t in a role you’ve always pictured yourself in. You’re unhappy being the company’s data janitor, and want to work on projects that will actually utilize the skills you’ve spent so long to gain. 

Eventually, you only have two options left — stay in the company for a couple of years and continue performing tasks you don’t enjoy, or leave and find an organization with projects that are better aligned to your goals.

Here’s the problem:

 
The scenario above might sound unlikely to you, but it is one of the biggest complaints I’ve heard from data scientists around me. Many of them had very different expectations of what their job scope would entail, but ended up performing data reporting and analysis tasks on a daily basis.

After staying in companies like these for some time, many data scientists end up losing touch of their machine learning skills, since they haven’t been working on real-world ML projects for years. 

When looking for another job, they aren’t able to apply to mid or senior-level data science job listings, since they simply don’t have the required expertise. These individuals generally end up having to make a career pivot, and go on to become data or reporting analysts.

Reason #2: Inability to Add Business Value

 
Another common reason behind the frustration of most data scientists is the inability to add business value with their machine learning models.

In my opinion, this issue is experienced even more frequently than the previous one — since it is also faced by organizations that have a well-defined job scope and a suitable data pipeline in place.

Here are a few reasons data scientists fail to build models that add value to organizations:

The age-old gap between tech and business:

 
Stakeholders and upper-management are fairly non-technical, and aren’t always aware of what’s possible with machine learning modelling. There is so much hype around the field, that as a data scientist, you will hear some fairly ambitious requests from your managers.

It is up to you to explain to them whether a project can be successfully completed, and whether it really will bring the results they expect to see. Make sure that expectations are aligned with the potential outcome, so there won’t be much disappointment later on.

It might also be useful to create a cost-benefit report before working on any machine learning project, so that the company can collectively decide whether it is worth assigning time and resources to.

Not asking the right questions:

 
As a data scientist, you need to know if the models you’re building will add value to the business. 

Most data scientists I’ve seen are quick to start a project based on the instructions they’ve been given. They don’t ask the right questions. They don’t try to understand their manager’s train of thought.

When you do something simply because someone else tells you to, then you don’t have any insight into the value you bring to the table. If you are asked to explain why your work is useful, you will be unable to do so. 

How can you convince someone that your product works if you don’t know why you’re building it in the first place?

Lack of domain knowledge:

 
In order to ask the right questions, you need to know how the business works. 

The models you build should be tailored to a domain specific problem, and you need to understand the impact it will have on the end-user.

For example, if you’re building a model for a clothing company, you need to keep in mind that factors like seasonality will have an impact on the recommendations you provide consumers. 

I work in marketing, and most of this domain knowledge is acquired by working directly with the business team. Some of it comes from online courses. And a lot of it, of course, is based on my daily interaction with people and a basic understanding of how they act.

Depending on the field you work in, it is wise to take some time out to gain industry specific knowledge. This will be used in every step of your data science workflow — pre-processing, feature selection, feature weightage, and deciding on the occasional changes to be made even after model deployment.

So... How to be a data scientist who doesn’t hate their job?

 
Firstly, it is important to pick a company that will allow you to work on what you like. Steer clear of companies that have every tool stack listed on their job description. Before you apply, find these companies on LinkedIn and look at whether they’ve hired for a data science position before. If they haven’t, you might want to stay away, since it means that you’ll likely be made to perform every data related task there is.

If they have, find the profiles of these data scientists and check the description they entered for the role. See if it aligns with your expectations.

Make sure your interview process isn’t one-sided. Ask the interviewer as many questions as you can about the job scope and what it entails. If it doesn’t align with your expectations, then it might be better for you to look elsewhere.

Finally, make sure to spend some time gaining domain knowledge in the field you’re working in. Use this knowledge to ask your managers the right questions, and ensure that their expectations are aligned to the potential project outcome.

 
 
Natassha Selvaraj is a self-taught data scientist with a passion for writing. You can connect with her on LinkedIn.