Image created by Freepik

Introduction

 
MLOps is a relatively new term to the data industry. In the past, companies solely focused on hiring data scientists and machine learning practitioners. These individuals could build predictive models that helped companies automate workflows and make key decisions.

Over time, however, machine learning projects started to cause organizations more harm than good. They failed when put into production, leading to missed business opportunities and unhappy clients.

Why did this happen?

On paper, these companies did everything right. They hired a team of experienced data scientists who possessed the ability to build technically sound models. They also had domain experts who outlined the business use-case clearly, and were able to successfully translate a problem statement into a machine learning project.

The models built as a result of these initiatives performed impeccably in a test environment, but failed when rendered to an end-user. This was an outcome that nobody could have predicted, and ended up costing organizations millions of dollars.

Here’s the problem:

The focus of data scientists lies solely in model building. Once these machine learning models reach the hands of an end-user, there is no proper system in place to ensure that they are performing well in the real-world. Models can act very differently when placed in an environment that they weren’t trained in.

Since machine learning was so new to most organizations, these factors weren’t taken into consideration in the past, which led to disastrous consequences and poor performance. 

Here are a few examples of real-world machine learning projects gone wrong due to the lack of continuous model monitoring and maintenance:

Feedback loop 

Predpol is a company that aims to predict poverty crime using statistical analysis. The software works out when and where a crime will take place, and sends officers to the area to make arrests. The algorithm would then re-train on this data to make future crime predictions. 

Ironically, the process above ended up creating a feedback loop. Every time an arrest was made, the model was updated and sent more law enforcement officers to the same area. This in turn, led to a higher number of arrests in specific places, which was again fed to the model. The Predpol system ended up predicting higher crime rates in areas that had more police officers rather than regions with a larger number of criminals.

This is an example of a failed machine learning project. Model predictions were leaked into training data, which was then used to further enhance the algorithm. 

Data Drift 

 
The real-world is unpredictable and continuously changing. Due to this, a machine learning model that performs well one day can be highly inaccurate the next. 

An example of this can be seen in a company that works to build machine learning models for healthcare. This organization deployed an algorithm to predict 30-day hospital readmissions. 

After around 2–3 months, an initially effective model started making highly inaccurate predictions. Every hospital the model was deployed to faced the same issue, and the algorithm’s poor performance made clients unhappy.

A team was assigned to look into this, and they realized that model performance faltered every time there was a change in the training database. For example, if the hospital made changes in the insurance they accepted, this had a direct impact on the type of patients who went to the ER, which in turn negatively affected the model’s accuracy.

After realizing the issue at hand, they immediately worked to remediate the situation. An entire team of technical and business savvy individuals were assigned to monitor changes in incoming data and ensure that these changes were reflected in the predictive algorithm. 

This phenomenon of change in input data is called data drift, and is one of the most common causes of model degradation in companies.

For example, if you worked at an organization with a primary audience demographic of 18–34 year old women and trained your model on their behavioural attributes, you introduce a bias into the algorithm.

If the company launches a new product line catered towards an older group of individuals, the model will no longer be relevant. It needs to be re-trained on data that accurately represents the company’s current user behaviour. 

Data drift needs to be detected quickly, and model updates should be done as fast as possible to ensure that companies face minimal loss.

Seasonality

 
Seasonality is a characteristic in which data experiences regular and predictable changes at specific intervals of time. Machine learning models need to be updated regularly with seasonal changes in mind.

For example, if you were to build a recommender system for Starbucks using from the last 3 months, your algorithm is going to make predictions based on beverages that customers enjoyed during that particular season. 

In the next few months, once the climate changes, so will most user’s coffee drinking habits. If this isn’t captured in the model you build, you are likely to try and entice customers with an unfavourable choice of beverages, which will incur a loss to the company. 

The Rise Of MLOps Engineers

 
Above are just a few examples of factors that if not taken into consideration, can have a huge negative impact on the performance of productionized algorithms.
As organizations started losing time and money over ineffective machine learning projects, they investigated the issue. Why were they not seeing results despite hiring highly capable data scientists?

Once employers understood that the issue lay in the lack of proper model deployment procedures, they realized that there was a need for an entirely new role — a person who possessed machine learning and operational skills, who would be able to handle the workflow that took place post model-building.

This position was called MLOps (Machine Learning Operations). 

Skills Required Of An MLOps Engineer

 
An MLOps engineer is responsible for model deployment and continuous maintenance. 

If you want to become an MLOps engineer, you need to have knowledge of machine learning algorithms. You will be working to refactor other data scientist’s code to make it production ready, and should be able to understand their work.

Apart from machine learning skills, you also need to have a foundational knowledge of DevOps. DevOps is a role that integrates the job scope of software developers and operations teams to automate workflows. An MLOps engineer’s role is very similar to the profession of a DevOps engineer, except that the former works with machine learning models.

You will need to learn DevOps concepts such as automating workflows using CI/CD pipelines. CI and CD stand for Continuous Integration and Continuous Deployment, and enable code changes made by the data scientist to be delivered quickly and reliably into production.

The MLOps Engineer Role — Next Steps

 
While most data enthusiasts tend to focus solely on developing their machine learning and data science skills, becoming a data scientist isn’t your only career option in the industry.

MLOps is rapidly growing, and the market for solutions in this field is projected to rise to $4B by 2025.

Also, the MLOps domain contains a large number of career opportunities, as companies are currently facing a shortage of employees who possess the combined skillset of data scientists and DevOps engineers. 

If you are looking to make a job transition and are currently weighing your options, it might be a good idea for you to consider pursuing a career in MLOps, as it is a fairly unsaturated field with an amazing opportunity for growth.

 
 
Natassha Selvaraj is a self-taught data scientist with a passion for writing. You can connect with her on LinkedIn.