Image by Author
 

A data science project has many elements to it. There are many people involved in the process, and many challenges are faced along the way. A lot of companies see the need for data science, and it has been implemented in our lives today. However, some struggle with how to make use of their data analytics and which path to use to get there. 

The biggest assumption that companies make when using data science, is to imply that due to their use of programming language, it imitates the same methodology as software engineering. However, the models' built-in data science and software are different. 

Data science requires its unique lifecycle and methodologies to be successful. 

 

Data Science Lifecycle

 

The data science lifecycle can be broken up into 7 steps. 

 

Business Understanding

 

If you are producing anything for a company, your number 1 question should be ‘Why?’. Why do we need to do this? Why is it important to the business? Why? Why? Why?

The data science team is responsible for building a model and producing data analytics based on what the business requires. During this phase of the data science lifecycle, the data science team and executives of the company should be identifying the central objectives of the project, for example looking into the variables that need to be predicted. 

What kind of data science project is this based on? Is it a regression or classification task, clustering, or anomaly detection? Once you understand the overall objective of your object, you can keep on asking why, what, where, when and how! Asking the right questions is an art, and will provide the data science team with in-depth context to the project. 

 

Data Mining

 

Once you have all the business understanding that you require for the project, your next step will be initiating the project by gathering data. The data mining phase includes gathering data from a variety of sources that are in line with your project objective. 

The questions that you will be asking during this phase are: What data do I require for this project? Where can I get this data from? Will this data help fulfill my objective? Where will I store this data? 

 

Data Cleaning

 

Some data scientists choose to blend the data mining and data cleaning phases together. However, it is good to distinguish the phases for better workflow. 

Data cleaning is the most time-consuming phase in the data science workflow. The bigger your data, the longer it takes. It can typically take up to 50-80% of a data scientist's time to complete. The reason it takes so long is because data is never clean. You can be dealing with data that has inconsistencies, missing data,  incorrect labels, spelling mistakes, and more. 

Before performing any analytical work, you will need to correct these errors to ensure that the data you plan to work with is correct and will produce accurate outputs. 

 

Data Exploration

 

After a lot of time and energy spent cleaning the data, you now have squeaky-clean data that you can work with. Data exploration time! This phase is the brainstorming of your overall project objective. You want to dive deep into what you can find from the data, hidden patterns, creating visualizations to find further insights and more. 

With this information, you will be able to create a hypothesis that is in line with your business objective and use it as a reference point to ensure you are on task. 

 

Feature Engineering

 

Feature engineering is the development and construction of new data features from raw data. You take the raw data and create informative features that are in line with your business objective. The feature engineering phase consists of feature selection and feature construction.

Feature selection is when you cut down the number of features you have which add more noise to the data than actual valuable information. Having way too many features can lead to a curse of dimensionality, an increased complexity in the data for the model to easily and effectively learn from. 

Feature construction is in the name. It is the construction of new features. Using the features you currently have, you can create new features, for example, if your objective is concentrated on senior members, you can create a threshold for the age you want.

This phase is very important as it will influence the accuracy of your predictive model. 

 

Predictive Modeling

 

This is where the fun starts, and you will see if you’ve met your business objective. Predictive modeling consists of training the data, testing it, and using comprehensive statistical methods to ensure that the outcomes from the model are significant to the hypothesis created. 

Based on all the questions you asked in the ‘Business Understanding’ phase, you will be able to determine which model is right for your task at hand. Your choice of model may be a trial and error process, but this is important to ensure that you create a successful model that produces accurate outputs. 

Once you have built your model, you will want to train it on your dataset and evaluate its performance. You can use different evaluation metrics such as k-fold cross-validation to measure the accuracy and continue to do this till you are happy with your accuracy value. 

Testing your model using testing and validation data ensures accuracy and that your model performs well. Feeding your data with unseen data is a good way to see how the model performs with data that it hasn’t been trained on before. It puts your model to work!

 

Data Visualisation

 

Once you are happy with your model's performance, you are ready to go back and explain it all to the executives in the company. Creating data visualizations is a good way to explain your findings to people who are not technical, and is also a good way to tell a story about the data.

Data visualization is a combination of communication, statistics, and art. There are so many ways that you can present your data findings in an aesthetically pleasing way. You can use tools such as Matplotlib Documentation, Seaborn Tutorial, and Plotly Library. If you are using Python, have a read of this: Make Amazing Visualizations with Python Graph Gallery. 

And just like that you’re at the end of the life cycle, but remember it’s a cycle. So you have to go back to the start: Business Understanding. You will need to evaluate the success of your model regarding the original business understanding and objective, along with the hypothesis created.

 

Data Science Project Management Methodologies

 

Now we have gone through the data science lifecycle, you must be thinking this seems very simple. It’s just one step after the other. But we all know things aren’t that straightforward. In order to make it as simple and effective as possible, management methodologies need to be put in place. 

Data science projects are not solely under the data scientists' responsibility anymore - it is a team effort. Therefore, standardizing project management is imperative, and there are methods that you can use to ensure this. Let’s look into them.

 

Waterfall Methodology

 

Just like a waterfall, the waterfall methodology is a sequential development process that flows through all the stages of a project. Each phase will need to be completed in order for the next phase to begin. There is no overlap between phases, making it an effective method as there are no clashes. If you have to revisit the previous phases, it means that the team has planned poorly. 

It is made up of five phases:

1. Requirements
2. Design
3. Implementation
4. Verification (Testing)
5. Maintenance (Deployment)

So when should you use the waterfall methodology? As it flows like water, everything needs to be clear. This means that the objective is defined, the team knows the technology stack inside out, and the project elements are all in place to ensure a smooth and effective process. 

But let’s come back to reality. Are data science projects easily flowing like water? No. They require a lot of experimentation, requirement changes, and more. However, that does not mean that you cannot use elements of the waterfall methodology. Waterfall methodology requires a lot of planning. If you plan everything, yes you may still come across 1 or 2 problems on the way, but the challenges will be less and not as harsh on the process. 

 

Agile Methodology

 

The Agile methodology was born in early 2001 when 17 people came together to discuss the future of software development. It was founded on 4 core values and 12 principles.

The agile methodology is more in line with today's technology, as it works in a fast-paced, ever-changing technology industry. If you are a tech professional, you know that the requirements in a data science or software project change all the time. Therefore, having the right method in place which allows you to quickly adapt to these changes is important.

The agile methodology is a perfect data science project management method as it allows the team to continuously review the requirements of the project as it grows. Executives and data science managers can make decisions about changes that need to be made during the development process, rather than at the end once it's all complete. 

This has shown to be highly effective as the model evolves to reflect user-focused outputs, saving time, money and energy. 

An example of an agile method is Scrum. The scrum method uses a framework that helps to create structure in a team using a set of values, principles, and practices. For example, using Scrum, a data science project can break up its larger project into a series of smaller projects. Each of these mini-projects will be called a sprint and will consist of sprint planning to define objectives, requirements, responsibilities and more. 

 

Hybrid Methodology

 

Why not use two different methods together? This is called a hybrid method, where two or more methodologies are used to create a method that is entirely unique to the business. Companies can use hybrid methods for all types of projects, however, the reasoning behind it is down to product delivery. 

For example, if a customer requires a product but is not happy with the timeframe of production based on using sprints in an Agile method. So it seems like the company needs to do a bit more planning right? What method has a lot of planning? Yes, that’s right, Waterfall. The company can adopt waterfall into their method to cater specifically for the customer's requirement. 

Some companies may have mixed emotions about combining an agile method with a non-agile method such as Waterfall. These two methods can co-exist, however, it is the company's responsibility to ensure a simple approach that makes sense, measure the success of the hybrid method, and provide productivity. 

 

Research and Development

 

Some may consider this as a methodology, however, I believe that this is an important foundation for the data science project process. Just like the waterfall methodology, there is no harm in planning and preparing yourself with as much information as possible.

But that’s not what I am talking about here. Yes, it’s great to research everything before you start a project. But a good way to ensure effective project management is to see your project as a research and development project. It is an effective tool for data science team collaboration.

You want to walk before you run and operate your data science project like it is a research paper. Some data science projects have harsh deadlines which make this process difficult, however, rushing your end product always comes with further challenges. You want to build an effective and successful model that meets your initial data science lifecycle phase: Business Understanding. 

Research and development in a data science project keeps the doors open to innovation, increases creativity and does not limit the team to settle with something that could be much greater!

 

Wrapping it up

 

Although there are different methodologies to choose from, ultimately it comes down to the operations of the business. Some methods that are popular in one company, may not be the best approach for another company. 

Individuals may have different ways of working, so the best approach is to create a method that works for everyone. 

Want to learn about automating your data science workflow, have a read of this: Automation in Data Science Workflows.
 
 
Nisha Arya is a Data Scientist, Freelance Technical Writer and Community Manager at KDnuggets. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.