- Building Massively Scalable Machine Learning Pipelines with Microsoft Synapse ML - Nov 30, 2021.
The new platform provides a single API to abstract dozens of ML frameworks and databases.
Machine Learning, Microsoft, Pipeline, Scalability
- Build a Serverless News Data Pipeline using ML on AWS Cloud - Nov 18, 2021.
This is the guide on how to build a serverless data pipeline on AWS with a Machine Learning model deployed as a Sagemaker endpoint.
AWS, NLP, Pipeline, Python, Sagemaker, Text Summarization
- KDnuggets™ News 21:n44, Nov 17: Don’t Waste Time Building Your Data Science Network; 19 Data Science Project Ideas for Beginners - Nov 17, 2021.
Don’t Waste Time Building Your Data Science Network; 19 Data Science Project Ideas for Beginners; How I Redesigned over 100 ETL into ELT Data Pipelines; Anecdotes from 11 Role Models in Machine Learning; The Ultimate Guide To Different Word Embedding Techniques In NLP
Beginners, Data Science, ETL, Machine Learning, NLP, Pipeline, Project, Word Embeddings
How I Redesigned over 100 ETL into ELT Data Pipelines - Nov 15, 2021.
Learn how to level up your Data Pipelines!
ELT, ETL, Pipeline, SQL
Design Patterns for Machine Learning Pipelines - Nov 2, 2021.
ML pipeline design has undergone several evolutions in the past decade with advances in memory and processor performance, storage systems, and the increasing scale of data sets. We describe how these design patterns changed, what processes they went through, and their future direction.
Data Preprocessing, ETL, Machine Learning, Pipeline
- ETL and ELT: A Guide and Market Analysis - Oct 29, 2021.
ETL and related techniques remain a powerful and foundational tool in the data industry. We explain what ETL is and how ETL and ELT processes have evolved over the years, with a close eye toward how third-generation ETL tools are about to disrupt standard data processing practices.
Data Preparation, ELT, ETL, Market Research, Pipeline
- Four Different Pipes for R with magrittr - Oct 6, 2021.
The magrittr package supplies the pipe operator (%>%), but it turns out that the package actually contains four pipe operators in total. Let's go into them a bit.
Pipeline, R, Tidyverse
- Adventures in MLOps with Github Actions, Iterative.ai, Label Studio and NBDEV - Sep 16, 2021.
This article documents the authors' experience building their custom MLOps approach.
GitHub, Machine Learning, MLOps, Pipeline, Python, Workflow
- The Prefect Way to Automate & Orchestrate Data Pipelines - Sep 13, 2021.
I am migrating all my ETL work from Airflow to this super-cool framework.
Airflow, Data Workflow, Pipeline, Prefect, Python
- Build a synthetic data pipeline using Gretel and Apache Airflow - Sep 2, 2021.
In this blog post, we build an ETL pipeline that generates synthetic data from a PostgreSQL database using Gretel’s Synthetic Data APIs and Apache Airflow.
Airflow, Pipeline, Postgres, SQL, Synthetic Data
- KDnuggets™ News 21:n33, Sep 1: Top Industries Hiring Data Scientists; The Most Important Tool for Data Engineers - Sep 1, 2021.
The top industries hiring Data Scientists; The most important tool for data engineers (hint - it is not technical); How to Engineer Date Features in Python; 15 Python Snippets to Optimize your Data Science Pipeline
Data Engineer, Data Science, Hiring, Industry, Pipeline, Python
- 15 Python Snippets to Optimize your Data Science Pipeline - Aug 25, 2021.
Quick Python solutions to help your data science cycle.
Data Science, Optimization, Pipeline, Python
Prefect: How to Write and Schedule Your First ETL Pipeline with Python - Aug 16, 2021.
Workflow management systems made easy — both locally and in the cloud.
Cloud, ETL, Pipeline, Python
- Development & Testing of ETL Pipelines for AWS Locally - Aug 2, 2021.
Typically, development and testing ETL pipelines is done on real environment/clusters which is time consuming to setup & requires maintenance. This article focuses on the development and testing of ETL pipelines locally with the help of Docker & LocalStack. The solution gives flexibility to test in a local environment without setting up any services on the cloud.
AWS, Data Engineering, ETL, Pipeline
- Building Machine Learning Pipelines using Snowflake and Dask - Jul 28, 2021.
In this post, I want to share some of the tools that I have been exploring recently and show you how I use them and how they helped improve the efficiency of my workflow. The two I will talk about in particular are Snowflake and Dask. Two very different tools but ones that complement each other well especially as part of the ML Lifecycle.
Dask, Machine Learning, Pipeline, Snowflake
- How to Use Kafka Connect to Create an Open Source Data Pipeline for Processing Real-Time Data - Jul 23, 2021.
This article shows you how to create a real-time data pipeline using only pure open source technologies. These include Kafka Connect, Apache Kafka, Kibana and more.
Data Processing, Kafka, Open Source, Pipeline, Real-time
- Supercharge Your Machine Learning Experiments with PyCaret and Gradio - May 31, 2021.
A step-by-step tutorial to develop and interact with machine learning pipelines rapidly.
Deployment, Machine Learning, Pipeline, PyCaret, Python
- Kedro-Airflow: Orchestrating Kedro Pipelines with Airflow - Mar 12, 2021.
The Kedro team and Astronomer have released Kedro-Airflow 0.4.0 to help you develop modular, maintainable & reproducible code with orchestration superpowers!
Data Science, Interview, Pipeline, Python, Workflow
- Feature Store as a Foundation for Machine Learning - Feb 19, 2021.
With so many organizations now taking the leap into building production-level machine learning models, many lessons learned are coming to light about the supporting infrastructure. For a variety of important types of use cases, maintaining a centralized feature store is essential for higher ROI and faster delivery to market. In this review, the current feature store landscape is described, and you can learn how to architect one into your MLOps pipeline.
Data Engineering, Data Infrastructure, Data Lake, Feature Engineering, Feature Store, Machine Learning, Metadata, MLOps, Pipeline
Cleaner Data Analysis with Pandas Using Pipes - Jan 15, 2021.
Check out this practical guide on Pandas pipes.
Data Analysis, Data Cleaning, Pandas, Pipeline, Python
- Feature Store vs Data Warehouse - Dec 22, 2020.
A feature store is a data warehouse of features for machine learning. Differently from a data warehouse, it is dual-database: one serving features at low latency to online applications and another storing large volumes of features. Learn how Data Scientists leverage this capability in production-deployed models.
Data Warehouse, Databases, Feature Store, Pipeline
Unit Test Your Data Pipeline, You Will Thank Yourself Later - Aug 11, 2020.
While you cannot test model output, at least you should test that inputs are correct. Compared to the time you invest in writing unit tests, good pieces of simple tests will save you much more time later, especially when working on large projects or big data.
Data Science, Pipeline, Programming
- A Tour of End-to-End Machine Learning Platforms - Jul 29, 2020.
An end-to-end machine learning platform needs a holistic approach. If you’re interested in learning more about a few well-known ML platforms, you’ve come to the right place!
AirBnB, Data Science Platform, Google, Machine Learning, MLOps, Netflix, Pipeline, Uber, Workflow
Deploy Machine Learning Pipeline on AWS Fargate - Jul 3, 2020.
A step-by-step beginner’s guide to containerize and deploy ML pipeline serverless on AWS Fargate.
AWS, Docker, Kubernetes, Machine Learning, Pipeline, PyCaret
- A TensorFlow Modeling Pipeline Using TensorFlow Datasets and TensorBoard - Jun 23, 2020.
This article investigates TensorFlow components for building a toolset to make modeling evaluation more efficient. Specifically, TensorFlow Datasets (TFDS) and TensorBoard (TB) can be quite helpful in this task.
Modeling, Pipeline, TensorFlow
- Simplified Mixed Feature Type Preprocessing in Scikit-Learn with Pipelines - Jun 16, 2020.
There is a quick and easy way to perform preprocessing on mixed feature type data in Scikit-Learn, which can be integrated into your machine learning pipelines.
Data Preprocessing, Pipeline, Python, scikit-learn
- Deploy a Machine Learning Pipeline to the Cloud Using a Docker Container - Jun 12, 2020.
In this tutorial, we will use a previously-built machine learning pipeline and Flask app to demonstrate how to deploy a machine learning pipeline as a web app using the Microsoft Azure Web App Service.
Cloud, Docker, Machine Learning, Pipeline, PyCaret, Python
- Nitpicking Machine Learning Technical Debt - Jun 8, 2020.
Technical Debt in software development is pervasive. With machine learning engineering maturing, this classic trouble is unsurprisingly rearing its ugly head. These 25 best practices, first described in 2015 and promptly overshadowed by shiny new ML techniques, are updated for 2020 and ready for you to follow -- and lead the way to better ML code and processes in your organization.
Pages: 1 2
Best Practices, DevOps, Explainability, Interpretability, Machine Learning, Monitoring, Pipeline, Technical Debt, Version Control
Build and deploy your first machine learning web app - May 22, 2020.
A beginner’s guide to train and deploy machine learning pipelines in Python using PyCaret.
App, Flask, Heroku, Machine Learning, Modeling, Open Source, Pipeline, PyCaret, Python
- Managing Machine Learning Cycles: Five Learnings from comparing Data Science Experimentation/ Collaboration Tools - Jan 29, 2020.
Machine learning projects require handling different versions of data, source code, hyperparameters, and environment configuration. Numerous tools are on the market for managing this variety, and this review features important lessons learned from an ongoing evaluation of the current landscape.
Collaboration, Comet.ml, Data Operations, Data Workflow, DataOps, MLflow, MLOps, Pipeline, Reproducibility
- Live Webinar: Learn how to build better machine learning pipelines - Jan 6, 2020.
In this webinar, Jan 15 @ 12PM EST, we'll offer solutions to the common challenges data scientists and data engineers face when building a machine learning pipeline. Register now to attend live or to watch a recording afterwards.
cnvrg.io, Machine Learning, MLOps, Pipeline, Webinar
Build Pipelines with Pandas Using pdpipe - Dec 13, 2019.
We show how to build intuitive and useful pipelines with Pandas DataFrame using a wonderful little library called pdpipe.
Data Preparation, Data Preprocessing, Pandas, Pipeline, Python
- Spark NLP 101: LightPipeline - Nov 27, 2019.
A Pipeline is specified as a sequence of stages, and each stage is either a Transformer or an Estimator. These stages are run in order, and the input DataFrame is transformed as it passes through each stage. Now let’s see how this can be done in Spark NLP using Annotators and Transformers.
Apache Spark, NLP, Pipeline, Spark NLP
Automated Machine Learning Project Implementation Complexities - Nov 22, 2019.
To demonstrate the implementation complexity differences along the AutoML highway, let's have a look at how 3 specific software projects approach the implementation of just such an AutoML "solution," namely Keras Tuner, AutoKeras, and automl-gs.
Automated Machine Learning, Keras, Pipeline, Python
- Testing Your Machine Learning Pipelines - Nov 14, 2019.
Let’s take a look at traditional testing methodologies and how we can apply these to our data/ML pipelines.
Machine Learning, Pipeline, Python
- 5 Step Guide to Scalable Deep Learning Pipelines with d6tflow - Sep 16, 2019.
How to turn a typical pytorch script into a scalable d6tflow DAG for faster research & development.
Deep Learning, Pipeline, Python, PyTorch, Workflow
- Data Pipelines, Luigi, Airflow: Everything you need to know - Mar 27, 2019.
This post focuses on the workflow management system (WMS) Airflow: what it is, what can you do with it, and how it differs from Luigi.
Data Workflow, Pipeline, Python, Workflow
- Webinar: The Value-Based Return on Creating a High-Quality Data Pipeline,
Sep 12 - Sep 7, 2018.
Learn why data quality and data integration are key to delivering meaningful, actionable results, and how to develop data and analytics strategies that offer visibility into healthcare cost and quality.
Data Quality, Healthcare, Looker, Pipeline
- Manage your Machine Learning Lifecycle with MLflow – Part 1 - Jul 5, 2018.
Reproducibility, good management and tracking experiments is necessary for making easy to test other’s work and analysis. In this first part we will start learning with simple examples how to record and query experiments, packaging Machine Learning models so they can be reproducible and ran on any platform using MLflow.
Databricks, Life Cycle, MLflow, Pipeline, Python, Workflow
- KDnuggets™ News 18:n22, Jun 6: 10 More Free Must-Read Books for Machine Learning and Data Science; Beginner Guide to Data Science Pipeline - Jun 6, 2018.
Summer. Time to sit back and unwind. Or get your hands on some free machine learning and data science books and learn! Here is a great selection to get started.
Data Science, Free ebook, Keras, Pipeline, R
A Beginner’s Guide to the Data Science Pipeline - May 29, 2018.
On one end was a pipe with an entrance and at the other end an exit. The pipe was also labeled with five distinct letters: "O.S.E.M.N."
Beginners, Data Science, Pipeline
- Deep Learning With Apache Spark: Part 1 - Apr 18, 2018.
First part on a full discussion on how to do Distributed Deep Learning with Apache Spark. This part: What is Spark, basics on Spark+DL and a little more.
Apache Spark, Databricks, Deep Learning, Pipeline
- A Beginner’s Guide to Data Engineering – Part II - Mar 15, 2018.
In this post, I share more technical details on how to build good data pipelines and highlight ETL best practices. Primarily, I will use Python, Airflow, and SQL for our discussion.
Pages: 1 2
AirBnB, Data Engineering, Data Science, ETL, Pipeline, Python, SQL
- Using AutoML to Generate Machine Learning Pipelines with TPOT - Jan 29, 2018.
This post will take a different approach to constructing pipelines. Certainly the title gives away this difference: instead of hand-crafting pipelines and hyperparameter optimization, and performing model selection ourselves, we will instead automate these processes.
Automated Machine Learning, Hyperparameter, Optimization, Pipeline, Python, scikit-learn, Workflow
A Beginner’s Guide to Data Engineering – Part I - Jan 25, 2018.
Data Engineering: The Close Cousin of Data Science.
Pages: 1 2
Data Engineer, Data Engineering, ETL, Pipeline
- Managing Machine Learning Workflows with Scikit-learn Pipelines Part 3: Multiple Models, Pipelines, and Grid Searches - Jan 24, 2018.
In this post, we will be using grid search to optimize models built from a number of different types estimators, which we will then compare and properly evaluate the best hyperparameters that each model has to offer.
Data Preprocessing, Hyperparameter, Optimization, Pipeline, Python, scikit-learn, Workflow
- KDnuggets™ News 18:n04, Jan 24: TensorFlow vs XGBoost; Machine Learning Pipelines in Python; Semi-Supervised Machine Learning - Jan 24, 2018.
Gradient Boosting in TensorFlow vs XGBoost; Managing Machine Learning Workflows with Scikit-learn Pipelines Part 2; Using Genetic Algorithm for Optimizing Recurrent Neural Networks; The Value of Semi-Supervised Machine Learning; Comparing Machine Learning as a Service: Amazon, Microsoft Azure, Google Cloud AI
Gradient Boosting, Machine Learning, MLaaS, Pipeline, TensorFlow, Workflow, XGBoost
- Managing Machine Learning Workflows with Scikit-learn Pipelines Part 2: Integrating Grid Search - Jan 19, 2018.
Another simple yet powerful technique we can pair with pipelines to improve performance is grid search, which attempts to optimize model hyperparameter combinations.
Data Preprocessing, Hyperparameter, Optimization, Pipeline, Python, scikit-learn, Workflow
- Managing Machine Learning Workflows with Scikit-learn Pipelines Part 1: A Gentle Introduction - Dec 7, 2017.
Scikit-learn's Pipeline class is designed as a manageable way to apply a series of data transformations followed by the application of an estimator.
Data Preprocessing, Pipeline, Python, scikit-learn, Workflow
How to Build a Data Science Pipeline - Jul 14, 2017.
Start with y. Concentrate on formalizing the predictive problem, building the workflow, and turning it into production rather than optimizing your predictive model. Once the former is done, the latter is easy.
Data Science, Pipeline, Production
