Topic: Data Engineering
Data Engineering is a close cousin of Data Science. If we can compare Data Scientists to pilots (who fly the surveillance planes to learn what is happening), then Data Engineers would be the airplane builders and mechanics. They build and maintain the data and analytic pipelines that are used by Data Scientists, BI analysts, and others.
This page features the most recent and most popular posts on Data Engineering and Data Engineers
Latest posts on Data Engineering
- Data Engineering Technologies 2021 - Sep 21, 2021Emerging technologies supporting the field of data engineering are growing at a rapid clip. This curated list includes the most important offerings available in 2021.
- Data Scientists Without Data Engineering Skills Will Face the Harsh Truth - Sep 14, 2021Although the role of the data scientist is still evolving, data remains at its core. Setting the right expectations for what you will do as a data scientist is important, and, to be sure, knowing the tools of data engineering will get yourself ready for the real world.
- The Most Important Tool for Data Engineers - Aug 26, 2021And it has nothing to do with Python or SQL
- Model Drift in Machine Learning – How To Handle It In Big Data - Aug 17, 2021Rendezvous Architecture helps you run and choose outputs from a Champion model and many Challenger models running in parallel without many overheads. The original approach works well for smaller data sets, so how can this idea adapt to big data pipelines?
- Development & Testing of ETL Pipelines for AWS Locally - Aug 2, 2021Typically, development and testing ETL pipelines is done on real environment/clusters which is time consuming to setup & requires maintenance. This article focuses on the development and testing of ETL pipelines locally with the help of Docker & LocalStack. The solution gives flexibility to test in a local environment without setting up any services on the cloud.
Most popular (badge-winning) recent posts on Data Engineering
- Data Scientists Without Data Engineering Skills Will Face the Harsh Truth [Gold Blog]Although the role of the data scientist is still evolving, data remains at its core. Setting the right expectations for what you will do as a data scientist is important, and, to be sure, knowing the tools of data engineering will get yourself ready for the real world.
- Analytics Engineering Everywhere [Gold Blog]Many new roles have appeared in the data world ever since the rise of the Data Scientist took the spotlight several years ago. Now, there is a new core player ready to take center stage, and we may see in five years, nearly every organization will have an Analytics Engineering team.
- Data Scientist, Data Engineer & Other Data Careers, Explained [Platinum Blog]In this article, we will have a look at five distinct data careers, and hopefully provide some advice on how to get one's feet wet in this convoluted field.
- Why You Should Consider Being a Data Engineer Instead of a Data Scientist [Silver Blog]A new king of the jungle has emerged.
- 9 Skills You Need to Become a Data Engineer [Silver Blog]A data engineer is a fast-growing profession with amazing challenges and rewards. Which skills do you need to become a data engineer? In this post, we’ll take a look at both hard and soft skills.
- Data Science Learning Roadmap for 2021 [Gold Blog]Venturing into the world of Data Science is an exciting, interesting, and rewarding path to consider. There is a great deal to master, and this self-learning recommendation plan will guide you toward establishing a solid understanding of all that is foundational to data science as well as a solid portfolio to showcase your developed expertise.
- Data Engineering — the Cousin of Data Science, is Troublesome [Gold Blog]A Data Scientist must be a jack of many, many trades. Especially when working in broader teams, understanding the roles of others, such as data engineering, can help you validate progress and be aware of potential pitfalls. So, how can you convince your analysts to realize the importance of expanding their toolkit? Examples from real life often provide great insight.
- Why the Future of ETL Is Not ELT, But EL(T) [Platinum Blog]The well-established technologies and tools around ETL (Extract, Transform, Load) are undergoing a potential paradigm shift with new approaches to data storage and expanding cloud-based compute. Decoupling the EL from T could reconcile analytics and operational data management use cases, in a new landscape where data warehouses and data lakes are merging.
- Introduction to Data Engineering [Gold Blog]The Q&A for the most frequently asked questions about Data Engineering: What does a data engineer do? What is a data pipeline? What is a data warehouse? How is a data engineer different from a data scientist? What skills and programming languages do you need to learn to become a data engineer?
- The Rise of the Machine Learning Engineer [Gold Blog]The evolution of Big Data into machine learning applications ushered in an exciting era of new roles and skillsets that became necessary to implement these technologies. With the Machine Learning Engineer being such a crucial component today, where the evolution of this field will take us tomorrow should be fascinating.
- Skills to Build for Data Engineering [Silver Blog]This article jumps into the latest skill set observations in the Data Engineering Job Market which could definitely add a boost to your existing career or assist you in starting off your Data Engineering journey.
- 7 Resources to Becoming a Data Engineer [Gold Blog]An estimated 8,650% growth of the volume of Data to 175 zetabytes from 2010 to 2025 has created an enormous need for Data Engineers to build an organization's big data platform to be fast, efficient and scalable.
- A Winning Game Plan For Building Your Data Science Team [Silver Blog]We need to understand the responsibilities, capabilities, expectations and competencies of the Data Engineer, Data Scientist and Business Stakeholder.
- A Beginner’s Guide to Data Engineering – Part I [Silver Blog]Data Engineering: The Close Cousin of Data Science.
- 37 Reasons why your Neural Network is not workingOver the course of many debugging sessions, I’ve compiled my experience along with the best ideas around in this handy list. I hope they would be useful to you.
- 5 Career Paths in Big Data and Data Science, ExplainedSexiest job... massive shortage... blah blah blah. Are you looking to get a real handle on the career paths available in "Data Science" and "Big Data?" Read this article for insight on where to look to sharpen the required entry-level skills.