- Loading Terabytes of Data from Postgres into BigQuery - Apr 9, 2018.
Despite the fact that an ETL task is pretty challenging when it comes to loading Big Data, there’s still the scenario in which you can load terabytes of data from Postgres into BigQuery relatively easy and very efficiently.
Tags: BigQuery, ETL, NoSQL, Postgres, SQL, Statsbot
- A Beginner’s Guide to Data Engineering – Part II - Mar 15, 2018.
In this post, I share more technical details on how to build good data pipelines and highlight ETL best practices. Primarily, I will use Python, Airflow, and SQL for our discussion.
Pages: 1 2
Tags: AirBnB, Data Engineering, Data Science, ETL, Pipeline, Python, SQL
A Beginner’s Guide to Data Engineering – Part I - Jan 25, 2018.
Data Engineering: The Close Cousin of Data Science.
Pages: 1 2
Tags: Data Engineer, Data Engineering, ETL, Pipeline
Are Data Lakes Fake News? - Sep 6, 2017.
The quick answer is yes, and the biggest problem is that the term “Data Lakes” has been overloaded by vendors and analysts with different meanings, resulting in an ill-defined and blurry concept.
Tags: Data Lakes, Data Warehouse, ETL, Fake News, Hadoop
- How to Choose a Data Format - Nov 3, 2016.
In any data analytics project, after business understanding phase, data understanding and selection of right data format as well as ETL tools is very important task. In this article, a very useful and practical set of guidelines is explained covering data format selection and ETL phases of project lifecycle.
Pages: 1 2
Tags: Data Cleaning, Data Engineering, Data Preparation, ETL, Hadoop, HDFS
- Automating Data Ingestion: 3 Important Parts - Sep 9, 2016.
In the day and age of ‘Big Data”, data ingestion has to be automated on some level. How best to automate it?
Tags: Automation, Claudia Perlich, Data Preparation, ETL, Quora
- Choosing Tools for Data ETLs - Aug 9, 2016.
Which tool should I use for my data pipelines? Get some advice from a data scientist recently having gone through this pipeline tool selection process.
Tags: AirBnB, Data Cleaning, Data Preparation, ETL
- Engineers Shouldn’t Write ETL: A Guide to Building a High Functioning Data Science Department - Mar 28, 2016.
An exploration of data science team building, with insight into why engineers should not write ETL, and other not-so-subtle pieces of advice.
Pages: 1 2 3
Tags: Advice, Data Engineering, Data Scientist, ETL, Stitch Fix
- Data Lake Plumbers: Operationalizing the Data Lake - Feb 18, 2016.
Gain insight into data lakes, their benefits, when they are appropriate, and how to operationalize them. How do they compare to the data warehouse?
Tags: Data Lake, Data Warehouse, ETL, Hadoop
- 3 Reasons Big Data Projects Fail - Aug 24, 2015.
Download Lavastorm whitepaper: How to Overcome 3 Key Big Data Challenges - how to operationalize the results, how to enable ETL to handle complexities of Big Data, and more.
Tags: Big Data, ETL, Lavastorm, Project Fail
- Interview: Joseph Babcock, Netflix on Genie, Lipstick, and Other In-house Developed Tools - Jun 16, 2015.
We discuss role of analytics in content acquisition, data architecture at Netflix, organizational structure, and open-source tools from Netflix.
Tags: Data Science, ETL, In-house, Interview, Joseph Babcock, Netflix, Open Source, Tools
