What is a staging area?
tronghuy28071. Introduction 2. What is a staging area 3. The advantages of having a staging area 5. Conclusion 6. Further reading 1. Introduction Working with data pipelines, you might have…
Learning Data Engineering in 2023
What core skills do you need to succeed as a Data Engineer? What is the state of data engineering today? In 2023 the demand for data engineers is showing no…
Understanding Database Types
The success of a software application often hinges on the choice of the right databases. As developers, we're faced with a vast array of database options. It is crucial for…
Installing Apache Airflow on AWS EC2
Table of Contents Setting up EC2 on AWS Python and Pip Creating a virtual environment Installing Airflow Initialising Airflow source: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+logos Apache Airflow is a widely used open source tool in…
Data Engineering Design Principles You Should Follow
Design principles to follow that will take your data engineering skills to the next level. Data pipelines don’t need to be too complex. Software engineering is far more mature than…