Image by Author

If you are looking to get into the world of data, particularly Data Engineering - then this blog can provide you with valuable resources to help your studies. Let’s first break down what the difference is between a data scientist and a data engineer in the simplest way. 

A data scientist's main focus is to explore data, build models, and implement machine learning algorithms. A data engineer's main focus is ensuring that the algorithms built work effectively in production infrastructure and creating data pipelines. 

Data engineers are responsible for everything surrounding the organization's data infrastructure. This infrastructure will store the business's critical information, ranging from small databases to large-scale systems. The aim is to ensure that the data's foundation is solid and secure in order for critical analysis to be performed and reports can be produced. 

If you are still keen on learning about data engineering, here are some valuable GitHub repositories to help you.

DataTalks.Club - data-engineering-zoomcamp

Repository link: data-engineering-zoomcamp

As mentioned in the name, DataTalksClub is a global online community of data enthusiasts, that talk about everything data. They have a 9-week syllabus to help you learn about data engineering. The weeks are broken down into:

• Week 1: Introduction & Prerequisites
• Week 2: Data ingestion
• Week 3: Data Warehouse
• Week 4: Analytics Engineering
• Week 5: Batch processing
• Week 6: Streaming
• Week 7, 8 & 9: Project

You can join the next cohort, however, you can also do it in your own time. All the materials of the course are freely available, and DataTalks.Club provide you with a suggested syllabus week by week to help you. 

Cookbook

Repository link: Cookbook

Andreaz Kretz, the author of The Data Engineering Cookbook published the book on GitHub. His aim with this book was to provide a starting point for newbies in the data engineering world. He helps you to identify the important topics you need to learn about to become a successful Data Engineer. 

The book focuses on five different types of content to help you with data engineering: articles published by the author, links to their podcast episodes (video & audio), 200+ links to helpful websites that he recommends, data engineering interview questions and case studies.

Data-Engineering-HowTo

Repository link: Data-Engineering-HowTo

If you need guidance on the different topics you need to learn to become a Data Engineer. The Data-Engineering-HowTo provides you with a list of different resources where you can gain useful data engineering knowledge. 

The repo starts with the basics of the world of data engineering, such as the hierarchy needs, beginner's guide, and more. There are also resources for talks, algorithms & data structures, SQL, programming, databases, distributed systems, books, courses, blogs, tools, cloud platforms, and more. 

Awesome Data Engineering

Repository link: awesome-data-engineering

If you have a good foundation of the basics of data engineering or need a better focus on the tooling, this GitHub repository provides you with a curated list of the type of data engineering tools you make come across. 

To become a successful data engineer, you need to be confident with the tooling. This repo goes through all the types of tools available for: 

1. Databases
2. Ingestion
3. File System
4. Serialization format
5. Stream Processing
6. Batch Processing
7. Charts and Dashboards
8. Workflow
9. Data Lake Management
10. ELK Elastic Logstash Kibana
11. Docker
12. Datasets
13. Monitoring
14. Community

Data Engineer Roadmap

Repository link: data-engineer-roadmap

If you are more of a visual person and need help you the route you need to take to become a successful Data Engineer - this repo is for you. It provides you with a complete visualisation of the modern data engineering landscape and acts as a study guide.

The author of the repo stated that:

“Beginners shouldn’t feel overwhelmed by the vast number of tools and frameworks listed here. A typical data engineer would master a subset of these tools throughout several years depending on his/her company and career choices.”

Overall, this roadmap visualisation is an effective syllabus for aspiring data engineers.

Start Data Engineering

Repository link: Start Data Engineering

If you’re feeling confident in your data engineering skills and want to start putting them to the test. Joseph Machado talks all about data engineering, data modelling, software engineering, and system design.

He provides you with a step-to-step guide on how to begin the project, which will be useful to your data engineering study aswell as part of your portfolio for when you’re ready to apply for jobs. 

Data Engineering Projects

Repository link: Data-Engineering-Projects

If you are looking for more projects that apply to the principles of data engineering, this GitHub repo provides you with the following 7 different types of projects:

1. Postgres ETL 
2. Cassandra ETL 
3. Web Scraping using Scrapy, MongoDB ETL 
4. Data Warehousing with AWS Redshift 
5. Data Lake with Spark & AWS S3 
6. Data Pipelining with Airflow 
7. Capstone Project 

Data Engineering Interview Questions

Repository link: data-engineering-interview-questions

Let’s say you’re feeling confident with your data engineering skills, you’ve put them to the test, and now you’re ready to apply for that job you’ve been working hard for. You will need to prepare for the type of interview questions that may appear on the day. 

This GitHub repo has more than 2000+ questions to help you prepare for your Data Engineer interview. They also provide you with the answers, allowing you to learn where your strengths and weaknesses lie in data engineering. 

Conclusion

The above resources on GitHub will help you to become a successful Data Engineer in no time. If you need a study roadmap, have a read of The Complete Data Engineering Study Roadmap. It provides you with a list of topics, areas and resources to help your data engineering journey.
 
 
Nisha Arya is a Data Scientist and Freelance Technical Writer. She is particularly interested in providing Data Science career advice or tutorials and theory based knowledge around Data Science. She also wishes to explore the different ways Artificial Intelligence is/can benefit the longevity of human life. A keen learner, seeking to broaden her tech knowledge and writing skills, whilst helping guide others.