Data Engineering (ETL)

Data Engineering (ETL)#

GitHub Repository#

Viridi Battery Issues Reporting

Overview#

This project demonstrates my expertise in building ETL pipelines to handle large-scale data ingestion, transformation, and loading. I optimized battery-related issue reporting for Viridi using structured pipelines and scalable infrastructure.

Skills Used#

  • PySpark

  • AWS Step Functions

  • SnapLogic

  • Data Lake architecture

System Design#

View System Design

Video Demonstration#

Watch Demo