Solutions for the KodeKloud Engineer 2.0 tasks.
Technologies used: GCS, Compute Engine, Mage, BigQuery, Looker, Python
This project leverages GCS, Composer, Dataflow, BigQuery, and Looker on Google Cloud Platform (GCP) to build a robust data engineering solution for processing, storing, and reporting daily transaction data in the online food delivery industry.
Connect Four Data Engineering Project: leveraging GCS for scalable and durable storage, Dataflow for data extraction and transformation, BigQuery as the data repository, Slack Integration for real-time sharing, Looker for insightful reports and visualizations, and Email Scheduler for automated report delivery.
This project was born out of the need to know the weather forecast for Paris/Vilnius while attending the KubeCon Europe conference. Fetches next-day forecast for Paris and Vilnius using a weather API, securely stores data in GCS bucket, and sends personalized SMS updates via Twilio. Powered by GCP and automated with Composer/Airflow
This project demonstrates an end-to-end solution for processing and analyzing real-time conversations data from a JSON file using GCP services and infrastructure automation, showcasing data storage, streaming, processing, and analysis at scale.