Karachi AQI Forecasting System

Overview

Karachi AQI Forecasting System is a production-oriented, serverless machine learning system designed to forecast Karachi's Air Quality Index across 24-hour, 48-hour, and 72-hour horizons.

The project was developed during my Data Science internship and evolved from an earlier Hopsworks-based architecture into a more stable Google Cloud Platform stack using BigQuery for feature storage and Google Cloud Storage for model versioning.

The system includes automated data pipelines, horizon-specific forecasting models, scheduled CI workflows, explainability tooling, and a deployed Streamlit dashboard.

Problem

AQI forecasting is more difficult than simple nowcasting because the model must predict future air quality without having access to future ground-truth pollutant readings.

The goal was to build a practical forecasting system that could:

• collect updated weather and air-quality data
• engineer temporal forecasting features
• train separate models for multiple prediction horizons
• generate 24h, 48h, and 72h forecasts
• store data and models in cloud infrastructure
• automate pipeline execution
• expose results through a production dashboard

Architecture

The final architecture uses four lightweight pipeline scripts:

1. feature_pipeline.py

   • Fetches Open-Meteo data
   • Engineers forecasting features
   • Writes features to BigQuery

2. training_pipeline.py

   • Trains horizon-specific models
   • Evaluates candidate algorithms
   • Uploads trained models to Google Cloud Storage

3. inference_pipeline.py

   • Generates 24h, 48h, and 72h AQI forecasts

4. backfill_pipeline.py

   • Loads historical Open-Meteo data to bootstrap the training dataset

The system intentionally uses a flat pipeline structure rather than a large package hierarchy to keep debugging and operations simple.

Cloud Architecture

The production data and model layer is built on Google Cloud Platform.

BigQuery

BigQuery acts as the feature store and contains the engineered time-series data used for training and inference.

Google Cloud Storage

Google Cloud Storage acts as the model registry.

Separate model artifacts are maintained for:

• 24-hour forecasting
• 48-hour forecasting
• 72-hour forecasting

The architecture was migrated from Hopsworks to BigQuery and GCS after repeated cluster stability and resource-scheduling issues.

Data Pipeline

Open-Meteo is used as the primary data source for weather and air-quality features.

The feature pipeline engineers:

• 1-hour lag features
• 3-hour lag features
• 6-hour lag features
• 12-hour lag features
• 24-hour lag features
• rolling statistics
• cyclical hour encodings
• cyclical day-of-week encodings

A historical backfill pipeline was also used to bootstrap the feature store with sufficient training history.

Data Quality Challenges

One of the most important parts of the project was diagnosing data-quality problems rather than simply tuning models.

An early version suffered from distribution shift because synthetic backfill data had a significantly different AQI distribution from the real label data.

This resulted in negative R² scores.

The affected feature data was rebuilt so the training set contained a cleaner and more consistent distribution.

This demonstrated that improving data quality had a larger impact on the system than simply increasing model complexity.

Feature Leakage Discovery

Another major issue appeared when early model versions produced unusually high R² values around 0.92.

Investigation revealed that forecast PM2.5 features were acting as a direct proxy for the AQI target because PM2.5 mathematically maps to AQI through the underlying AQI calculation.

Those features were removed from training.

SHAP analysis confirmed the fix by showing that feature importance became distributed across multiple predictors instead of being dominated by one leaked feature.

Timestamp Gap Problem

A second forecasting issue came from missing hourly timestamps.

Using a simple row-based shift such as shift(-24) did not always represent exactly 24 hours into the future when timestamp gaps existed in the dataset.

In some cases, the shifted label corresponded to 30, 60, or even more than 100 hours ahead.

The training pipeline was updated to explicitly validate:

target_timestamp - current_timestamp == forecast_horizon

Rows with corrupted horizon alignment were removed before training.

This restored the models to valid positive R² performance.

Model Comparison

The project compared multiple model families:

• Ridge Regression
• Random Forest
• XGBoost
• LSTM

Random Forest was ultimately selected as the best overall model across the forecast horizons.

Final Model Performance

Final Random Forest metrics:

• 24h forecast

  • RMSE: 9.818
  • R²: 0.2997

• 48h forecast

  • RMSE: 9.588
  • R²: 0.3297

• 72h forecast

  • RMSE: 10.308
  • R²: 0.2336

These results are reported as genuine future forecasting performance rather than same-day AQI reconstruction.

Explainability

The dashboard includes SHAP and LIME explainability tools to help inspect which features contribute most strongly to model predictions.

This was particularly useful during feature-leakage investigation and model validation.

Streamlit Dashboard

The deployed Streamlit application provides:

• 3-day AQI forecasts
• current AQI display
• model comparison
• exploratory data analysis
• historical AQI trends
• SHAP explainability
• LIME explanations
• Plotly visualizations

Automation and CI/CD

GitHub Actions is used to automate the project pipelines.

Scheduled workflows can run the same scripts used during local development, creating a consistent workflow between development and cloud execution.

Operational Improvements

The production dashboard also required several performance and reliability fixes.

These included:

• increasing model cache lifetime to prevent unnecessary model downloads
• fixing stale historical-data caching
• correcting outdated "Last Updated" timestamps
• resolving data freshness issues

These problems reinforced that operational reliability is part of machine learning engineering, not separate from it.

What I Learned

This project taught me that production ML performance depends heavily on data engineering, temporal correctness, infrastructure reliability, and monitoring.

The most important lessons were:

• data quality can matter more than model complexity
• high accuracy can indicate leakage rather than success
• row-based shifting is unsafe when timestamps contain gaps
• forecasting and nowcasting should not be benchmarked as equivalent problems
• cloud infrastructure and caching issues are real production concerns
• simpler architectures can be easier to operate and debug