Forecast Tracker

Household Energy Usage

A data engineering pipeline to track the output of my energy usage forecasts.

• code

About

One of the components of my household energy usage project is a forecasting API. As with any predictive analytics project, it’s important to track predictions to assess the true quality of the work.

Software

• Python 3.11
  • Polars
  • Ibis
  • Pyarrow
  • Hamilton (maybe)
• AWS S3

Pipeline

This work uses a parquet file to save the forecasts. The file is uploaded to an AWS S3 bucket.

I’m considering using Hamilton to learn how to use a formal pipeline tool.

For data manipulations I’ve used polars and ibis. Polars is a modern DataFrame library and adoption is growing. I’ve used polars before but I wanted to gain more experience. This is the first project I’ve used Ibis. Ibis is a DataFrame interface that interacts with various back-ends. I like the idea of Ibis because it standardizes data work regardless of the back-end.