Development Guide

This guide will help you set up Open Electricity for local development. The project consists of two main components:

• A FastAPI web API server
• An Arq background worker for processing tasks

Prerequisites

Installing uv

We use uv as our package manager and virtual environment tool. It’s significantly faster than pip and provides better dependency resolution.

Install uv using one of these methods:

macOS/Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Project Setup

1. Clone the repository:

git clone https://github.com/opennem/opennem.git
cd opennem

2. Create the virtual environment and install dependencies:

uv install
source .venv/bin/activate  # On Windows use: .venv\Scripts\activate

uv install
.venv\Scripts\activate

Core Dependencies

Open Electricity uses several key libraries:

• FastAPI - Modern web framework for building APIs
• SQLAlchemy 2.0 - SQL toolkit and ORM
• asyncpg - Async PostgreSQL driver
• Alembic - Database migration tool
• Arq - Async job queue and worker
• Pydantic - Data validation using Python type annotations
• uvicorn - ASGI server for running the API
• TimescaleDB - Time-series database extension for PostgreSQL
• Redis - In-memory data store used by Arq

Running the Application

The application consists of two main processes that need to be run:

1. API Server

The FastAPI application serves the REST API endpoints. Run it with:

uv run api

This will start the API server at http://localhost:8000

The OpenAPI documentation will be available at:

• http://localhost:8000/docs (Swagger UI)

2. Background Worker

The Arq worker processes background tasks like data ingestion and exports. Run it with:

uv run worker

Development Database

1. Install PostgreSQL 17 locally

2. Create a database:

createdb opennem

3. Run migrations:

uv run alembic upgrade head

Local Services

We provide a Docker Compose configuration for running required services locally. Start the services with:

docker-compose up -d

This will start:

• PostgreSQL 17 with TimescaleDB extension
• Redis server for the task queue

Environment Setup

1. Copy the example environment file:

cp .env.example .env

2. Edit the .env file with your local settings. The example file contains all required variables with sensible defaults for local development.

Configuration Settings

All application settings are defined in opennem/settings_schema.py and are loaded from environment variables. Key settings include:

• Database connection details
• Redis configuration
• API settings
• Authentication settings
• Logging configuration

You can view all available settings and their documentation in the schema file. Each setting can be overridden using environment variables with the OPENNEM_ prefix.

Example settings from settings_schema.py:

Code Quality Tools

We use several tools to maintain code quality:

• Ruff - Fast Python linter and formatter
• mypy - Static type checker
• pytest - Testing framework

Run the quality checks:

API Documentation

The API documentation is automatically generated from the OpenAPI schema. You can view it at:

• Local development: http://localhost:8000/docs
• Production: https://api.openelectricity.org.au/docs

Getting Help

If you need help:

1. Check the project documentation
2. Open an issue on GitHub
3. Join our community discussions

Contributing

1. Create a new branch for your feature
2. Make your changes
3. Run the test suite
4. Submit a pull request

Please follow our coding standards:

• Use type hints for all function parameters and returns
• Write docstrings for all functions and modules
• Follow PEP 8 style guidelines
• Write tests for new functionality