GridWorks JournalKeeper

---

GridWorks JournalKeeper persists inbound RabbitMQ AMQP messages from the GridWorks fleet into a shared PostgreSQL schema. There are two ingestion paths:

• Live path — JournalKeeper(ActorBase) consumes the broker's ear_tx audit exchange, parses payloads with SemaCodec, and writes rows via SemaMessagePersistor into the messages table.
• Backfill path — S3MessageImporter walks a date range of S3-archived messages and runs them through the same persistor for catch-up after gaps.

The PostgreSQL schema itself lives in the sibling repo gridworks-data (gw_data package); journalkeeper imports from it and never defines its own SQLAlchemy models.

---

Development quick start

Prerequisites

• Python 3.12 or 3.13
• uv
• Docker (for the dev RabbitMQ broker and gridworks-data's PostgreSQL container)
• psql (PostgreSQL client) — optional but useful
• pre-commit — optional; see below

Set up

From the repository root:

uv sync --all-extras    # installs runtime + dev deps
cp template.env .env    # then edit .env with your local broker + db URL

Run tests:

uv run pytest

Lint:

uv run ruff check .

Type-check:

uv run mypy --strict src

Optional: pre-commit

pipx install pre-commit
pre-commit install

pre-commit runs ruff + the other configured hooks before each commit.

---

Runtime dependencies

JournalKeeper needs:

• a running RabbitMQ broker (publishes from scada / LTN / weather are consumed)
• a PostgreSQL database with the gw_data schema applied

Both run as Docker containers locally.

RabbitMQ

If you're working on this repo in isolation (not as part of a larger simulation that brings its own broker), spin up a local dev broker from the sibling gridworks-base repo:

cd ../gridworks-base
./x86.sh        # on Apple silicon: ./arm.sh

That starts the gw-dev-rabbit container (the GHCR-baked image with the canonical exchange/binding topology) on the standard ports (5672 AMQP, 1885 MQTT, 15672 mgmt UI). Then point this repo at it in .env:

GJK_RABBIT__URL=amqp://smqPublic:smqPublic@localhost:5672/d1__1

The same image runs in CI (see .github/workflows/tests.yml), so what you test locally matches what CI tests.

Full RabbitMQ setup details in gridworks-base.

PostgreSQL

The schema is owned by the sibling gridworks-data repo. Follow its README to bring up the timescale/timescaledb-ha:pg18-ts2.25 container (typically on 5433:5432 with POSTGRES_PASSWORD), run the server-init script (gw_admin / gw_writer / gw_reader roles), and uv run alembic upgrade head to create the tables (messages, g_nodes, reading_channels, …).

Then point this repo at it in .env:

GJK_DB_URL=postgresql+psycopg2://gw_writer:<password>@localhost:5433/gridworks

gw_writer is the right role for an app that only inserts rows; gw_admin is reserved for migrations and gw_reader for analytics. Migrations live in gridworks-data, not here.

---

Production notes

JournalKeeper runs against the production RabbitMQ broker at hw1-1.electricity.works (see internal credentials in 1Password) and writes to the production PostgreSQL instance. The production gridworks-data schema is the source of truth for the table shape.

---

Publishing a release

JournalKeeper uses tag-driven releases. The version in pyproject.toml is the single source of truth and must match the Git tag exactly.

Release checklist

1. Update the version — edit pyproject.toml:

   [project]
   version = "0.1.0"

2. Commit and merge to main through the standard branch → dev → main PR flow.

3. Tag on main:

   git checkout main
   git pull origin main
   git tag v0.1.0
   git push origin v0.1.0

This triggers the Release workflow.

---

Contributing

This repository uses:

• ruff for linting and formatting
• mypy for type checking
• pytest for tests
• pre-commit for enforcement (optional)

Before committing:

uv run ruff check .
uv run mypy --strict src
uv run pytest

Or, if you have pre-commit installed:

pre-commit run --all-files

---

License

MIT — see LICENSE.