RFC: Running Nebraska as a distributed service with PSQL logical replication

[Moustafa-Moustafa]

Status

We now have a design doc proposed in PR #1405. It's the most up-to-date source for the design. This issue still has the original proposal below, with a few sections updated to match the current direction.

Problem

We want to be able to host Nebraska as a managed service and need to run multiple instances across different regions, each backed by its own PostgreSQL database, so that nodes can reach the nearest instance with low latency and we are able to scale.

Nebraska today assumes a single instance with a single database. Even Omaha update checks (POST /v1/update/) write telemetry to the database, so read replicas are not an option. Each region needs its own writable database.

The challenge is keeping publisher metadata (apps, channels, packages, groups) consistent across all regional databases while letting each region write telemetry locally.

How Nebraska's schema already helps

Nebraska's tables split almost cleanly into two categories:

Admin tables (publisher metadata, must be consistent everywhere): application, package, channel, groups, team, users, flatcar_action, package_channel_blacklist, package_file

Runtime tables (per-region telemetry, stays local): instance, instance_application, instance_status_history, event, activity, instance_stats

Reference / seed tables (identical on every node, populated by migrations, not replicated): event_type

All foreign keys point from runtime tables to admin tables, never the other direction. This makes PostgreSQL logical replication a natural fit: replicate admin tables from a primary database to regional subscriber databases, while each region writes runtime data locally.

Two tables mix admin and runtime concerns and need to be split first: groups (admin policy fields plus runtime-mutated rollout_in_progress) and activity (admin-originated rows for channel package updates plus runtime-originated rows for rollout lifecycle and instance update results). See Proposal #1 and #2 below.

Proposal

Control node: one Nebraska instance that accepts admin API writes and replicates publisher metadata to all edges.

Edge nodes: regional Nebraska instances that serve Omaha update checks and write telemetry locally. Admin API calls return 403. The syncer is disabled.

Control                          Edge (per-region)
┌──────────────────┐         ┌──────────────────┐
│ Nebraska Control │         │ Nebraska Edge    │
│ admin.Service    │         │ runtime.Service  │
│ runtime.Service  │         │ (no admin writes)│
└────────┬─────────┘         └────────┬─────────┘
         │                            │
    ┌────┴─────┐                 ┌────┴────┐
    │Primary DB│ ──logical──>    │ Sub DB  │
    │ admin    │  replication    │ admin RO│
    │runtime   │  (admin only)   │runtime  │
    └──────────┘                 └─────────┘

The specific changes:

1. Split groups into admin and runtime (#1396)

Introduce a node-local group_local sidecar holding rollout_in_progress and a nullable override for each admin policy field on groups. Reads return the override if set, otherwise the admin default. After this, groups is a pure admin table safe to replicate.

2. Split activity into admin and runtime (#1398)

The single activity table holds two kinds of rows: admin-originated (channel package updates, class=6) and runtime-originated (rollout lifecycle, instance update results, classes 1–5). Split into admin_activity (replicated control → edge) and activity (runtime, local per node), with an all_activity UNION ALL view so existing readers are unchanged. activity.id is converted from serial to uuid so each node can generate ids without coordination. This split is what makes the database-level admin/runtime enforcement (see Enforcement layers below) work for activity: the runtime role can write to activity but not admin_activity.

3. Sub-package structure for compile-time enforcement

Split pkg/api/ into sub-packages:

• admin/ - write operations for admin tables (control node only)
• runtime/ - write operations for runtime tables (all instances)
• dbreads/ - read operations shared by both (all instances)

pkg/api/
├── admin/          # Write operations for admin tables (control node only)
│   ├── service.go
│   ├── applications.go
│   ├── channels.go
│   ├── groups.go
│   ├── packages.go
│   ├── teams.go
│   ├── users.go
│   ├── actions.go
│   └── admin_activity.go
├── runtime/        # Write operations for runtime tables (all instances)
│   ├── service.go
│   ├── events.go
│   ├── instances.go
│   ├── updates.go
│   ├── group_local.go
│   └── activity.go
├── dbreads/        # Read operations (all instances)
│   ├── queries.go
│   ├── applications.go
│   ├── channels.go
│   ├── groups.go
│   ├── instances.go
│   ├── packages.go
│   └── activity.go
├── internal/shared/ # Constants and helpers shared across sub-packages
├── api.go          # Core types, DB connection, migrations
└── db/migrations/

admin.Service and runtime.Service both embed dbreads.Queries for reads. internal/shared/ holds constants and helpers needed by multiple sub-packages. Go package boundaries prevent runtime from calling admin methods and vice versa.

4. Instance mode

NEBRASKA_INSTANCE_MODE environment variable. When set to edge, admin writes return 403 and the syncer is disabled. When unset, everything works as today. Fully backward compatible. The accepted values are a small validated allowlist (control, edge, or unset/single); any other value fails the process at startup with a clear error.

5. Operator-provisioned database roles

The operator provisions least-privilege Postgres roles for admin and runtime serving. Nebraska reads connection strings from environment variables; it does not create, drop, or rotate roles. See §4.5 of the design doc (#1405) for the full operator contract.

Schema migrations

Schema changes follow a variant of the expand–contract pattern for the replication scenario where additive changes (e.g. new nullable columns, new tables, relaxed constraints) are applied to subscribers before the primary and subtractive changes (e.g. drops, tightened constraints) are applied to the primary before subscribers and modifications (e.g. type changes, renames, NOT NULL tightening) are decomposed into a multi-release sequence: add new -> dual-write -> backfill -> switch readers -> stop writing old -> drop old. The invariant is that the subscriber schema is always a superset of what the primary currently writes, which is exactly what PostgreSQL's logical replication requires ("intermittent errors can be avoided by applying additive schema changes to the subscriber first").

Enforcement layers

The admin/runtime boundary is enforced at three levels:

1. Compile-time (Go packages): runtime/ cannot call admin/ methods.
2. HTTP-level (handler guards): edge nodes return 403 for admin API calls.
3. Database-level (PostgreSQL roles): the runtime role physically cannot write to admin tables.

PoC results

I put together a simple two-region PoC with a control node and an edge node, each running on separate infrastructure with their own PostgreSQL database and logical replication between them. The implementation covers all the changes described above with all existing tests passing.

Some indicative numbers (not a proper benchmark, just what I saw during testing):

• Apps, channels, packages, and groups created on the primary are visible on the subscriber within about 150ms
• Omaha update checks on the edge return the correct version immediately after replication
• Admin API writes on edge nodes are blocked with 403
• Telemetry (instance registrations, events) stays local to each region
• In-memory caches (app IDs, group track names) use invalidate-on-miss so replicated data is served on first access

The PoC code is on my fork: Moustafa-Moustafa#1. This is of course just a PoC. I'll split my changes into smaller PRs when we are ready to move forward with this proposal.

Would love to get feedback on the approach.

Rollout

• PR 1 — group_local sidecar split (feat(db): make rollout state runtime-local for distributed Nebraska #1396)
• PR 2 — activity row-level split + all_activity view (feat(db): split activity into admin/runtime tables for distributed Nebraska #1398)
• PR 3 — Extract pkg/api/dbreads/
• PR 4 — Split writes into pkg/api/admin/ and pkg/api/runtime/
• PR 5 — Conditional role grants migration + two-DSN migration/serving split (NEBRASKA_MIGRATIONS_DB_URL)
• PR 6 — Instance mode: NEBRASKA_INSTANCE_MODE + requirePrimary middleware + validated allowlist
• PR 7 — Per-edge override management endpoint
• PR 8 — Operator documentation

[ervcz]

Hi @Moustafa-Moustafa , thank you for the proposal and the PoC. I like the design and I tried to review the PoC changes too.

I checked the code and have a few smaller implementation details to propose to discuss.

1. Admin can not pause a group on subscribers

policy_updates_enabled lives only in group_state, which is per-region by design. So when an admin sets it to false, only the primary honors it. Subscribers create their own row via the trigger with the default true and keep handing out updates. UI shows "paused" but updates still flow.

Did I miss something, or is this an oversight?

The original column had two writers with different intents (admin = global pause, rollout engine = per-region safe-mode auto-disable) that can not share one storage location. One option:

• Keep policy_updates_enabled on groups (replicated, admin-controlled).
• Add a separate flag like safe_mode_disabled on group_state for the rollout engine.
• Read path checks both.

2. Cache invalidate-on-miss runs in single-instance mode too

dbreads.GetAppID and GetGroupID invalidate the entire cache and rebuild it on every miss (applications.go#L163-L170, groups.go#L183-L188). Single-instance hosters don't need this (apps and groups appear in their cache instantly because they're created locally).
Could this code path be gated to subscriber mode only?

3. UpdateChannel no longer logs the channel-package activity

The old code added an activityChannelPackageUpdated entry when a channel's package changed. The new admin/channels.go doesn't, and the constant is gone from shared/constants.go. The frontend still has UI for it.

I assume this happened because activity is a runtime table and the new boundary makes it awkward to write from admin.Service. Was it intentional?

[Moustafa-Moustafa]

Thanks @ervcz , all three are valid findings.

I'm addressing these 3 points below, but I'd also appreciate your input on the overall proposal covering the admin/runtime/dbreads package split, logical replication approach, instance mode, and DB role bootstrap. The PoC is more of a validation of my proposal and to better demonstrate the changes and I used an agent to expedite making most of the mechanical changes for the PoC. That said, your findings are all valid and are capturing real flaws.

1. For policy_updates_enabled, you're right, admin pause doesn't propagate to subscribers. I think your suggestion can work. I moved policy_updates_enabled back to the groups table (replicated and admin-controlled) and added a separate safe_mode_disabled flag on group_state for the runtime's auto-disable on failed rollouts. The read path checks both.

2. For cache invalidate-on-miss, the invalidation only happens on the error path (when app/group not found), so the cost is one extra SELECT per unknown ID. In single-instance mode it's unnecessary overhead but I'd say it's negligible. While gating behind a mode check is possible, It's much simpler as-is, specially for one query on a rare error path.

3. For activityChannelPackageUpdated, this was an unintentional regression during the sub-package split. To fix it, we can split the activity table to admin_activity (replicated) holds admin-originated events like channel package updates, while activity (local to subscriber) holds runtime events. An all_activity SQL VIEW unions both tables so the existing query/UI code reads from one source with no duplication. admin.Service writes to admin_activity, runtime.Service writes to activity. Each subscriber sees its own runtime events plus all admin events replicated from the primary.

I have updated my PoC PR with the above proposed fixes.

[ervcz]

Thanks @Moustafa-Moustafa for breaking this into smaller PRs. On the direction you asked about, I think the approach is right. A few things I would like to think through with you before the bigger pieces arrive.

The safe mode brake. When an edge trips its brake, how does an operator see it from the control side, and how do they turn it back on? If the answer is to go straight to that region's database, that is fine, but it is worth documenting for operators later, with clear warnings.

Role bootstrap. The password is regenerated each startup and kept only in memory, which is a clean way to avoid storing it. What I want to understand is how that works with more than one instance. What does a second pod connect as, and what happens to the first when the second starts up?

Instance mode. How should a node handle a mode value it does not recognize? In a templated deploy the value can drift to something like Subscriber or replica, and I would not want that handled by accident.

[Moustafa-Moustafa]

Thank you @ervcz for your review, excited to get this moving.

The safe mode brake. When an edge trips its brake, how does an operator see it from the control side, and how do they turn it back on? If the answer is to go straight to that region's database, that is fine, but it is worth documenting for operators later, with clear warnings.

Single-instance. The brake trips by setting group_local.policy_updates_enabled_override = false. The admin default on groups.policy_updates_enabled is untouched. The UI and API return COALESCE(override, default), so the effective value correctly shows false. When the admin re-enables updates, the same handler clears the override back to NULL and the effective value follows the default again.

Distributed. The control node behaves exactly as in single-instance for its own state. Each edge node owns its own brake on its own group_local, and reads the admin default from the replicated groups row. The override is either false (brake tripped) or NULL (no override). It is never set to true. "enabled" is expressed by clearing the override and letting the admin default through. So an admin flipping policy_updates_enabled on the control node propagates as a default change to every edge but never overwrites a local brake. That gives the admin a global pause/resume lever while leaving each edge's brake authoritative locally.

Operator-facing recovery. We plan to add a new runtime endpoint, UpdateGroupOverrides, that an operator calls against a specific edge node to clear (or set) that node's overrides. It is the only intended way to release a brake on an edge. The existing UpdateGroup admin endpoint will be guarded by requirePrimary in a later PR, so it cannot be used against edges. The new override endpoint will deliberately not be guarded to enable the per-node control.

Aggregated control-side view. The current proposal is one-way control → edge replication, with no edge → control channel for runtime state. As a consequence the control node does not show which edges have tripped brakes, and there is no central "clear everywhere" button beyond the admin default. We think the simpler story is worth it and edges keep a bit of autonomy, the admin default still works as a global lever, and fleet-wide visibility lives outside Nebraska, if needed.

Role bootstrap. The password is regenerated each startup and kept only in memory, which is a clean way to avoid storing it. What I want to understand is how that works with more than one instance. What does a second pod connect as, and what happens to the first when the second starts up?

The PoC's role bootstrap is single-pod. A second pod's startup rotates the shared role's password and breaks the first pod's open connections.

For the actual implementation I'd keep the in-process bootstrap but reshape it such that each mode gets a parent group role (nebraska_admin / nebraska_runtime, NOLOGIN) that owns the grants, and each pod creates its own LOGIN child role (nebraska_runtime_pod_<random-suffix>) IN ROLE <parent> so it inherits the parent's permissions. Pods don't share a password and can't clobber each other. Startup also sweeps any sibling child roles older than an hour with no live backends in pg_stat_activity, so unclean exits don't leak roles indefinitely.

An alternative is to push role provisioning onto the operator instead of NEBRASKA_DB_URL they'd pass admin or runtime-scoped credentials with the grants already set. That's simpler in Nebraska but breaks every time a migration adds a new runtime table and the operator forgets to update the grants, and it removes the transparent experience the PoC was aiming for. I'd lean on the per-pod-child-role approach for that reason.

Longer term I'm also looking at making the credential acquisition pluggable through pgx's BeforeConnect hook so hosters can wire in cloud-IAM auth (e.g. Entra tokens on Azure, RDS IAM on AWS, etc.) but that's a separate follow-up, beyond the current PoC.

Instance mode. How should a node handle a mode value it does not recognize? In a templated deploy the value can drift to something like Subscriber or replica, and I would not want that handled by accident.

You're right, the PoC treats the env var pretty loosely, but a real implementation needs to be stricter. I'll leave the specifics to the PR that actually introduces the mode flag, but the shape I'd lean toward is a small allowlist: unset (or "single") means single-instance, "primary" and "subscriber" map to those modes, and anything else fails the process at startup with a clear error. That way a templated deploy that drifts to "Subscriber" or "replica" crashes loudly instead of silently being treated as a primary.

Let me know if we need to have more concrete plans for any of those at this stage.