observability: promote slot-read profiling into request-scoped NDJSON telemetry

[Jesssullivan]

Why

#40 gave us phase-level slot-read profiling, and live beta checks now show the remaining tail is a generalized cold first-read cost rather than a later-date-specific outlier.

What is still missing is a durable observability surface:

• hot-path HTTP handlers still emit ad-hoc console.log / console.error
• slot-read profiles are not correlated with request-level context
• there is no consistent event schema for perf analysis across /health, /availability/*, and booking flows
• Modal logs are usable, but not yet structured enough for reliable phase analysis or downstream ingestion

Goals

• route hot-path bridge logs through the NDJSON logger instead of raw console output
• add request-scoped context where practical:
  • endpoint
  • service id / resolved service name
  • date / datetime
  • request id or correlation token
  • runtime metadata (modalEnvironment, release sha/version, flow owner)
• emit slot-read profile events as structured NDJSON records with a stable event name/schema
• distinguish cold-path events from warm-path events explicitly
• make the logs legible for both Modal tailing and future external ingestion

Deliverables

• replace ad-hoc availability/service logs in src/server/handler.ts and related hot paths with ndjsonLog or the shared structured logger
• define a stable event taxonomy for:
  • request started
  • request completed
  • request failed
  • slot profile long-tail
  • service catalog resolution
• document the env knobs for profiling / forced logging in the README
• add focused tests for log formatting/helpers where appropriate

Success Criteria

• a cold /availability/slots read can be traced in logs with enough structure to separate navigation, settle, wait, DOM-read, and parse time
• downstream log analysis does not depend on parsing human prose strings
• Modal runtime logs become a reliable evidence source for #148-style perf investigation and future paper material

Related

• follows #20
• supports #10
• should inform future runtime portability work beyond Modal

[Jesssullivan]

First implementation slice is up in #45.

PR: #45

What it covers:

• request-scoped NDJSON telemetry in the middleware server
• correlation fields on slot-read profile events
• structured startup/error/service-catalog logging on the hot path
• README observability env/docs cleanup

This is the right first step for Modal log analysis because it improves signal without changing deploy shape or runtime behavior.

[Jesssullivan]

First implementation slice merged in #45 at 84bb7f0.

What landed:

• request-scoped NDJSON telemetry in the middleware server
• requestId correlation on slot-read profile events
• structured startup/error/service-catalog logging on the hot path
• README observability env/docs cleanup

This should make Modal log evidence materially more useful for the next cold-path analysis pass.

[Jesssullivan]

Bridge telemetry is now live on beta via 84bb7f0.

Important cross-repo constraint discovered during rollout verification:

• the app path (beta.massageithaca.com/api/schedule/*) does not currently preserve a caller-supplied x-request-id into the bridge
• MassageIthaca uses a cached remote scheduling adapter with static headers (Authorization only), so request-scoped correlation is not available through the app path today
• the app response also does not surface the bridge request id back out

Practical implication:

• the next real evidence loop for cold-path phase analysis requires either direct bridge access/log tailing, or a future cross-repo request-correlation change
• repeated app-side curls alone can confirm latency shape, but not correlate a specific request to Modal log lines

That is a good reason to keep this issue open even after #45: the telemetry is deployed, but the evidence-gathering workflow is only fully unlocked for direct bridge access right now.

[Jesssullivan]

Direct bridge verification after 84bb7f0:

• the live direct bridge URL is public and healthy
• direct GET /health now reports release 84bb7f0
• the direct Modal edge preserves caller-supplied x-request-id
• the direct response also includes modal-function-call-id

That means the bridge-side observability story is now good enough for direct runtime investigation.

The remaining limitation is app-path correlation, not bridge-path correlation. A paired MassageIthaca issue now tracks forwarding request-scoped correlation ids into the remote bridge path.