Incident Report: Generated Media Send Outage

Summary

On July 6, 2026, generated voice/photo action results stopped sending from production bridges even though media generation webhooks continued to complete. Text replies continued to work.

The immediate cause was a production Railway environment variable:

SPECTRUM_ACTION_OUTBOUND_POLL_MS=2147483647

That value is about 24.8 days. The bridge accepted it as a valid finite integer and used it as the interval for the async action-result outbound drain. In practice, this disabled generated-media result sending.

The outage began immediately after the production bridge services redeployed around 2026-07-06 02:06 PDT. Those deployments picked up the bad environment value. The value is not present in the repo, and the code default is 2000 ms.

Impact

At 2026-07-07 00:02:46 PDT, there were 372 actionable stale generated-media jobs left:

At 2026-07-07 00:04 PDT, 366 remaining stale action-result:% media jobs were manually cancelled to avoid out-of-context sends. The delta from 372 to 366 was due to live bridge drain movement between measurement and cancellation.

Earlier stale cleanup had already cancelled 256 jobs with:

stale-cleanup 2026-07-06: pending>30m, no active sending chain

Timeline

All times are America/Los_Angeles unless noted.

What Happened

Generated media actions are asynchronous:

1. The agent requests media generation. 2. A child app/action webhook acknowledges the result. 3. The result is inserted into photon_outbound_jobs with a client_guid like action-result:<eventId>. 4. scripts/spectrum-bridge.ts runs drainCachedOutboundJobs() on an interval. 5. The drain claims pending action-result jobs and calls sendOutboundContent().

The bridge interval is configured by:

SPECTRUM_ACTION_OUTBOUND_POLL_MS

The code default is:

DEFAULT_ACTION_OUTBOUND_POLL_MS = 2_000

But production bridge containers were running with:

SPECTRUM_ACTION_OUTBOUND_POLL_MS=2147483647

Because the env parser accepted any finite number and only clamped to a minimum of zero, this value was accepted. The resulting setInterval() was effectively scheduled once every 24.8 days, so drainCachedOutboundJobs() did not run during normal operation.

Media generation itself was healthy. Prod action_events after the outage start showed generated-media action events being acknowledged:

The failure was downstream: generated media result jobs were created but not drained/sent.

Why It Started Then

The variable and drain code existed before the outage. The drain code was introduced on 2026-06-28 with a 2 second default. Generated media continued sending successfully until shortly before the 2026-07-06 02:06 PDT redeploy.

The correlated trigger was the bridge redeploy around 02:06 PDT:

Those deployments used commit 51700a4, whose message was Replace dead Railway-deploy Action with test-only CI. That commit did not change media sending or bridge drain behavior. The likely sequence is:

1. The bad Railway env var existed in production config. 2. Existing containers were likely still running from an earlier env snapshot. 3. The 02:06 redeploy recreated bridge containers. 4. New containers picked up SPECTRUM_ACTION_OUTBOUND_POLL_MS=2147483647. 5. Generated media action-result jobs stopped draining.

The Railway CLI does not expose variable edit history or audit metadata. The remaining unknown is who or what set SPECTRUM_ACTION_OUTBOUND_POLL_MS=2147483647, and when. That must be checked in Railway dashboard/audit logs if available.

Detection

The incident was detected by user report: image/voice generation and sending appeared down in production.

Diagnosis showed:

Resolution

Immediate mitigation:

1. Set SPECTRUM_ACTION_OUTBOUND_POLL_MS=2000 on all production bridge services. 2. Redeployed:

Manual cancellation reason:

manual-cancel 2026-07-07 incident: stale generated-media backlog; avoid out-of-context resend

Root Cause

Production bridge services had an unsafe environment value for the action-result drain interval:

SPECTRUM_ACTION_OUTBOUND_POLL_MS=2147483647

The bridge accepted the value without an upper bound or startup warning. This disabled the async generated-media outbound drain for practical purposes.

Contributing Factors

Corrective Actions

P0:

P1:

P2:

Status

Mitigated.

Current production bridge runtime has SPECTRUM_ACTION_OUTBOUND_POLL_MS=2000, and the stale generated-media backlog from the incident has been cancelled.