MOS Self-Healing

A recovery layer for the MOS pipeline that finds silently stalled work and fixes it with bounded, idempotent retries before a human ever has to look.

Overview

MOS is a distributed pipeline: jobs fan out into shots, generative tasks, and After Effects renders on a Windows EC2 fleet, then delivery. Before this layer, recovery was manual, and most failures showed up as silence rather than errors: orphaned tasks, lost renders, stuck jobs.

Detect

• Scans for stuck state using per-step staleness TTLs.

Recover

• Orphaned tasks: drain-reset on graceful shutdown, plus a scheduler reaper for hard kills.
• Stranded and stale renders are found and re-queued.
• Transient production failures are classified and retried, capped at two attempts with a cooldown and a transactional lease.
• Render workers heal at the infrastructure level.

Escalate

• Deterministic or exhausted failures open a productionIncident with a named failure class and go to operators via Slack.
• A read-only reliability audit checks six cross-system invariants, such as credits, subscriptions, and delivered media.

Rules: sweeps never race each other, time alone never destroys work, and every repair is dry-run first, idempotent, and audited.

Stack

• RUNTIME: TypeScript 5, Node.js 20, pnpm monorepo
• COMPUTE: ECS Fargate (incl. Spot), EC2 Auto Scaling (Windows Server 2022)
• AWS: SSM, SQS, CloudWatch
• DATA: Firestore
• OPS: Slack webhooks, Winston, Jest