Disaster recovery

How to get Marketing OS back after the production database, Redis or a secret is lost, how much data that costs, and how long it takes. Each number here was either read from the Railway project or measured in a drill. The "Last drill" table at the bottom says which one.

Production is the Railway project marketing-keyescode, environment production (see DEPLOYMENT.md). It has four services: @marketing-os/api, @marketing-os/web, Postgres (image ghcr.io/railwayapp-templates/postgres-ssl:18, with extensions vector, citext, pgcrypto and uuid-ossp) and Redis. It also has one storage bucket, assets.

#The numbers

To read the schedules again, open the Railway dashboard → Postgres → Backups. If you have a CLI token, you can also query volumeInstanceBackupScheduleList on volume instance 589fffac-d80a-4e53-9e39-9235337c17db (Postgres) or 6a6871ad-4cd1-4cc5-998d-d576ee274f5f (Redis, see below).

#Before you touch anything

1. Never restore over the live database unless it is already gone. The copy paths (A, C) restore into a new service. A copy that turns out wrong costs nothing, while an overwrite that turns out wrong costs everything.
2. Stop the writers first if the data is being damaged right now, for example by a bad migration or a runaway job. Scale @marketing-os/api to 0 replicas (Settings → Deploy). All crons and BullMQ workers run inside the API process, so this stops everything that writes.
3. Take a copy of whatever is still there before you change it: pg_dump per path A, step 1, even from a damaged database.
4. Note the time. It is your incident start and your drill number.

#Path A — logical copy into a new Postgres (drilled)

Use this when the source database is still readable: moving hosts, recovering rows a bad deploy damaged, or the drill. It loses nothing written before the dump started.

The local pg_dump must match the server's major version, which is 18, so run the client from a postgres:18 container. The API's DATABASE_URL is the private *.railway.internal host, which is unreachable from a laptop, so use the Postgres service's DATABASE_PUBLIC_URL. Run all of this from a directory linked with railway link --project marketing-keyescode --environment production.

1. Dump (read-only against the source):

   sh

   mkdir -p ./dr && railway run --service Postgres -- sh -c \
     'docker run --rm -e U="$DATABASE_PUBLIC_URL" -v "$PWD/dr":/out postgres:18 \
        pg_dump --format=custom --no-owner --no-acl --file=/out/prod.dump "$U"'

2. Record the "before" numbers from the source:

   sh

   railway run --service Postgres -- sh -c \
     'DATABASE_URL="$DATABASE_PUBLIC_URL" pnpm --filter @marketing-os/api dr:smoke'

3. Create the new Postgres. For a drill, put it in a separate scratch project (railway init --name dr-drill-YYYYMMDD, then railway add --database postgres). That way nothing can appear in, or be staged into, production. For a real cut-over, add it to marketing-keyescode itself. Check that the new service's image is postgres-ssl:18 or newer and that SELECT * FROM pg_available_extensions WHERE name = 'vector' returns a row. Give it a TCP proxy on port 5432 (Settings → Networking) so a laptop can reach it. The URL is then postgresql://$PGUSER:$PGPASSWORD@<proxy-host>:<proxy-port>/$PGDATABASE.

4. Restore:

   sh

   docker run --rm -e U="<new public url>" -v "$PWD/dr":/out postgres:18 \
     pg_restore --no-owner --no-acl --exit-on-error -d "$U" /out/prod.dump

5. Migrate. This is a no-op when prod was already at head, and it brings an older snapshot forward:

   sh

   pnpm --filter @marketing-os/api build
   DATABASE_URL="<new public url>" pnpm --filter @marketing-os/api migration:run:prod

6. Smoke the data: DATABASE_URL="<new public url>" pnpm --filter @marketing-os/api dr:smoke. The counts must equal step 2's, and so must the newest metrics_daily day and the newest migration.

7. Point an API at it.

   • Drill: run the built API locally with its own throwaway Redis, never a shared one. Leave SENTRY_DSN out: an empty string fails env validation.

     sh

     docker run -d --name dr-drill-redis -p 6391:6379 redis:7-alpine
     DATABASE_URL="<new public url>" REDIS_URL=redis://localhost:6391 PORT=4791 \
       node apps/api/dist/main.js

     The local .env supplies the remaining secrets. Its TOKEN_ENCRYPTION_KEY is not prod's, so stored OAuth tokens will not decrypt. That is fine for the smoke, which reads only synced data, and it also means the crons cannot reach a real ad account.

   • Real cut-over: on @marketing-os/api, change DATABASE_URL to a reference to the new service (${{NewPostgres.DATABASE_URL}}), then deploy. railway.json runs migration:run:prod at start.

8. Smoke the app.

   • GET /health returns 200 with "status":"ok", and both "db":"up" and "redis":"up". It answers 503 when either one is down. At the drill, /health checked only Postgres.
   • Sign in. POST /auth/login returns 200, then GET /auth/me returns 200.
   • Load one business's ads manager: GET /organizations/:orgId/businesses/:businessId/entities/campaigns?from=…&to=… returns rows with non-zero metrics.

   In a drill you know no production password, so insert a throwaway user into the copy only. Hash its password with argon2.hash, set emailVerifiedAt, and give it an owner membership in an organization that has metrics_daily rows. The drill copy is deleted afterwards anyway.

9. Tear down (drill): stop the API, docker rm -f dr-drill-redis, delete the scratch Railway project, and delete prod.dump. It holds every customer's data, encrypted tokens included, so it must not linger on a laptop.

#Path B — Railway snapshot restore (not yet drilled)

Use this when the Postgres volume itself is lost or corrupt, or when the data has to go back to a snapshot. It loses up to 24 hours (the RPO).

Railway restores a volume backup only into the same service. Postgres → Backups → pick the snapshot → Restore stages a new volume, named after the snapshot's date, mounted at /var/lib/postgresql/data. The previous volume is kept, unmounted, so the step can be undone. Review the staged change, then click Deploy. @marketing-os/api keeps the same DATABASE_URL, because it is a reference to the same service. Restart the API, then run path A steps 5, 6 and 8 against prod.

This path has not been drilled, because the only way to drill it is on the production service. The next drill should do one of these:

• Restore a snapshot into a duplicate environment. Railway backups can be restored only within the same project and environment, so first check whether a duplicated environment carries the volume's backups.
• Enable PITR (path C) and drill that instead.

#Path C — point-in-time recovery (not enabled)

Railway can archive Postgres WAL continuously to a bucket (pgBackRest). It then restores to any second in about the last 4 weeks, into a new service (Postgres-restored-YYYYMMDD-HHMM), and never touches the source. That would cut the RPO from 24 hours to about a minute and turn path B into a copy instead of an overwrite. Enabling it redeploys Postgres (a short blip) and adds a bucket's storage cost, so it is the owner's call. It is off today.

#Redis — what is lost and what comes back

Redis holds only BullMQ queues and the per-credential rate-limit counters. Sessions are Postgres rows, cron locks are Postgres advisory locks, and there is no cache.

Redis has a volume (redis-volume, volume instance 6a6871ad-4cd1-4cc5-998d-d576ee274f5f) with a daily Railway backup schedule at 18:39 UTC, kept 6 days. It was added 2026-09-24. Before that, the only snapshot was a manual one from 2026-09-12. Even so, treat Redis as disposable: the default recovery is a fresh, empty Redis plus the clean-up below. A snapshot restore is best-effort. It may bring back queued jobs, but it would also replay jobs that already ran since the snapshot, such as posts published or budgets changed. Use it only when you have checked that replaying them is safe, and then run the same clean-up.

Nothing recurring lives in Redis. There are 22 queues and no BullMQ repeatables. Every schedule is either an in-process @nestjs/schedule cron behind a Postgres advisory lock or one of four older setInterval timers. All of them re-derive their work from Postgres (see scheduled jobs and ADR 0004). What a wipe loses is the jobs that were queued and not yet run. Several features mark a Postgres row in-flight before they enqueue, and nothing un-sticks those rows by itself.

Rebuilt automatically, so no action is needed:

• Scheduled social posts still at scheduled. The time lives in social_posts.scheduledAt, and the per-minute sweeper, plus a 10-second boot pass, picks them up.
• The hourly syncs (Google Ads and TikTok Ads incrementals, Shopify), the conversions they re-offer (orders up to 7 days old), the daily audience sync, the daily digest and the weekly competitor refresh. Each returns on its next tick.
• Recommendation batches stuck at generating stop blocking after 24 hours.
• Rate-limit counters simply reset. The limiter also fails open while Redis is down.

Lost, needing a SQL fix and a re-trigger. Find the rows stuck in-flight with an updatedAt from before the wipe, mark them failed, and then re-trigger them from the app:

Also lost for good: social-post metrics checkpoints, a delayed job per post, so the +72-hour outcome fold never happens for posts published in that window. Knowledge embeddings are also lost; re-run pnpm --filter @marketing-os/api backfill:embeddings.

#Secrets

Both of these live in Railway's variables on @marketing-os/api, and are escrowed outside Railway too (see "Key escrow" below). Railway keeps no history of a variable, so if one is overwritten or the project is lost, put the escrow copy back.

• TOKEN_ENCRYPTION_KEY lost = every stored credential is unrecoverable. AES-256-GCM (common/crypto.service.ts) encrypts every *Encrypted column. That covers Meta, social, Google (GA4/GSC and Ads), TikTok Ads, Shopify and Local Falcon tokens, customers' own model API keys, Slack webhook URLs, outbound webhook URLs and signing secrets, publish-target secrets, and checkup tokens. A database restore does not bring them back without this key. Every customer would have to reconnect every integration and re-issue every webhook secret. Do not rotate it casually either: rotation means exactly the same thing.
• JWT_SECRET lost or rotated = recoverable, with some disruption. Everyone's access tokens (15 min) stop verifying, but refresh sessions are Postgres hashes, so users are signed back in silently. Outstanding export download links, signed asset and thumbnail URLs, and any OAuth connect in progress stop working. The per-visitor free-tool quotas, which are salted with it, reset.
• Provider secrets (META_APP_SECRET, SHOPIFY_API_SECRET, STRIPE_WEBHOOK_SECRET, OAuth client secrets, RESEND_API_KEY) can be re-issued from each provider's dashboard.

#Key escrow

This table records where each copy is, never the values. Update it whenever a secret is rotated, because the escrow copy has to be replaced at the same time.

To restore one, copy it from 1Password into @marketing-os/api → Variables, then redeploy. Never paste a value into a ticket, a log or chat.

#Uploaded files (assets bucket)

Uploaded images, generated creative and data-export zips live in the Railway storage bucket assets (S3_* on the API). Railway buckets have no snapshot or backup feature, so a lost bucket means lost uploads. Export zips are disposable (24-hour links). The database rows that reference a lost upload survive, but the files behind them are gone. This is a known gap and is not covered by the RPO above.

#DNS and rollback

• Domains. marketing.keyescode.com (web) and marketing-api.keyescode.com (API) are CNAMEs in the keyescode.com DNS to Railway targets. Keep them DNS-only if Cloudflare proxies the zone. Cookies ride COOKIE_DOMAIN=.keyescode.com. A restore that stays inside Railway changes no DNS, because the services keep their domains. Moving hosts means re-adding the custom domains on the new services, updating both CNAMEs, and waiting for the certificate. Then re-register every OAuth redirect URI and Meta callback listed in DEPLOYMENT.md §4, because they are bound to these hostnames.
• Code rollback. In Railway → the service → Deployments, redeploy the last good build. Migrations are forward-only in prod, so rolling code back past a migration leaves the newer schema in place. That is fine for additive migrations. For a destructive one, restore the data (path A, or B) instead of rolling back.
• Undoing a restore. Path A leaves the old database untouched: point DATABASE_URL back at it. Path B leaves the old volume unmounted in the project: re-mount it the same way.

#Last drill

Row counts (from dr:smoke):

Notes from this drill:

• About 40 seconds of the clock went to two operator slips: creating an environment in the production project before switching to a separate project, and an empty SENTRY_DSN that failed env validation. The runbook above now avoids both.
• A new Railway Postgres has no public URL until it gets a TCP proxy. DATABASE_PUBLIC_URL is empty until then.
• Not exercised: a Railway snapshot restore (path B), prod's TOKEN_ENCRYPTION_KEY decrypting a restored token, and a Railway-hosted API redeploy onto the new database. The RTO adds about 6 minutes for that last one, based on the recent deploy duration.

Drill again every quarter, after any change to the database image or host, or once the database passes 1 GB. Add a new row above; don't replace this one.