Runbook: `IndexRoutingLegacyFallback`

Source of truth: tools/observability/alerts/angarabase_alerts.yaml. Backed by: RM-0.6.3.8 S5 + S7. Synergy alert: binds Track 1 storage-correctness counter to Track 2 alerting layer.

What It Means

angarabase_index_routing_legacy_total{db="<db>"} > 0 post-upgrade — this instance has secondary indexes whose catalog records (IndexDef.index_db_name) do not yet contain the owning DB. They work correctly through legacy fallback to base.adb, but this means:

the index physically lives in base.adb instead of <db>.adb,
a backup of only <db>.adb will lose this index on restore,
RFC-2026-087 §4.1 invariant (per-DB co-location of pages) is violated.

Severity

warning. Not critical — data is not lost and the index works, but migration is required.

Why this alert fires after upgrade

RM-0.6.3.8 introduced per-DB IndexStore page routing. Binaries before RM-0.6.3.8 created IndexDef without the index_db_name field. After upgrade, such indexes decode with index_db_name = None and go through the legacy path, incrementing this counter.

Initial response

# Which DBs contain legacy indexes
curl -sf http://127.0.0.1:9898/metrics | rg index_routing_legacy_total

# List indexes on the problematic DB
psql -d <db> -c "SELECT schemaname, tablename, indexname FROM pg_indexes \
                 WHERE schemaname NOT IN ('pg_catalog','information_schema');"

Mitigation: DROP + CREATE INDEX

For each affected index, run (during a maintenance window):

\c <db>
DROP INDEX IF EXISTS public.<index_name>;
CREATE INDEX <index_name> ON public.<table> (<col>);

After recreation, the counter stops growing (the old IndexDef record is replaced with a new one that has the correct index_db_name = Some("<db>"); pages are written to <db>.adb).

Verification

# Counter should not grow after migration
watch -n 30 'curl -sf http://127.0.0.1:9898/metrics | rg index_routing_legacy_total'

And check the per-DB file size:

ls -lh <datadir>/<db>.adb
ls -lh <datadir>/base.adb   # should shrink after indexes move

Escalation

Not required — this is a planned migration, not an incident. If the counter grows without an upgrade, this is a bug; escalate.

Backup and restore — why legacy indexes broke backup
Upgrade and migration
RFC-2026-087 §4.1 + Addendum §S2 (RM-0.6.3.8)

Keyboard shortcuts

AngaraBook