Observability Metrics Reference

Full AngaraBase metrics reference with diagnostic routes and a quick reference card. Canonical source: this runbook in angarabook/src/operations/.

Quick Reference Card (Top-10 for wallboard)

Print this and keep it near the on-call desk. These 10 metrics cover 80% of production incidents.

Full Metrics Reference

Connections and Sessions

Connection diagnostics:

SELECT pid, state, consumer_id, wait_event FROM angara_stat_activity;

Transactions and MVCC

PromQL — TPS:

rate(angarabase_txn_commit_total[1m])

PromQL — Conflict ratio:

rate(angarabase_txn_commit_conflicts_total[5m]) / rate(angarabase_txn_commit_total[5m])

WAL and durability

PromQL — WAL durability gap (bytes):

angarabase_transaction_log_flush_lsn - angarabase_transaction_log_durable_lsn

Storage and buffer pool

PromQL — Buffer pool hit ratio:

rate(angarabase_buffer_pool_hit_total[5m]) /
  (rate(angarabase_buffer_pool_hit_total[5m]) + rate(angarabase_buffer_pool_miss_total[5m]))

Checkpoint and bgwriter

SQL — bgwriter state:

SELECT * FROM angara_stat_bgwriter;

PromQL — checkpoint avg duration:

rate(angarabase_checkpoint_duration_ms_sum[5m]) / rate(angarabase_checkpoint_duration_ms_count[5m])

Query execution

PromQL — P99 latency:

histogram_quantile(0.99,
  rate(angarabase_query_exec_duration_ms_bucket[5m])
)

SQL — slow queries:

SELECT query, calls, mean_exec_time_ms, max_exec_time_ms
FROM angara_stat_statements
ORDER BY mean_exec_time_ms DESC LIMIT 10;

Memory

QoS Scheduler

Troubleshooting by Dashboard

Route 1: High P99 latency

angarabase_query_exec_duration_ms P99 > 500ms?
  │
  ├─ Yes → angara_stat_activity: any waiting sessions?
  │        │
  │        ├─ Yes (wait_event != '') → Lock contention or WAL sync wait
  │        │   → check angarabase_txn_commit_conflicts_total
  │        │   → check angarabase_wal_sync_wait_total
  │        │
  │        └─ No → angara_stat_statements: top queries by max_exec_time_ms
  │            → EXPLAIN the top query
  │            → check buffer_pool_miss_total rate (I/O bound?)
  │
  └─ No → baseline normal, false alarm

SQL:

SELECT query, calls, max_exec_time_ms, mean_exec_time_ms
FROM angara_stat_statements
ORDER BY max_exec_time_ms DESC LIMIT 5;

Route 2: QPS Drop (sudden SELECT rate drop)

rate(angarabase_query_exec_total_ok_select[1m]) dropped sharply?
  │
  ├─ connections_active also dropped → process restarted? uptime < 60s?
  │   → check logs for panic / OOM / segfault
  │
  ├─ connections_active high, QPS low → scheduler saturation?
  │   → qos_rejected_* > 0?
  │   → qos_blocking_inflight high?
  │   → spawn_blocking_active ≈ spawn_blocking_max?
  │
  └─ Connections normal → long transaction blocking?
      → angara_stat_activity WHERE state = 'idle in transaction'
      → txn_oldest_snapshot_age_seconds > 60s?

Route 3: GC Pressure / MVCC bloat

mvcc_history_versions_total grows monotonically without decrease?
  │
  ├─ txn_oldest_snapshot_age_seconds > 120s → long open snapshot
  │   → find pid from angara_stat_activity ORDER BY query_start ASC
  │   → terminate or wait for completion
  │
  ├─ columnar_pending_deleted_rows > 1M → compaction lagging
  │   → check Background Compactor in angara_stat_activity
  │   → temporarily SET angarabase.compaction_enabled = true
  │
  └─ memory_rss_bytes grows together → GC bloat + memory pressure
      → see mvcc-gc.md runbook

Route 4: Checkpoint Issues

checkpoint_errors_total changed?
  │
  ├─ Yes → inspect logs immediately (disk full? I/O error?)
  │   → storage_backpressure_events_total > 0?
  │   → df -h on data directory
  │
  └─ No, but dirty_pages_total high (> 10,000)?
      → checkpoint cannot keep up with writes
      → lower checkpoint_interval_ms
      → or limit write throughput
      → SQL: SELECT * FROM angara_stat_bgwriter;

Memory and Buffer Pool Metrics (RM-0.6.5.8)

Goal

Keep the minimum sufficient signal set for:

• durability;
• concurrency/locks;
• storage/checkpoint;
• recovery.

Metrics source

• ANGARABASE_METRICS_ADDR=host:port
• endpoint: GET /metrics (Prometheus format)

Must-have groups

• Transactions / concurrency
• Transaction log / durability
• Locks
• Storage / writeback / checkpoint
• Query diagnostics / stats
• Recovery / replay outcomes

Memory and Buffer Pool Metrics (RM-0.6.5.8)

PromQL — Alert when approaching soft limit:

# Replace <soft_limit_bytes> with soft_limit_mb * 1024 * 1024
# For example, for soft_limit_mb = 4096: threshold = 4294967296
angarabase_memory_rss_bytes > <soft_limit_bytes> * 0.9

Storage and Checkpoint Metrics (RM-0.6.5.8)

Visibility Map and Index-Only Scan (RM-0.6.4.3)

Specific metric names linked to dashboard panels: see the table. The full name contract is pinned by a test; link below in “Contract pinning”.

New RM-0.6.4.0 Metrics (WAL Commit Path + Durability)

Added in Sprint 2/3 RM-0.6.4.0 (RFC-2026-090). Cover the new sync_at_commit mode and the durability barrier group.

curl -sf http://127.0.0.1:9898/metrics | rg "wal_(sync_wait|group_commit_wait)|wait_events_total\\{event=\"wal_"

Diagnostics by mode

• relaxed: wal_sync_wait_total and wal_group_commit_wait_total are close to 0.
• group_commit: wal_group_commit_wait_total grows; wal_sync_wait_total is usually noticeably lower.
• sync_at_commit / strict: wal_sync_wait_total grows; wait_events_total{event="wal_sync"} reflects long-term sync-path load.

Durability mode is checked through env ANGARABASE_TRANSACTION_LOG_DURABILITY. SQL SET durability / COMMIT WITH DURABILITY are reserved for v0.6.5 → SQLSTATE 0A000. Details: WAL writer contract spec (wal_writer_contract_v0.md) and RFC-2026-090.

HTAP / Vector Execution Metrics (RM-0.6.4.13 / RM-0.6.4.14 / RM-0.6.6.9)

HTAP-specific metrics for diagnosing vector and stream execution paths. The label contract is stable starting with v0.6.x.

curl -sf http://127.0.0.1:9898/metrics | grep -E "scan_stream|vector_fallback|vector_memory|columnar_manifest|vector_columnar_native|columnar_batched_scan|segments_pruned|parallel_agg"

Note: reason= labels on angarabase_scan_stream_materialize_total are a stable operator-facing contract within v0.6.x.

Columnar DV Pressure (RM-0.6.4.19 Track C C2)

angarabase_columnar_pending_deleted_rows — signed gauge showing the total number of logically deleted rows in live segments that have not yet been reclaimed by compaction.

• Increment on AttachDeleteVector (on every columnar DELETE): +row_count from the DV op.
• Decrement on compact_l0_to_l1: -rows_reclaimed by number of rows not included in the L1 pack.

Normally, the gauge grows after DELETE and decreases after a Background Compactor run. If the gauge grows monotonically, compaction is lagging or fully disabled.

curl -sf http://127.0.0.1:9898/metrics | rg "pending_deleted_rows"

Alert rule (DV fragmentation)

# Alert if accumulated DV pressure > 5 million rows.
angarabase_columnar_pending_deleted_rows > 5_000_000

Recommended severity:

• warning when >1M rows — compaction is likely lagging;
• critical when >10M rows — scan performance degradation is possible.

Interpretation:

• gauge ≤ 0 — normal (all DV reclaimed, possibly a small transient underflow during replay);
• gauge grows without decrease for > 30 minutes — check Background Compactor (angara_stat_activity, angarabase_columnar_compaction_total).

Heap fetch fallback reason metrics (RM-0.6.5.6)

• angarabase_heap_point_fetch_fallback_reason_stale_tid_index_total — fallback due to stale tid index
• angarabase_heap_point_fetch_fallback_reason_not_found_total — fallback due to row not found

Quick check (curl):

curl -s http://localhost:8080/metrics | grep "fallback_reason"
# angarabase_heap_point_fetch_fallback_reason_stale_tid_index_total 0
# angarabase_heap_point_fetch_fallback_reason_not_found_total 0

PromQL — fallback rate by reason:

rate(angarabase_heap_point_fetch_fallback_reason_stale_tid_index_total[5m])
rate(angarabase_heap_point_fetch_fallback_reason_not_found_total[5m])

If stale_tid_index grows, there may be an issue with the V3 chain path or index rebuild. If not_found grows, data loss or an MVCC visibility bug is possible.

QoS Scheduler and spawn_blocking (RM-0.6.4.10 / RM-0.6.4.19)

RM-0.6.4.10 adds runtime signals for QoS scheduler and blocking path. They help distinguish SQL contention from scheduler saturation: if QoS rejections or qos_blocking grow, the problem is in execution queues, not in row/table locks.

curl -sf http://127.0.0.1:9898/metrics | rg "qos_(queued|rejected|blocking)|spawn_blocking"

QoS queues by level:

rate({__name__=~"angarabase_qos_queued_.*_total"}[5m])

QoS rejections by level:

rate({__name__=~"angarabase_qos_rejected_.*_total"}[5m])

Alert on any scheduler rejection:

sum(rate({__name__=~"angarabase_qos_rejected_.*_total"}[5m])) > 0

Blocking pressure:

angarabase_qos_blocking_inflight > 0

Blocking budget headroom:

angarabase_spawn_blocking_max - angarabase_spawn_blocking_active

Interpretation:

• queued_background_total grows but no rejected_* — scheduler accepts batch workload; usually normal;
• rejected_background_total grows — batch/ETL is too aggressive; lower concurrency or raise ANGARABASE_QOS_MAX_QUEUED;
• rejected_critical_total grows — production incident candidate: CRITICAL workload should not regularly hit the queue cap;
• qos_blocking_inflight > 0 together with growth in qos_blocking wait event means pressure in the blocking runtime path.

Query Execution Duration Histogram (RM-0.6.5.10)

angarabase_query_exec_duration_ms — histogram of SQL query execution latency.

Note (RM-0.6.5.10 S6): histogram_quantile(0.99) is correct only if the value is < 10,000ms. At p99 ≥ 10,000ms, inspect the share in bucket +Inf. Buckets: [1,5,10,50,100,500,1000,2500,5000,10000,+Inf] ms.

SLO-oriented usage

• Latency: histogram_quantile() over *_bucket (p95/p99)
• Throughput: rate() over counters
• Errors/contention: conflict/timeout/deadlock rates
• Saturation: backpressure counters and queue depth

Contract pinning

Must-have metric names are considered part of the operability contract and are protected by a test:

• crates/angarabase/src/metrics.rs
• prometheus_export_contains_must_have_metrics_names

Next

• Performance tuning guide — which metrics to read first during degradation.
• Parallel runtime observability runbook — narrow metrics for the parallel runtime.
• MVCC and GC operator minimum — separate MVCC/GC metrics and alerts package.