Monitoring

Goal

Connect Prometheus and Grafana to AngaraBase metrics and get a working dashboard for ops/triage monitoring.

Prerequisites

Running angarabase-server (see Quickstart)
Access to the AngaraBase metrics listener (ANGARABASE_METRICS_ADDR)
Running Prometheus instance
Running Grafana instance

Step 1: enable the metrics endpoint

Set the listener address in the config file:

[ops]
metrics_addr = "127.0.0.1:9898"

Or use the environment variable override (takes precedence over config):

export ANGARABASE_METRICS_ADDR=127.0.0.1:9898
angarabase-server --config /etc/angarabase/angarabase.conf

Verify the endpoint responds:

curl -sS http://127.0.0.1:9898/metrics | rg '^angarabase_' -m 5
curl -sS http://127.0.0.1:9898/health/live
curl -sS http://127.0.0.1:9898/health/ready
curl -sS http://127.0.0.1:9898/health/startup

See Configuration — Ops knobs for the full [ops] section reference.

OpenTelemetry tracing

AngaraBase also exposes opt-in OTel-style span export for query lifecycle triage.

Minimal setup example:

export ANGARABASE_OTEL_ENABLED=1
export ANGARABASE_OTEL_EXPORTER=file
export ANGARABASE_OTEL_ENDPOINT=artifacts/otel/spans.jsonl
export ANGARABASE_OTEL_SAMPLE_RATE_PPM=1000000

Spans are emitted for bounded stage names (accept, auth, session, parse, plan, execute, storage_io, commit/rollback) and are intended for diagnostics evidence, not for raw SQL capture.

Step 2: configure the Prometheus scrape

Add a job to prometheus.yml:

scrape_configs:
 - job_name: angarabase
 scrape_interval: 15s
 metrics_path: /metrics
 static_configs:
 - targets: ["127.0.0.1:9898"]

Restart Prometheus and verify in the UI (/targets) that the angarabase target is UP.

Step 3: add the Prometheus data source in Grafana

Open the Grafana UI.
Navigate to Connections → Data sources.
Add Prometheus.
Set the URL to your Prometheus instance (e.g., http://127.0.0.1:9090).
Click Save & test and confirm the connection succeeds.

Step 4: import the AngaraBase dashboard

Go to Dashboards → New → Import.
Upload the file tools/observability/grafana/angarabase-overview-v2.json.
Select the Prometheus data source added in the previous step.
Save the dashboard.

Alternative — download the JSON directly from the server:

curl -fsS http://127.0.0.1:9898/grafana/angarabase-overview.json -o angarabase-overview.json

Then import angarabase-overview.json in Grafana.

Expected result

After import, the dashboard should display the following panels:

Panel	Key metric(s)
QPS	`angarabase_queries_total`
Query latency p50/p95/p99	histogram buckets
Slow query count	`angarabase_slow_query_total`
Active connections	gauge
TPS / commit latency	`angarabase_commits_total`
Lock contention	lock wait counters
Buffer pool pressure	`angarabase_buffer_pool_hit_total`, `angarabase_buffer_pool_miss_total`
BRIN range efficiency	`angara_brin_range_efficiency`
Mutation policy rejections	`angara_table_no_delete_rejected_*`, `angara_table_mutation_epoch`
WAL lag / fsync latency	WAL counters
IO latency / checkpoint / GC pressure	IO counters

If panels are empty, see Troubleshooting below.

Troubleshooting

No data in panels

Check http://<prometheus>/targets: the angarabase target must be UP.
Ensure ANGARABASE_METRICS_ADDR matches the targets value in Prometheus.
Verify metrics use the angarabase_ prefix:

curl -sS http://127.0.0.1:9898/metrics | rg '^angarabase_' -m 20

Data source test fails in Grafana

Verify the data source URL and network reachability from Grafana to Prometheus.
For Docker/K8s use the service DNS/hostname (not 127.0.0.1) when Grafana and Prometheus run in different containers.

Readiness probe is not `ready`

GET /health/ready returns a reason in JSON — this is the primary triage signal.
Collect a diagnostics bundle and attach summary.json:

tools/diagnostics_bundle/run.sh \
 --root artifacts/diagnostics/<incident-id> \
 --config <path> \
 --metrics-url http://<metrics-host>/metrics

For further investigation see Diagnostics and Support.

AngaraBook