AngaraBase USDT Probes Operations Guide

Audience: Database operators, performance engineers, eBPF developers

Overview

AngaraBase включает User Statically-Defined Tracing (USDT) probes — zero-overhead instrumentation points для интеграции с eBPF-инструментами (bpftrace, bcc, pg_expecto).

Key Benefits:

• Zero overhead when not attached (NOP instructions)
• Real-time attachment без рестарта процесса
• Корреляция engine-level событий с OS-level метриками
• Совместимость с существующими eBPF toolchains

Probe Taxonomy: 12 probe points покрывают query lifecycle и wait events.

Probe Architecture

Probe Categories

Probe Signature

All probes follow consistent ABI:

// Start probes (2 arguments)
probe angarabase:query_start(uint64_t session_id, uint64_t query_fingerprint);
probe angarabase:lock_wait_start(uint64_t session_id, uint8_t lock_type);

// End probes (3-4 arguments) 
probe angarabase:query_end(uint64_t session_id, uint64_t query_fingerprint, 
 uint64_t duration_us, uint8_t outcome);
probe angarabase:lock_wait_end(uint64_t session_id, uint64_t wait_duration_us);

Probe Locations

Probes fire at same instrumentation points as WaitEventGuard RAII:

#![allow(unused)]
fn main() {
// Example: Lock wait instrumentation
let _guard = WaitEventGuard::enter(session_id, WaitEventType::Lock, WaitEvent::RowLock);
// probe_lock_wait_start! fires here

// Blocking operation (lock acquisition)
let result = lock_manager.acquire_lock(&key);

// probe_lock_wait_end! fires when _guard drops
}

Compilation and Feature Flags

Build Configuration

Default build (probes enabled):

cargo build --release
# USDT probes compiled in (zero overhead when not attached)

Minimal build (probes excluded):

cargo build --release --no-default-features
# No USDT probes, smaller binary

Verify probe compilation:

# Check probes are embedded
readelf -n ./target/release/angarabase-server | grep -A5 "stapsdt"

# List available probes
bpftrace -l 'usdt:./target/release/angarabase-server:angarabase:*'

Expected output:

usdt:./target/release/angarabase-server:angarabase:query_start
usdt:./target/release/angarabase-server:angarabase:query_end
usdt:./target/release/angarabase-server:angarabase:phase_start
usdt:./target/release/angarabase-server:angarabase:phase_end
usdt:./target/release/angarabase-server:angarabase:lock_wait_start
usdt:./target/release/angarabase-server:angarabase:lock_wait_end
usdt:./target/release/angarabase-server:angarabase:io_start
usdt:./target/release/angarabase-server:angarabase:io_end
usdt:./target/release/angarabase-server:angarabase:net_stall_start
usdt:./target/release/angarabase-server:angarabase:net_stall_end
usdt:./target/release/angarabase-server:angarabase:sched_wait_start
usdt:./target/release/angarabase-server:angarabase:sched_wait_end

bpftrace Examples

Query Tags

query_tag is a u64 hash (xxh64) propagated through all probe fires for a query/session. You can set it per-session:

SET query_tag = 'demo-heavy-report';

or per-query:

/*+ angarabase:tag=demo-heavy-report */ SELECT * FROM orders;

The scripts below filter on “tag != 0” (tagged traffic only), so regular traffic keeps zero-overhead behavior.

# 1) Lock + IO latency by tagged query
bpftrace -e '
 usdt:./angarabased:angarabase:lock_wait_end /arg2 != 0/ {
 @lock_wait_us_by_tag[arg2] = hist(arg1);
 }
 usdt:./angarabased:angarabase:io_end /arg2 != 0/ {
 @io_wait_us_by_tag[arg2] = hist(arg1);
 }
'

# 2) Per-operator breakdown (fires only for tag != 0)
bpftrace -e '
 usdt:./angarabased:angarabase:operator_end /arg1 != 0/ {
 @op_duration_us[arg1, arg2] = hist(arg4);
 @op_rows_out[arg1, arg2] = sum(arg3);
 }
'

# 3) Compare p99 across tagged query cohorts (A/B)
bpftrace -e '
 usdt:./angarabased:angarabase:query_end /arg4 != 0/ {
 @query_total_us_by_tag[arg4] = hist(arg2);
 @query_rows_by_tag[arg4] = sum(arg5);
 }
'

Basic Query Monitoring

1. Query Throughput and Latency

#!/usr/bin/env bpftrace
# query_latency.bt - Monitor query execution time

usdt:./angarabase-server:angarabase:query_start {
 @start[arg0] = nsecs; // session_id -> start_time
 @queries++;
}

usdt:./angarabase-server:angarabase:query_end {
 $duration_us = arg2; // duration from probe
 $outcome = arg3; // 0=Ok, 1=Error
 
 delete(@start[arg0]);
 
 if ($outcome == 0) {
 @latency_us = hist($duration_us);
 @successful++;
 } else {
 @failed++;
 }
}

interval:s:5 {
 printf("\n=== Query Stats (5s) ===\n");
 printf("Total queries: %d\n", @queries);
 printf("Successful: %d\n", @successful);
 printf("Failed: %d\n", @failed);
 print(@latency_us);
 
 clear(@queries); clear(@successful); clear(@failed);
}

END {
 clear(@start); clear(@latency_us);
}

2. Per-Phase Timing Breakdown

#!/usr/bin/env bpftrace
# phase_timing.bt - Analyze query phase performance

usdt:./angarabase-server:angarabase:phase_start {
 $session_id = arg0;
 $phase = arg1; // 1=parse, 2=plan, 3=execute, 4=commit
 @phase_start[$session_id, $phase] = nsecs;
}

usdt:./angarabase-server:angarabase:phase_end {
 $session_id = arg0;
 $phase = arg1;
 $duration_us = arg2;
 
 delete(@phase_start[$session_id, $phase]);
 
 if ($phase == 1) {
 @parse_time = hist($duration_us);
 } else if ($phase == 2) {
 @plan_time = hist($duration_us);
 } else if ($phase == 3) {
 @exec_time = hist($duration_us);
 } else if ($phase == 4) {
 @commit_time = hist($duration_us);
 }
}

interval:s:10 {
 printf("\n=== Phase Timing Distribution ===\n");
 printf("Parse time (us):\n"); print(@parse_time);
 printf("Plan time (us):\n"); print(@plan_time);
 printf("Execute time (us):\n"); print(@exec_time);
 printf("Commit time (us):\n"); print(@commit_time);
}

END {
 clear(@phase_start);
 clear(@parse_time); clear(@plan_time); 
 clear(@exec_time); clear(@commit_time);
}

Lock Contention Analysis

3. Lock Wait Time Distribution

#!/usr/bin/env bpftrace
# lock_contention.bt - Monitor lock contention

usdt:./angarabase-server:angarabase:lock_wait_start {
 $session_id = arg0;
 $lock_type = arg1; // Lock type enum
 @lock_start[$session_id] = nsecs;
 @lock_requests[$lock_type]++;
}

usdt:./angarabase-server:angarabase:lock_wait_end {
 $session_id = arg0;
 $wait_duration_us = arg1;
 
 delete(@lock_start[$session_id]);
 
 @lock_wait_time = hist($wait_duration_us);
 
 if ($wait_duration_us > 1000) { // > 1ms
 @slow_locks++;
 printf("SLOW LOCK: session=%d wait=%d us\n", $session_id, $wait_duration_us);
 }
}

interval:s:5 {
 printf("\n=== Lock Contention Stats ===\n");
 printf("Lock requests by type:\n"); print(@lock_requests);
 printf("Lock wait time distribution (us):\n"); print(@lock_wait_time);
 printf("Slow locks (>1ms): %d\n", @slow_locks);
 
 clear(@lock_requests); clear(@slow_locks);
}

END {
 clear(@lock_start); clear(@lock_wait_time);
}

I/O Performance Monitoring

4. Storage I/O Latency

#!/usr/bin/env bpftrace
# io_latency.bt - Monitor storage I/O performance

usdt:./angarabase-server:angarabase:io_start {
 $session_id = arg0;
 $op_type = arg1; // I/O operation type
 $bytes = arg2; // Bytes to read/write
 
 @io_start[$session_id] = nsecs;
 @io_bytes += $bytes;
 @io_ops++;
}

usdt:./angarabase-server:angarabase:io_end {
 $session_id = arg0;
 $latency_us = arg1;
 
 delete(@io_start[$session_id]);
 
 @io_latency = hist($latency_us);
 
 if ($latency_us > 10000) { // > 10ms
 @slow_io++;
 printf("SLOW I/O: session=%d latency=%d us\n", $session_id, $latency_us);
 }
}

interval:s:5 {
 printf("\n=== I/O Performance Stats ===\n");
 printf("Total I/O ops: %d\n", @io_ops);
 printf("Total I/O bytes: %d\n", @io_bytes);
 printf("I/O latency distribution (us):\n"); print(@io_latency);
 printf("Slow I/O ops (>10ms): %d\n", @slow_io);
 
 clear(@io_ops); clear(@io_bytes); clear(@slow_io);
}

END {
 clear(@io_start); clear(@io_latency);
}

Advanced Use Cases

Correlation with System Events

5. Engine + Kernel Correlation

#!/usr/bin/env bpftrace
# engine_kernel_correlation.bt - Correlate AngaraBase events with kernel

#include <linux/sched.h>

// Track AngaraBase query starts
usdt:./angarabase-server:angarabase:query_start {
 @query_pids[pid] = 1;
 @query_start_time[arg0] = nsecs; // session_id -> start_time
}

// Track context switches for AngaraBase processes
tracepoint:sched:sched_switch {
 $prev_pid = args->prev_pid;
 $next_pid = args->next_pid;
 
 // Track when AngaraBase process gets scheduled out
 if (@query_pids[$prev_pid]) {
 @context_switches[$prev_pid]++;
 }
}

// Track page faults for AngaraBase processes 
tracepoint:exceptions:page_fault_user {
 if (@query_pids[pid]) {
 @page_faults[pid]++;
 }
}

usdt:./angarabase-server:angarabase:query_end {
 $session_id = arg0;
 $duration_us = arg2;
 
 delete(@query_start_time[$session_id]);
 
 printf("Query session=%d duration=%d us ctx_switches=%d page_faults=%d\n",
 $session_id, $duration_us, @context_switches[pid], @page_faults[pid]);
}

END {
 clear(@query_pids); clear(@query_start_time);
 clear(@context_switches); clear(@page_faults);
}

Custom Aggregations

6. Top Slow Queries by Fingerprint

#!/usr/bin/env bpftrace
# top_slow_queries.bt - Track slowest queries by fingerprint

usdt:./angarabase-server:angarabase:query_end {
 $session_id = arg0;
 $query_fingerprint = arg1;
 $duration_us = arg2;
 $outcome = arg3;
 
 if ($outcome == 0 && $duration_us > 1000) { // Successful queries > 1ms
 @slow_queries[$query_fingerprint] = hist($duration_us);
 @query_count[$query_fingerprint]++;
 @total_time[$query_fingerprint] += $duration_us;
 }
}

interval:s:30 {
 printf("\n=== Top 10 Slowest Query Fingerprints ===\n");
 print(@slow_queries, 10);
 
 printf("\n=== Query Execution Counts ===\n");
 print(@query_count, 10);
 
 printf("\n=== Total Time by Fingerprint (us) ===\n");
 print(@total_time, 10);
}

END {
 clear(@slow_queries); clear(@query_count); clear(@total_time);
}

Production Deployment

Performance Impact

Recommended Monitoring Strategy

Production Environment

# Light monitoring - query throughput and basic latency
bpftrace query_latency.bt

# Periodic deep-dive - run for 5 minutes every hour
*/60 * * * * timeout 300 bpftrace phase_timing.bt > /var/log/angarabase-phase-timing.log

Troubleshooting Environment

# Full monitoring during incident investigation
bpftrace lock_contention.bt &
bpftrace io_latency.bt &
bpftrace engine_kernel_correlation.bt &

# Stop all monitoring
pkill bpftrace

Security Considerations

Required Capabilities

# BPF capability required for probe attachment
sudo setcap cap_bpf+ep /usr/bin/bpftrace

# Alternative: run as root (not recommended)
sudo bpftrace query_latency.bt

Access Control

# Restrict bpftrace access to specific group
groupadd angarabase-monitoring
usermod -a -G angarabase-monitoring monitoring-user

# Set appropriate permissions
chown root:angarabase-monitoring /usr/bin/bpftrace
chmod 750 /usr/bin/bpftrace

Integration with Monitoring Systems

Prometheus Integration

#!/bin/bash
# angarabase_probe_exporter.sh - Export probe metrics to Prometheus

# Run bpftrace and parse output
bpftrace -e '
usdt:./angarabase-server:angarabase:query_end {
 @query_duration_sum += arg2;
 @query_count++;
}

interval:s:15 {
 printf("angarabase_query_duration_sum %d\n", @query_duration_sum);
 printf("angarabase_query_count %d\n", @query_count);
 clear(@query_duration_sum); clear(@query_count);
}
' | while read line; do
 echo "$line" > /var/lib/prometheus/node-exporter/angarabase-probes.prom.$$
 mv /var/lib/prometheus/node-exporter/angarabase-probes.prom.$$ \
 /var/lib/prometheus/node-exporter/angarabase-probes.prom
done

Grafana Dashboard

{
 "dashboard": {
 "title": "AngaraBase USDT Probes",
 "panels": [
 {
 "title": "Query Rate",
 "type": "stat",
 "targets": [
 {
 "expr": "rate(angarabase_query_count[5m])",
 "legendFormat": "Queries/sec"
 }
 ]
 },
 {
 "title": "Average Query Latency",
 "type": "stat", 
 "targets": [
 {
 "expr": "rate(angarabase_query_duration_sum[5m]) / rate(angarabase_query_count[5m])",
 "legendFormat": "Avg latency (us)"
 }
 ]
 }
 ]
 }
}

Troubleshooting

Common Issues

1. No Probes Listed

Symptoms:

$ bpftrace -l 'usdt:./angarabase-server:angarabase:*'
# No output

Causes:

• Binary compiled with --no-default-features
• Wrong binary path
• Probes stripped during build

Solution:

# Verify binary has probes
readelf -n ./angarabase-server | grep stapsdt

# Check build configuration
./angarabase-server --version | grep usdt

# Rebuild with probes
cargo build --release --features usdt

2. Permission Denied

Symptoms:

$ bpftrace query_latency.bt
ERROR: failed to attach probe

Causes:

• Missing BPF capabilities
• SELinux/AppArmor restrictions
• Kernel version incompatibility

Solution:

# Add BPF capability
sudo setcap cap_bpf+ep /usr/bin/bpftrace

# Check kernel version (requires 4.9+)
uname -r

# Temporary workaround
sudo bpftrace query_latency.bt

3. High Overhead

Symptoms:

• CPU usage increase > 5%
• Query latency degradation
• Memory usage growth

Causes:

• Too many active probes
• Heavy aggregation (histograms)
• High-frequency events

Solution:

# Reduce monitoring frequency
interval:s:60 { ... } # Instead of interval:s:5

# Use sampling
usdt:./angarabase-server:angarabase:query_start / pid % 10 == 0 / {
 # Monitor only 10% of queries
}

# Limit histogram buckets
@latency = hist($duration_us, 0, 10000, 100); # 100 buckets max

4. Missing Events

Symptoms:

• Expected probes don’t fire
• Incomplete event sequences
• Zero counts in monitoring

Causes:

• Process not using instrumented code paths
• Probe attachment race condition
• Incorrect probe signatures

Solution:

# Verify process is running instrumented code
ps aux | grep angarabase-server

# Check probe attachment
bpftrace -l 'usdt:*:angarabase:*' | grep $(pgrep angarabase-server)

# Debug probe firing
bpftrace -e 'usdt:./angarabase-server:angarabase:* { printf("probe=%s\n", probe); }'

Best Practices

Probe Development

1. Start Simple: Begin with basic query_start/query_end monitoring
2. Add Gradually: Introduce additional probes based on specific needs
3. Test Impact: Measure overhead before deploying to production
4. Use Sampling: For high-traffic systems, sample events (e.g., 1 in 100)

Script Organization

# Directory structure
/opt/angarabase/monitoring/
├── scripts/
│ ├── query_latency.bt
│ ├── lock_contention.bt
│ └── io_performance.bt
├── dashboards/
│ └── grafana-angarabase-probes.json
└── exporters/
 └── prometheus-exporter.sh

Monitoring Strategy

1. Always-On: Basic query throughput and latency
2. Periodic: Detailed phase timing (every hour for 5 minutes)
3. On-Demand: Lock contention and I/O analysis during incidents
4. Correlation: Engine + kernel events for deep troubleshooting

References

• RFC-2026-369: USDT eBPF Observability Probes v0
• bpftrace documentation: https://github.com/iovisor/bpftrace
• BCC tools: https://github.com/iovisor/bcc
• USDT specification: https://sourceware.org/systemtap/wiki/UserSpaceProbeImplementation
• Linux eBPF: https://ebpf.io/

Contact: Database SRE team Last updated: 2026-03-12 by current implementation phase