Structured Logging

Status: ✅ Active Scope: Production operations, diagnostics, troubleshooting

Overview

AngaraBase uses structured logging for production observability and diagnostics. This replaces ad-hoc println! calls with consistent, level-based log messages using the Rust log crate and env_logger backend.

Key Benefits

Consistent Format: All log messages follow structured key=value format
Level Control: Runtime-configurable verbosity (error/warn/info/debug/trace)
Production Ready: Suitable for log aggregation systems (ELK, Splunk, etc.)
Performance: Low overhead when disabled, structured context when enabled

Log Levels

Level Policy

Level	Usage	Examples
`error`	System failures, data corruption, unrecoverable errors	Database corruption, OOM, panic recovery
`warn`	Recoverable failures, degraded performance, misconfigurations	Failed heartbeat, audit sink errors, fallback modes
`info`	Operational events, lifecycle changes, important state transitions	Instance startup, lease acquisition, stats completion
`debug`	Detailed diagnostics, performance metrics, internal state	Micro-rescan progress, MVCC recovery details, buffer pool stats
`trace`	Very verbose debugging, hot path details	Individual tuple processing, lock acquisition

Production Recommendations

Production: info (default) - captures operational events without performance impact
Troubleshooting: debug - detailed diagnostics for performance analysis
Development: trace - full verbosity for debugging

Configuration

Static Configuration (angarabase.conf)

[logging]
log_level = "info"

Supported values: error, warn, info, debug, trace Default: info Restart required: No (dynamic setting)

Environment Variable Override

export ANGARABASE_LOG_LEVEL=debug
./angarabase-server --config /etc/angarabase/angarabase.conf

Environment variables take precedence over config file settings during bootstrap.

Runtime Configuration

Change log level without restart using SQL:

-- Check current setting
SELECT * FROM sys.settings WHERE name = 'logging.log_level';

-- Change to debug level
SELECT sys.set_setting('logging.log_level', 'debug');

-- Verify change
SELECT * FROM sys.settings WHERE name = 'logging.log_level';

Note: Runtime changes are volatile and reset on restart. Update angarabase.conf for persistent changes.

Log Format

Structure

All log messages follow this format:

2026-03-09T10:30:45.123Z INFO [angarabase_server::stats] stats_scheduler: auto_analyze_triggered table=users staleness_score=2.45

Components:

Timestamp: ISO 8601 with millisecond precision
Level: ERROR/WARN/INFO/DEBUG/TRACE
Module: Rust module path (e.g., angarabase_server::stats)
Context: Structured key=value pairs with operation context

Context Format

Log messages use consistent key=value structured context:

#![allow(unused)]
fn main() {
// Good: structured context
log::info!("instance_lease: acquired holder={} expires_at={}", holder_id, expires_at);

// Good: operation context
log::debug!("micro_rescan: completed table={} scanned_rows={} duration={:?}", table, rows, duration);

// Good: error context 
log::warn!("audit_sink: write_failed path={} err={}", path, error);
}

Context Guidelines:

Use snake_case for keys
Include operation prefix (e.g., micro_rescan:, instance_lease:)
Provide enough context for troubleshooting
Avoid sensitive data in logs (use IDs, not content)

Production Deployment

Log Aggregation

AngaraBase structured logs integrate well with log aggregation systems:

ELK Stack:

# logstash.conf
filter {
 if [program] == "angarabase-server" {
 grok {
 match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:module}\] %{GREEDYDATA:context}" }
 }
 kv {
 source => "context"
 field_split => " "
 value_split => "="
 }
 }
}

Splunk:

[angarabase]
EXTRACT-level = (?<level>ERROR|WARN|INFO|DEBUG|TRACE)
EXTRACT-module = \[(?<module>[^\]]+)\]
EXTRACT-operation = (?<operation>\w+):

Log Rotation

Configure log rotation for production deployments:

# /etc/logrotate.d/angarabase
/var/log/angarabase/*.log {
 daily
 rotate 30
 compress
 delaycompress
 missingok
 notifempty
 create 0644 angarabase angarabase
 postrotate
 systemctl reload angarabase-server
 endscript
}

Monitoring

Key log patterns to monitor:

# Error rate monitoring
grep "ERROR" /var/log/angarabase/server.log | wc -l

# Instance lease issues
grep "instance_lease.*failed" /var/log/angarabase/server.log

# Performance degradation
grep "stats_scheduler.*io_backpressure" /var/log/angarabase/server.log

# MVCC recovery events
grep "mvcc_recovery:" /var/log/angarabase/server.log

Troubleshooting

Common Issues

High log volume at debug level:

-- Reduce to info level
SELECT sys.set_setting('logging.log_level', 'info');

Missing log output:

# Check RUST_LOG environment variable
echo $RUST_LOG

# Override with explicit filter
export RUST_LOG=angarabase_server=debug,rustls=warn

Log format parsing issues:

Ensure timestamp parsing handles millisecond precision
Context parsing should handle key=value pairs with spaces
Module names contain :: separators

Performance Impact

Log level performance characteristics:

Level	Overhead	Use Case
`error`/`warn`	Minimal	Always safe in production
`info`	Low	Default production level
`debug`	Moderate	Short-term troubleshooting
`trace`	High	Development/debugging only

Recommendation: Use info for production, temporarily increase to debug for troubleshooting specific issues.

Migration Notes

From println! to log::*!

Current implementation migrated all production println! calls to structured logging:

60+ calls across 11 files replaced
Backward compatibility: No breaking changes to existing functionality
Enhanced context: More structured information for operations

Future Enhancements (Phase 2)

Planned for next releases:

Tracing integration: Distributed tracing with spans
OpenTelemetry: OTLP export for APM systems
Metrics correlation: Link logs with performance metrics
Sampling: Adaptive sampling for high-volume environments

AngaraBook