AngaraBase

Getting Started

Choose your entry point based on your role:

Key Features

PostgreSQL Compatibility Without Surprises

• Standard pgwire protocol: psql, JDBC, psycopg, node-postgres, pgx, Npgsql work just like with standard PostgreSQL.
• Contractual SQL subset: what is supported is supported fully; what is not supported returns an explicit SQLSTATE (0A000 etc.) instead of a silent bypass.
• Pinned compatibility tests (compat suite) as proof, rather than a “compatibility percentage”.

Read more: SQL compatibility overview.

Modern Engine: MVCC via UNDO-log, Not via Bloat

Unlike PostgreSQL, AngaraBase stores historical row versions in a separate UNDO-log (like Oracle/InnoDB), rather than in the table itself. As a result:

• heap pages contain only current row versions → less bloat;
• VACUUM in the traditional sense is not needed — old versions are cleaned up by the background AngaraGC according to a strict contract;
• visibility for transactions is determined deterministically by snapshot.

Read more: Transactions and MVCC.

Pluggable Storage From Day One

• Row-store (baseline) — for OLTP.
• AngaraMemory — in-memory tables with three durability tiers (none / logged / snapshotted) and hard row-cap.
• AngaraColumn — columnar storage for analytics (in roadmap; HTAP direction).

Storage engine is selected during CREATE TABLE ... WITH (storage='memory'|'row'|...).

Read more: Storage engine.

Cost-based Optimizer and Vectorized Execution

• AngaraPlan — CBO with robust planning, resilient to estimation errors.
• AngaraStat — statistics: HLL for NDV, equi-height histograms, MCV (reservoir sampling).
• AngaraFlow — streaming execution (Volcano).
• AngaraVector — vectorized path for scan/filter/project/join/aggregate; modes auto / force_vector / force_row. EXPLAIN explicitly shows VectorHashJoin, VectorAgg, etc.
• AngaraParallel — partitioned parallel join, DOP-caps (ANGARABASE_PARALLEL_DOP_CAP_*).

Read more: Query processing.

Multi-layered Security With Explicit Contracts

Security is built into the core, not “bolted on” as an afterthought. Six layers of protection, each with its own contract:

1. Transport and identity — TLS, SCRAM, cert authentication.
2. RBAC — who is allowed at all.
3. RLS v1 — which rows are visible and mutable.
4. Break-glass — the only way to bypass RLS (even SUPERUSER doesn’t have it), always with a REASON, TTL, and audit.
5. Audit chain — append-only, tamper-evident (SHA-256 chain).
6. TDE — encryption of pages, WAL, and audit-sink; fail-closed without a key.

Read more: Security model.

Operations Built for Predictability

• Per-database backup and restore (cold + online/PITR baseline).
• Clear diagnostics: EXPLAIN/EXPLAIN ANALYZE, sys.* views (sys.identity, sys.health, sys.settings, sys.tables, sys.column_stats, angara_stat_activity, angara_stat_statements).
• Structured logging, OpenTelemetry-style spans, USDT/eBPF probes.
• Prometheus metrics: angara_* named subsystems are visible in logs, metrics, and EXPLAIN.
• Native RPM/DEB packages, init-first service start fence, systemd units.

Read more: Operations runbooks.

Project Principles (Explicit Stance)

Full declaration: Project principles.

Documentation Sections

Tutorials

• What is AngaraBase
• AngaraBase Architecture
• Quickstart

Concepts (Explanation)

• Storage Engine — row-store, pages, slotted pages, pluggable storage
• Transactions and MVCC — UNDO-log MVCC, isolation, GC
• Indexes — AngaraTree (B+tree, BRIN), IndexStore
• Query Processing — AngaraPlan, AngaraStat, AngaraFlow, AngaraVector
• Catalog and Metadata — SysCatalog and sys.* views
• Instance Lifecycle — init, startup, recovery, shutdown

SQL Reference

• Compatibility Overview — policy, SQLSTATE codes, vector execution
• Data Types — supported types, casting, NULL
• DDL — CREATE/ALTER/DROP, indexes, constraints
• DML — INSERT, UPDATE, DELETE, mutation policies
• Queries — CTE, JOIN, aggregates, ORDER BY
• Partitioning — RANGE/LIST partitioning, routing, pruning

Security (How-to)

• Security Model
• Authentication
• Authorization (RBAC + RLS)
• Audit
• Encryption (TDE + client-side)
• Break-glass
• Hardening Runbook
• GOST Compliance

Operations (How-to)

• Installation — portable archive, RPM/DEB, source build
• Configuration — TOML, env, sys.settings
• Backup and Restore — cold + online/PITR
• Crash Recovery — host migration, WAL replay
• Version Upgrade
• Monitoring — Prometheus, Grafana, health probes
• Diagnostics — EXPLAIN, slow log, sys.*
• Logging, Tracing, USDT/eBPF probes
• GC auto-tuning
• Error debug runbook (10 minutes)
• GOST crypto setup
• Verify release artifacts

Operator deep-dives — runbooks (Reference)

• Operations overview
• Runbooks index
• Troubleshooting guide
• Disaster recovery playbook
• Performance tuning guide
• MVCC and GC operator minimum
• Diagnostics bundle runbook
• Security operations baseline
• Upgrade and migration
• Backup and restore (operator-level)
• Configuration schema reference
• Observability metrics checklist
• jemalloc heap profiling runbook
• Parallel runtime observability runbook
• Replication v2 operations guide
• Operational policies baseline
• Client compatibility baseline (operator)
• Testing and validation baseline
• Golden dataset management
• CI reproducibility contract
• Known issues baseline (operator)

Architecture (Reference)

• Architecture overview
• Layering and Boundaries

Reference

• Known Issues and SQLSTATE
• Glossary and Named Subsystems
• System Views sys.*
• Client Compatibility
• Support and Bug Report Artifact Collection
• Generated reference (auto-generated registries)

Changelog

• AngaraBook changelog (highlights) — concise change feed from a user perspective.

Community and Contribution

We welcome community contributors:

• Found a bug or regression? Gather artifacts according to the Support flow and open an issue.
• Want to change behavior? Propose an RFC via the project’s development process (internal loop).
• Want to help with documentation? AngaraBook is documentation-as-code in the same repository. Formatting rules: see the internal WRITING_RULES.md next to the book.
• Want to track development? Follow the changelog and release notes.

About the Documentation

AngaraBook is built on the principles of Diátaxis:

• Tutorials — learning through action, for new users.
• How-to guides (Security / Operations) — recipes for specific tasks.
• Reference (SQL / Architecture) — exact description of behavior.
• Explanation (Concepts) — why things are designed the way they are.

Quality Guarantees:

• Documentation is code: edited in the same repository, passes the same CI as the engine.
• Any claimed SQL/ops behavior is verified by pinned tests or explicitly marked as a roadmap feature.
• Anti-drift: command versions, configuration keys, and SQLSTATE codes are checked automatically.
• Public build passes a security gate: internal processes and confidential links do not make it into the public portal.

If you notice a discrepancy between the documentation and actual behavior — this is a documentation bug; please report it.

AngaraBase v0.6.x · Linux x86_64/aarch64 · glibc >= 2.28

What is AngaraBase

AngaraBase is a relational DBMS written in Rust, compatible with the PostgreSQL protocol and a subset of its SQL. It is designed for ERP/SaaS workloads, where predictable behavior, explicit boundaries, and an absence of “magic” are critical.

• Server platform: Linux x86_64 / aarch64 (glibc >= 2.28)
• Clients: any platform via standard PostgreSQL drivers
• Current branch: 0.6.x

What you get right now

The exact boundaries of SQL support and current limitations are documented in the SQL compatibility overview and Known issues. AngaraBase does not publish a “compatibility percentage” — instead, it provides an exact contract.

Principles

PostgreSQL Compatibility

AngaraBase implements the pgwire protocol as its primary API. From the client’s perspective, it is a standard PostgreSQL endpoint:

psql "host=127.0.0.1 port=5432 user=angara_root dbname=base sslmode=verify-full"

Full compat contract and smoke scenarios: SQL compatibility overview.

How AngaraBase differs from PostgreSQL

Who it is for

• ERP/SaaS teams (e.g. based on Odoo) who need a predictable PostgreSQL-compatible database with an explicit compatibility contract.
• DBAs who value explicit behavioral boundaries over “best-effort” compatibility.
• Engineers who care about knowing exactly what is supported and having reproducible tests as proof.
• The Community willing to participate in shaping a young DBMS.

What AngaraBase does not do (on the current branch)

• It does not provide distributed SQL and multi-master HA — these are for future major branches.
• It does not implement full PostgreSQL SQL — only a contractual subset with clear boundaries.
• It does not mask unsupported features — you get an explicit error with a SQLSTATE.
• It does not run on non-Linux servers. Clients are cross-platform.

Getting Started

Links

• Quickstart — build, run, and execute your first SQL in a few minutes.
• Architecture — how the DB is structured internally.
• SQL reference — what SQL is supported.
• Security model — security model.
• Operations — configuration and operations.
• Glossary — terms and named subsystems.

AngaraBase Architecture

This document provides an understanding of AngaraBase’s internal design at a level sufficient for making decisions: configuration choices, issue diagnostics, and assessing applicability.

Detailed technical specification: docs/01_ARCHITECTURE.md.

Multi-layered Architecture

AngaraBase consists of six layers. Each layer has its own API and depends only on the layers below it. This allows implementations (e.g., storage engine) to be swapped out without changing the other layers.

┌──────────────────────────────────────────────────────────────┐
│ TIER 1: CLIENT LAYER (Wire Protocol) │
│ │
│ • pgwire protocol (compatibility with psql, JDBC, etc.) │
│ • connection pooling │
│ • async event loop │
└─────────────────────────┬────────────────────────────────────┘
 │
┌─────────────────────────┴────────────────────────────────────┐
│ TIER 2: SESSION / TRANSACTION LAYER │
│ │
│ • sessions and session variables │
│ • transaction management (BEGIN/COMMIT/ROLLBACK/SAVEPOINT) │
│ • isolation levels (READ COMMITTED, REPEATABLE READ) │
│ • locks and deadlock detection │
└─────────────────────────┬────────────────────────────────────┘
 │
┌─────────────────────────┴────────────────────────────────────┐
│ TIER 3: QUERY EXECUTION LAYER │
│ │
│ • SQL query parsing │
│ • semantic validation and type checking │
│ • planning and optimization (AngaraPlan) │
│ • physical plan execution (AngaraFlow) │
└─────────────────────────┬────────────────────────────────────┘
 │
┌─────────────────────────┴────────────────────────────────────┐
│ TIER 4: CATALOG & TYPE SYSTEM │
│ │
│ • registry of tables, schemas, databases │
│ • registry of types, functions, operators │
│ • index registry (access methods) │
│ • system views sys.* │
└─────────────────────────┬────────────────────────────────────┘
 │
┌─────────────────────────┴────────────────────────────────────┐
│ TIER 5: STORAGE LAYER (Pluggable Storage) │
│ │
│ • row-store engine (OLTP baseline) │
│ • pluggable: in-memory and column-store (planned) │
│ • indexes (AngaraTree: B+tree, BRIN) │
│ • Transaction Log (WAL) — transaction journal │
└─────────────────────────┬────────────────────────────────────┘
 │
┌─────────────────────────┴────────────────────────────────────┐
│ TIER 6: SYSTEM LAYER │
│ │
│ • buffer manager and page cache │
│ • metrics and telemetry │
│ • crash recovery │
│ • resource scheduler (CPU, memory, I/O) │
└──────────────────────────────────────────────────────────────┘

What this means for you

• TIER 1: you connect using a standard PostgreSQL client — no special tools needed.
• TIER 2: transactions work as usual — BEGIN, COMMIT, ROLLBACK, SAVEPOINT.
• TIER 3: SQL queries go through a parser, optimizer, and executor. EXPLAIN shows the execution plan.
• TIER 4: metadata about tables, types, and indexes is accessible via sys.* system views (e.g. SELECT * FROM sys.tables).
• TIER 5: data is stored in a pluggable storage engine. Currently, it’s a row-store; in future versions, you’ll be able to choose the engine when creating a table.
• TIER 6: buffer, metrics, and recovery are part of the infrastructure layer working transparently. You interact with it via configuration and monitoring.

Named Components

Key AngaraBase subsystems have their own names. This simplifies diagnostics, documentation, and configuration — when you see a name in the logs or metrics, you know which part of the system it refers to.

Example: if EXPLAIN shows AngaraTree: Index Scan, it means the query is using a B+tree index. If AngaraGC appears in the logs, it’s the garbage collector for obsolete row versions.

Data Model

AngaraBase uses a four-level hierarchy (similar to MS SQL Server):

Instance (angarabased process)
 └─ Database
 └─ Schema
 └─ Table

• Instance — a single running angarabased process. It can contain multiple databases.
• Database — an isolated database. Each DB has its own data files, transaction log, and settings. Backup and restore operate on the individual database level.
• Schema — logical grouping of tables within a database (default is public).
• Table — a table containing data.

Example:

angarabased (instance)
 ├─ Database "odoo_prod"
 │ ├─ Schema "public"
 │ │ ├─ Table "res_partner"
 │ │ ├─ Table "sale_order"
 │ │ └─ ...
 │ └─ Schema "staging"
 │ └─ ...
 ├─ Database "analytics"
 │ └─ Schema "public"
 │ └─ ...
 └─ System catalog (sys.*)

Each database is independent: a backup of odoo_prod does not affect analytics, and vice versa.

Configuration Hierarchy

Settings in AngaraBase are applied at three levels, from broadest to narrowest:

Instance (angarabase.conf)
 └─ Database (ALTER DATABASE ... SET ...)
 └─ Session (SET ...)

A narrower level overrides a broader one:

• Instance — server settings (port, memory limits, file paths). Some of these require a restart.
• Database — database-specific settings (limits, storage parameters). Applied without a restart.
• Session — settings for the current connection (SET timezone = 'Europe/Moscow'). Active until the session ends.

What this means in practice

AngaraBase architecture is designed with several principles that affect daily operations:

• Connect with standard tools. pgwire compatibility means you don’t need custom drivers or libraries. psql, DBeaver, your Python or Java application — everything connects just like standard PostgreSQL.

• Per-database isolation. Each database is an independent unit for backup, restore, and configuration. This is handy for multi-tenant scenarios: each client can have their own DB with individual settings and a separate backup schedule.

• Clear diagnostics. System views sys.* provide access to metadata and system state. Named components (AngaraTree, AngaraPlan, etc.) are reflected in EXPLAIN, logs, and metrics — you always know what part of the system is involved.

• Pluggable storage. A row-store (optimized for OLTP) is available now. In future versions, you’ll be able to choose the storage engine when creating a table — in-memory for hot data, column-store for analytics.

• Fail-closed behavior. If an SQL construct isn’t supported — you’ll get an explicit error with an SQLSTATE code, not an unexpected result. This is predictable and safe for production.

Additional Resources

• Canonical architecture doc: docs/01_ARCHITECTURE.md — complete technical specification.
• Storage Engine: concepts/storage-engine.md — row-store, pages, pluggable storage.
• Query Processing: concepts/query-processing.md — parser, planner, optimizer, executor.
• Catalog and Metadata: concepts/catalog-and-metadata.md — SysCatalog and system views.

Quickstart (testing)

Goal

Start angarabased, connect via psql, execute basic DDL/DML, and verify that pgwire works.

Prerequisites

• Linux x86_64
• One of the installation options:
• Rust toolchain (see rust-toolchain.toml) for a source build,
• or the portable archive x86_64-unknown-linux-gnu (glibc >= 2.28).

Install from portable archive

mkdir -p /opt/angarabase
tar -xzf angarabase-0.6.3-x86_64-unknown-linux-gnu.tar.gz -C /opt/angarabase
/opt/angarabase/angarabase-0.6.3/bin/angarabase-server --version

If runtime glibc is below baseline (2.28), angarabase-server exits fail-closed with an explicit compatibility message.

Native package flow

For RPM/DEB deployments, service start is intentionally blocked before secure init:

angarabase-server --init /var/lib/angarabase --superuser admin --auth-mode scram --superuser-password-file /secure/path/pass.txt --require-auth
systemctl start angarabase

If you intentionally need trust bootstrap for isolated labs, it must be explicit:

angarabase-server --init /var/lib/angarabase --auth-mode trust --insecure-trust

Build

cargo build -p angarabase-server
cargo build -p angara-cli

Run server (local)

AngaraBase uses explicit instance initialization (--init) before regular startup.

Minimal path for testing (without manual config creation):

1. Run the one-time initialization in the instance directory.

target/debug/angarabase-server --init /tmp/angarabase-instance --superuser angara_root --superuser-password 'change-me' --auth-mode scram

By default, this will create:

• data/ at /tmp/angarabase-instance/data
• txlog/ at /tmp/angarabase-instance/txlog
• config angarabase.conf at /tmp/angarabase-instance/angarabase.conf

2. Start the server:

target/debug/angarabase-server --config /tmp/angarabase-instance/angarabase.conf

In this scenario, the SCRAM bootstrap user angara_root is used. For local trust/no-auth mode, you can explicitly run with --allow-insecure-no-auth.

SecurityContext note:

• In scram/cert modes, protected SQL execution requires session context.
• Minimal setup for tenant-scoped workloads:

SET SESSION CONTEXT 'app.tenant_id' = 'public';

Alternative path (if you want to use an existing config):

• --config <path> with --init is read as the input config if the file exists,
• and written as the output config if the file does not exist.

Examples:

# init using an existing config (input)
target/debug/angarabase-server --config ./angarabase.conf --init

# init and write a new config (output; file must not exist)
target/debug/angarabase-server --config /tmp/angarabase.conf --init /tmp/angarabase-instance

A shortcut is allowed for local development:

target/debug/angarabase-server --config angarabase.conf --dev

--dev preserves the auto-init behavior only for dev/test scenarios.

Connect with psql

psql "host=127.0.0.1 port=5432 user=angara_root dbname=base password=change-me sslmode=disable"

Smoke SQL

CREATE TABLE t (id INT PRIMARY KEY, v INT);
INSERT INTO t (id, v) VALUES (1, 10);
INSERT INTO t (id, v) VALUES (2, 20);
SELECT * FROM t ORDER BY id;

Restart check (DDL survives restart)

CREATE TABLE metadata (catalog) should survive restart.

1. Stop the server (Ctrl+C).
2. Start it again.
3. Verify that the table is visible:

SELECT table_name FROM sys.tables WHERE table_name = 't';

Sys introspection (sys.*)

Examples of useful queries:

SELECT * FROM sys.identity;
SELECT * FROM sys.health;
SELECT * FROM sys.settings WHERE name IN ('server.addr','storage.data_directory');
SELECT * FROM sys.tables;
SELECT * FROM sys.columns WHERE table_name = 't';

Optional: SQL shutdown (fail-closed)

By default, shutdown via SQL is disabled. To enable it (locally/for tests):

export ANGARABASE_ALLOW_SQL_SHUTDOWN=1

You can then request a shutdown from psql:

SELECT sys.request_shutdown();

If something fails

• Check “Known issues”: ../reference/known-issues.md
• For connecting clients (DBeaver, etc.): ../reference/client-compatibility.md
• To report bugs, gather artifacts according to ../reference/support.md.

What’s Next

Once the server has answered psql -h 127.0.0.1 and the basic SELECT has succeeded, logical next steps:

• What is AngaraBase — a product overview: what the project is for, how it differs from vanilla PostgreSQL.
• SQL Compatibility Overview — what parts of the standard you can use right now.
• Configuration — how to run the server beyond defaults, tailored to your scenario.
• Security Model — before letting anyone else in but yourself.

Connecting clients: psql, Python, JDBC

What you’ll get in 15 minutes

After this tutorial, you will have three working methods to connect to a locally running AngaraBase instance:

1. psql — interactive PostgreSQL console.
2. Python via psycopg[binary] — a typical application script.
3. JDBC via standard org.postgresql:postgresql — a typical Java/Kotlin/Scala stack.

All three methods work via the standard pgwire protocol: AngaraBase presents itself to clients as PostgreSQL, so no special drivers are needed.

Prerequisites

• AngaraBase running locally according to Quickstart. We assume the server is listening on 127.0.0.1:5432, and there is a user angara and a database angara_demo.
• Installed psql (any PostgreSQL version ≥ 13).
• Python 3.10+ (for Step 2).
• JDK 17+ and Maven/Gradle (for Step 3).

Verify that the server is responding:

psql --version
# psql (PostgreSQL) 16.4

ss -ltnp 'sport = 5432'
# LISTEN ... 127.0.0.1:5432 ...

If the port is not listening — go back to Quickstart and make sure angarabased started without errors.

Step 1. psql — interactive console

1.1. Connecting

psql 'postgresql://angara@127.0.0.1:5432/angara_demo'

Password (if set) — at the prompt. Success sign: the angara_demo=> prompt.

1.2. Minimal scenario: create a table, insert, select

-- Inside psql:
CREATE TABLE products (
    id    BIGINT PRIMARY KEY,
    name  TEXT NOT NULL,
    price NUMERIC(10, 2) NOT NULL
);

INSERT INTO products (id, name, price) VALUES
    (1, 'Coffee', 4.50),
    (2, 'Tea',    3.00);

SELECT id, name, price FROM products ORDER BY id;

Expected output:

 id | name   | price
----+--------+-------
  1 | Coffee |  4.50
  2 | Tea    |  3.00
(2 rows)

1.3. Useful \ commands

1.4. If something goes wrong

• could not connect to server: Connection refused — the server isn’t running or isn’t listening on 127.0.0.1. Check ss -ltnp 'sport = 5432' and the angarabased logs.
• authentication failed for user "angara" — password isn’t set or doesn’t match. See Authentication.
• feature_not_supported (0A000) — you hit an SQL construct that AngaraBase does not support. This is an explicit fail-closed contract; see Known Issues and SQLSTATE.

Step 2. Python via psycopg

2.1. Installing the driver

We use psycopg version 3 (binary wheel — without local compilation):

python3 -m venv .venv
source .venv/bin/activate
pip install 'psycopg[binary]>=3.1,<4'

2.2. Minimal script

Create a file connect_demo.py:

import psycopg

DSN = "postgresql://angara@127.0.0.1:5432/angara_demo"

with psycopg.connect(DSN) as conn:
    with conn.cursor() as cur:
        cur.execute(
            "INSERT INTO products (id, name, price) VALUES (%s, %s, %s)",
            (3, "Espresso", 4.25),
        )
        cur.execute("SELECT id, name, price FROM products ORDER BY id")
        for row in cur.fetchall():
            print(row)
    conn.commit()

Run:

python3 connect_demo.py

Expected output:

(1, 'Coffee', Decimal('4.50'))
(2, 'Tea', Decimal('3.00'))
(3, 'Espresso', Decimal('4.25'))

2.3. Important things to know about the Python client

• Parameterized queries are mandatory. Don’t substitute values via f"...{value}..." — this is a path to SQL injections. psycopg substitutes parameters on the driver side via server-side prepared statements.
• with conn: and conn.commit() are different things. The with conn: context manager guarantees connection closure, but does not auto-commit. The transaction is committed only by an explicit conn.commit().
• AngaraBase predictably returns SQLSTATE. Catch psycopg.errors.FeatureNotSupported and check e.diag.sqlstate == "0A000" to gracefully handle unsupported constructs (fail-closed contract).

Step 3. JDBC via org.postgresql:postgresql

3.1. Dependency

Maven (pom.xml):

<dependency>
  <groupId>org.postgresql</groupId>
  <artifactId>postgresql</artifactId>
  <version>42.7.4</version>
</dependency>

Gradle (build.gradle.kts):

dependencies {
    implementation("org.postgresql:postgresql:42.7.4")
}

3.2. Minimal class

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;

public class ConnectDemo {
    public static void main(String[] args) throws Exception {
        String url = "jdbc:postgresql://127.0.0.1:5432/angara_demo";
        java.util.Properties props = new java.util.Properties();
        props.setProperty("user", "angara");
        props.setProperty("preferQueryMode", "simple");

        try (Connection conn = DriverManager.getConnection(url, props)) {
            conn.setAutoCommit(false);

            try (PreparedStatement ps = conn.prepareStatement(
                    "INSERT INTO products (id, name, price) VALUES (?, ?, ?)")) {
                ps.setLong(1, 4L);
                ps.setString(2, "Cappuccino");
                ps.setBigDecimal(3, new java.math.BigDecimal("4.75"));
                ps.executeUpdate();
            }

            try (PreparedStatement ps = conn.prepareStatement(
                    "SELECT id, name, price FROM products ORDER BY id");
                 ResultSet rs = ps.executeQuery()) {
                while (rs.next()) {
                    System.out.printf(
                            "%d %s %s%n",
                            rs.getLong("id"),
                            rs.getString("name"),
                            rs.getBigDecimal("price"));
                }
            }

            conn.commit();
        }
    }
}

3.3. Important things to know about the JDBC client

• preferQueryMode=simple is the recommended default for AngaraBase. This disables the aggressive probe mode of the extended protocol that the driver uses for compatibility with PostgreSQL extensions. AngaraBase implements pgwire by contract, and some extended protocol checks are not needed.
• assumeMinServerVersion=9.0 — add to props if you plan to work via DBeaver/DataGrip; see Client compatibility → DBeaver.
• Transactions and setAutoCommit(false). AngaraBase implements MVCC via UNDO-log; explicit transactions provide a predictable snapshot. Do not leave long transactions open — this slows down AngaraGC.

What’s Next

• SQL Compatibility Overview — what exactly from the SQL standard is available via all three clients.
• Client Compatibility — reference guide for specific GUI tools and drivers (DBeaver, IntelliJ DataGrip, etc.).
• Security Model — how to set up TLS, SCRAM, and certificate authentication for production connections.
• Known Issues and SQLSTATE — what codes the client should expect instead of “magic” values.

Contracts in AngaraBase

Goal

Explain what the term “contract” means in AngaraBase, the levels of contracts that exist, and the mechanisms used to enforce them. This page is for users, DBAs, and new contributors: to make it clear when reading documentation, code, or error messages what guarantees can be relied upon, and what is explicitly outside the contract.

What We Mean by a Contract

A contract in AngaraBase is an explicit promise about the observable behavior of a component: what it takes as input, what it returns, what guarantees it provides, and how it behaves when boundaries are violated. A contract is not “best intentions” or “how it usually works” — it is a formalized agreement that both sides must adhere to: the implementation and the caller (or client).

A contract has three key properties:

1. Explicitness. The contract is documented in a single canonical source — not in a chat, not in a code comment, not in the “shared understanding of the team.”
2. Verifiability. Compliance with the contract is verified automatically: by tests, types, metrics, or lint checks.
3. Fail-closed. When a contract boundary is violated, the system returns an explicit error (with a known code), rather than “somehow continuing to work.”

If something in the system is not covered by a contract, it is explicitly marked as roadmap, experimental, or known limitation. Behavior outside a contract can change without a deprecation cycle.

Levels of Contracts

AngaraBase has several levels of contracts, each with its own source of truth and method of verification.

1. SQL Contract (External, for the User)

What is supported is supported fully. What is not supported returns an explicit SQLSTATE (0A000 feature_not_supported, etc.) rather than a silent bypass or distorted result.

• Source: SQL Compatibility Overview, Known Issues and SQLSTATE.
• Verification: pinned compat suite, regression tests for every documented SQLSTATE.
• What this means in practice: a client can catch psycopg.errors.FeatureNotSupported and know exactly that they hit a documented limitation, not a bug.

2. Configuration Contract

Every config key has a type, a default value, a range of acceptable values, and defined behavior for absence or invalid values.

• Source: Configuration, Configuration schema reference.
• Verification: the parser rejects unknown/invalid keys on startup (fail-closed), rather than silently ignoring them.
• Changing the semantics of a key goes through a deprecation cycle (see WRITING_RULES.md §9a).

3. Operational Contract

Metrics, USDT probes, names of sys.* views, backup/restore formats, log formats, and runbook output — all of these are public names that monitoring and operator automation rely on.

• Source: System tables, Observability metrics checklist, Backup and restore, USDT/eBPF probes.
• Principle: every resource boundary (buffer pool RAM budget, transaction write-set limit, snapshot age, etc.) must have a Prometheus metric and explicit fail-closed behavior on violation. A boundary without observability is not a contract.

4. Internal API Contracts (For Contributors)

Each core subsystem has a public Rust trait defining its semantics: TableEngine, PageProvider, TransactionLogSink, StorageIo, etc. Without implementing the contract, the code won’t compile — this is an “honest checked promise,” not just a “developer’s word.”

• Source: doc-comments on traits, Architecture overview, API Boundaries.
• Verification: Rust compiler + property-tests for invariants + layering lints (Core doesn’t depend on Adapters/Tooling).

5. Documentation Contract (Anti-drift)

Documentation is part of the code. Any change to a public contract (SQL surface, config keys, metrics, SQLSTATE, subsystem names, protective defaults, init/upgrade sequence) must be accompanied by an update to AngaraBook in the same PR.

• Source: WRITING_RULES.md §8 — Anti-drift contract.
• Verification: pre-commit / CI run tools/docs/lint_angarabook_public.py, tools/docs/check_public_build_security.py and mark drift as blocking.

How We Enforce Contracts

A contract without an enforcement mechanism is just a declaration. AngaraBase uses multiple layers of enforcement working together.

The Type System as the First Line of Defense

• Result<T, Error> instead of panics. unwrap() / expect() are forbidden in production code.
• Bounded generics and trait objects instead of dynamic dispatch where invariants can be encoded at the type level.
• No-panic policy: the server does not crash on user input — it returns an SQLSTATE.

Restrictive by Default + Fail-closed

Every component with a resource boundary must define:

• its boundary (e.g., buffer_pool_size_mb, txn_max_write_set_mb, max_concurrent_queries),
• behavior upon violation (an explicit error with a known SQLSTATE),
• the reaction of the caller code (Reaction Propagation Contract).

No boundary and no fail-closed behavior — no merge.

Pinned Tests and Golden Datasets

The contract for SQL compatibility and client compatibility is verified by pinned tests, not a “compatibility percentage.” If a test is pinned — changing behavior requires either updating the test with a justification, or rolling back the change.

More details: Testing and validation baseline, Golden dataset management, CI reproducibility contract.

Deprecation Cycle

When a contract is phased out, it does not silently disappear. A unified cycle is applied: Active → Deprecated → Removed, with at least one major release between Deprecated and Removed; before v1.0 — at least two minor versions. Every phase change is an atomic PR (code + AngaraBook + Migration steps).

Full procedure: WRITING_RULES.md §9a — Deprecation policy. Public list of all deprecated/removed contracts: Known Issues and SQLSTATE.

Security Gate as Fail-closed for Documentation

Public builds of AngaraBook pass through a security gate: links to internal directories (RFC/development/spec/rules) and paths with /internal/ are forbidden. Internal content is hidden via <!-- internal --> ... <!-- /internal --> blocks; the preprocessor fails closed on nested/unclosed markers.

What This Means for the User

• Predictable Behavior. If behavior is documented, it is stable within a major release. If documented as a limitation with an SQLSTATE, it will return exactly that SQLSTATE, rather than “sometimes working.”
• Safe Client Code. You can catch specific SQLSTATEs and build retry/error handling logic without heuristics or parsing text messages.
• Clear Upgrades. Changing the behavior of a public contract goes through an explicit deprecation cycle with migration steps; you see what will change and when, well in advance.
• Observable Guarantees. Every resource boundary has a metric; you can see utilization and rejects before they become incidents.

What This Means for Developers and Contributors

• Compiler Contract, Not Review Comment. If an invariant can be encoded in a trait or type, it is encoded. Code review catches what the compiler cannot.
• Single Source of Truth per Knowledge Type. No need to “search for the current truth across multiple documents”: there is a single canonical owner for each layer of the contract.
• Predictable Debt Management. A deprecated contract is tracked in reference/known-issues.md, and if not resolved in one PR, in the technical debt registry with a status of scheduled <target-train>.
• Anti-drift in a Single PR. Change the behavior — update AngaraBook right there. No “update docs later.”

What Is Not a Contract

To avoid false expectations, explicitly: the following are not considered contracts:

• internal module names, filenames, and private core functions (can change during refactoring without deprecation);
• behavior of features with status: experimental frontmatter or --experimental-* CLI flags — they were never considered stable;
• error message texts (the contract is the SQLSTATE and its meaning, not the text);
• undocumented side effects noticed empirically (“it works for me if…”);
• benchmarks and numerical latency values — this is observability, not a performance claim (a performance claim requires a pinned benchmark, see tools/perf_pack/).

If you rely on any of these in production, it is technical debt on the client side, and a future AngaraBase update will expose it.

Links

• Project Principles §1 — Restrictive by Default — the foundation of the fail-closed approach.
• SQL Compatibility Overview — the SQL contract.
• Known Issues and SQLSTATE — public list of boundaries and deprecated/removed contracts.
• Configuration schema reference — the configuration contract.
• Architecture overview, API Boundaries — internal API contracts and layering.
• Observability metrics checklist — observability as part of the contract.
• WRITING_RULES.md — the documentation contract (anti-drift, deprecation policy).

Storage Engine

Goal

Explain how AngaraBase stores data on disk, what file formats are used, and what storage engines are available (or planned).

Pluggable Storage Architecture

AngaraBase uses a modular storage architecture: the storage engine is responsible for physical data placement, while upper layers (SQL, transactions, indexes) interact via a unified interface. This allows swapping different engines for different workloads without modifying the SQL layer.

The current engine is the Row-Store. Column-Store and In-Memory Engines are planned for the future.

Row-Store (Current Engine)

Page-Based Heap Storage

Data is stored in fixed-size pages (16 KB). Each table is represented by a set of heap pages, where rows are placed sequentially.

Slotted Pages

Each page is structured as a slotted page:

┌─────────────────────────────────────┐
│ Page Header (LSN, checksum, flags) │
├─────────────────────────────────────┤
│ Slot Array → [offset₁, offset₂…] │
│ (grows downwards ↓) │
│ │
│ free space │
│ │
│ (row data grows upwards ↑) │
│ Row₂ data │ Row₁ data │
└─────────────────────────────────────┘

• Header contains the LSN (log sequence number), checksum, page_type, and flags.
• Slot array — an array of pointers to rows within the page. This allows moving rows within the page without altering external references (TID = page_id + slot_id).
• Row data is written from the end of the page towards the beginning.

Page types (page_type): 0 = data (heap), 1 = index (reserved), 2 = meta (reserved), 3 = overflow (reserved).

Page Checksums

Each page is protected by a checksum (CRC32C). When reading a page from disk, the checksum is verified; if it doesn’t match, the server returns an error rather than serving corrupted data (fail-closed with diagnostics).

File Formats

AngaraBase uses a per-database file model: each database consists of a pair of files.

AngaraTree indexes are stored inside the .adb file — page_type = 1 in the page header is reserved for them. There is no separate file for indexes.

Source of truth: crates/angarabase/src/on_disk.rs, angarabook/src/operations/upgrade-and-migration.md.

Data Directory Layout

The data directory is defined by the storage.data_directory setting. Typical layout:

data_directory/
├── VERSION # initialization marker (AVR1, 256 bytes, CRC32C)
├── base.adb # system database (SysCatalog) — heap pages
├── base.atl # WAL for the system database
├── mydb.adb # user DB — heap pages + index pages
├── mydb.atl # WAL for the user DB
└── …

WAL is not stored in separate segmented files (like wal_000001 in PostgreSQL). In AngaraBase, the WAL is a single .atl file per database, located in the same data_directory.

The storage.transaction_log_directory setting defines an alternative directory for .atl files (useful for placing WAL on a separate disk).

Key Settings

[storage]
data_directory = "/var/lib/angarabase/data"
transaction_log_directory = "/var/lib/angarabase/txlog"

More details on settings — Configuration.

Column-Store (Planned, v6)

A columnar engine based on an Arrow/Parquet-like format, oriented towards analytical queries (OLAP). Data is stored by columns, supporting compression and vectorized scans.

Status: not implemented, planned for the v6 roadmap.

In-Memory Engine (Planned, v5 — AngaraMemory)

An engine for storing data in RAM. Three modes are planned:

Status: in development.

HTAP Direction

AngaraBase’s long-term strategy is HTAP (Hybrid Transactional/Analytical Processing):

• Row-Store serves OLTP (transactional workloads).
• Column-Store serves OLAP (analytics).
• Between them lies asynchronous replication: data from the row-store is converted into columnar format for analytical queries.

This will allow running analytics on fresh data without ETL pipelines and without affecting transactional performance.

Related Sections

Concepts (What to read next)

• Transactions and MVCC — how page versions relate to MVCC and WAL.
• Indexes — how B+tree pages fit into the tablespace.
• Catalog and Metadata — where physical table metadata is visible from SQL.

How-to (What to do)

• Configuration — storage, wal, checkpoint settings.
• Backup and Restore — how to transfer a datadir between instances.
• Crash recovery — storage behavior after an unexpected shutdown.
• Diagnostics — how to view IO/page-cache metrics.

Reference

• System Views sys.* — sys.tablespaces, sys.health for inspecting storage state.
• Known Issues and SQLSTATE — STORAGE_* error section.

Transactions and MVCC

AngaraBase ensures concurrent access to data via MVCC (Multi-Version Concurrency Control). Transactions guarantee atomicity of changes, while MVCC allows readers and writers to operate simultaneously without mutually blocking each other.

Transaction Basics

Transaction Management

BEGIN; -- start an explicit transaction
SAVEPOINT sp1; -- create a savepoint
ROLLBACK TO SAVEPOINT sp1; -- rollback to the savepoint
COMMIT; -- commit the transaction
ROLLBACK; -- rollback the entire transaction

Autocommit

By default, AngaraBase operates in autocommit mode: each individual SQL statement executes as a standalone transaction. If the statement completes successfully, the result is committed automatically; on error, it is rolled back.

For operations affecting multiple rows or tables, use explicit transactions (BEGIN / COMMIT) to group changes into a single atomic unit.

MVCC: Row Versioning

The core idea of MVCC is that readers do not block writers, and writers do not block readers. This is achieved by storing multiple versions of each row.

Version Metadata

Each row version contains two system fields:

Visibility Rule

A row version is visible to a transaction with snapshot S if both conditions are met:

1. created_commit <= S — the version was created before or at the time of the snapshot
2. The version is not deleted, or deleted_commit > S — the deletion occurred after the snapshot

Write Operations

• INSERT — creates a new row version with created_commit = current epoch
• UPDATE — does not modify the row in-place. Instead, it marks the current version as deleted (deleted_commit = current epoch) and creates a new version with the updated data
• DELETE — marks the version as deleted (deleted_commit = current epoch)

Isolation Levels

Each transaction receives a snapshot — a fixed view of the data at a specific point in time.

READ COMMITTED (Default)

The snapshot is updated before each statement. The transaction sees all data committed before the start of the current statement. This is the recommended isolation level for most workloads.

SET TRANSACTION ISOLATION LEVEL READ COMMITTED;

REPEATABLE READ

The snapshot is fixed at the time of BEGIN and remains unchanged until the end of the transaction. All statements within the transaction see the same state of the data.

SET TRANSACTION ISOLATION LEVEL REPEATABLE READ;

SERIALIZABLE

As of version 0.6.4.4, AngaraBase implements full SERIALIZABLE (SSI) isolation level. In SERIALIZABLE mode, write skew and phantoms anomalies are prevented through SIREAD locks and tracking read-write anti-dependencies. Transactions violating serializability are aborted with code 40001.

SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;

Locks

Read Operations

Reading uses the MVCC snapshot and does not require locks. A reader never waits for a writer, and vice versa.

Write Operations (Lock-Free DML)

Writers use atomic operations (Compare-And-Swap) for row-level version installation, ensuring lock-free and conflict-free modifications. Traditional write locks are no longer held.

DDL Operations

Schema change operations (CREATE TABLE, ALTER TABLE, DROP TABLE) acquire table-level locks for the duration of the execution.

Deadlock Detection

AngaraBase detects deadlocks using:

• Timeout — if a transaction waits for a lock longer than a configured threshold, it is aborted
• Victim selection — upon detecting a cycle, the system chooses a victim transaction to rollback
• Deterministic lock ordering — an internal strategy for ordering locks to reduce the likelihood of deadlocks

Garbage Collection (AngaraGC)

Over time, the storage accumulates old row versions that are no longer visible to any active transaction. The AngaraGC subsystem is responsible for cleaning them up.

How It Works

1. GC watermark — calculated as the minimum snapshot among all active transactions: min(active_snapshots)
2. Row versions with deleted_commit < watermark are safe to delete — no active transaction can see them
3. Cleanup is performed by a background process without pausing query execution
4. Bounded slices — GC processes data in fixed-size chunks to avoid latency spikes
5. Epoch Reaper — a background worker (since version 0.6.5.24) that prevents the GC watermark from stalling due to abruptly disconnected or hung sessions.

Difference from PostgreSQL

AngaraBase does not have autovacuum in the traditional sense. It uses a hybrid design with an epoch-based watermark (similar to Oracle/InnoDB), allowing for more precise control over the cleanup timing.

Recommendations

• Use READ COMMITTED (the default level) for most workloads
• Avoid long-running transactions — they hold back the GC watermark and prevent the cleanup of old row versions, which increases disk space consumption
• When using REPEATABLE READ, be aware of potential write skew. If strict serializability is needed, use explicit locks (SELECT ... FOR UPDATE)

MVCC State Upon Crash Recovery

Upon restarting after a crash, AngaraBase restores the MVCC state from the transaction log (WAL).

What Is Restored

1. Committed transactions — transactions that managed to write a COMMIT to the WAL
2. Aborted transactions — incomplete transactions are marked as aborted
3. MVCC visibility — information about which row versions are visible for each commit epoch
4. Transaction counters — current commit epoch and other counters

Recovery Process

1. WAL scan — scanning the transaction log files in chronological order
2. MVCC replay — restoring in-memory MVCC structures from WAL records
3. Cleanup — marking uncompleted transactions as aborted

Limitations

• Backend requirement: MVCC recovery only works with transaction_log.backend = "file_bin"
• Memory rebuild: The MVCC state is rebuilt in memory, which can take time for large WAL volumes
• Read-your-writes: Immediately after restart, uncompleted transactions are invisible (marked as aborted)

Monitoring Recovery

-- Check the recovery mode
SELECT recovery_mode FROM sys.identity;

-- Check system health after recovery
SELECT txn_commit_epoch_current FROM sys.health;

Possible recovery_mode values:

• "normal" — normal start without recovery
• "crash_recovery" — recovery after a crash
• "forced_takeover" — forced instance lease takeover

Related Sections

• Storage Engine — storage architecture and page format
• Instance Lifecycle — instance lifecycle and crash recovery
• Crash Recovery — operational procedures for recovery
• SQL Reference — SQL statement syntax

Indexes

Goal

Explain what types of indexes are available in AngaraBase, when to use them, and how they interact with MVCC.

AngaraTree — index engine

AngaraTree is the index engine for AngaraBase. Indexes are stored in .atl files separate from heap data.

B+tree (default)

The primary index type. Suitable for equality and range queries; keys are stored in deterministic order.

-- Creating a B+tree index (equivalent forms):
CREATE INDEX idx_name ON orders (customer_id);
CREATE INDEX idx_name ON orders USING btree (customer_id);

A B+tree index accelerates:

• Exact matches: WHERE customer_id = 42
• Ranges: WHERE created_at >= '2026-01-01' AND created_at < '2026-02-01'
• Sorting: ORDER BY customer_id

BRIN (Block Range Index)

A compact index for data with a natural order (append-only, time-series). BRIN stores min/max values for ranges of heap pages, allowing entire blocks to be skipped during scans.

CREATE INDEX idx_ts ON events USING brin (created_at);

Supported key types:

How BRIN works: the index acts as an accelerator path — first it prunes blocks that do not contain the required values, then a heap fetch is performed with an MVCC predicate recheck. BRIN does not guarantee exactness — it only narrows the search area.

Efficiency metric: angara_brin_range_efficiency shows the fraction of blocks skipped thanks to BRIN. The closer to 1.0, the more efficient the index (data is well-clustered).

Hash / Bloom

Reserved as optional/future index types. Not currently implemented.

Indexes and MVCC

An index stores TID references (page_id, slot_id) to rows in the heap. The visibility of a row is determined not by the index, but by the MVCC layer when reading the heap page:

1. The query accesses the index → retrieves a set of TIDs.
2. For each TID, the heap page is read.
3. The MVCC layer checks the visibility of the row version for the current transaction.

Consequence: an index may contain references to invisible (obsolete) row versions. This is normal — such entries are filtered during the heap fetch.

IndexStore — persistent secondary indexes

AngaraBase supports persistent secondary indexes for RowStore tables via IndexStore.

How it works

• CREATE INDEX builds the index via a full table scan (build_from_rows) and saves the result.
• DML (INSERT/DELETE) automatically updates all table indexes — fail-closed: if the index update fails, the heap mutation is rolled back.
• The optimizer uses the index for WHERE col = value queries (O(log N) instead of O(N) seq_scan).

Resource constraints

Observability

Current Limitations

Attempting to create an unsupported index returns SQLSTATE 0A000 (feature_not_supported).

When to create indexes

Recommended:

• On columns frequently used in WHERE, JOIN ON, ORDER BY.
• BRIN — on time-series columns of tables with append_only = true, where data is inserted in ascending order of the key.

Not recommended:

• On tables with a small number of rows (a full scan will be faster).
• On columns with very low selectivity (e.g., boolean flags).
• Creating many indexes on a single table slows down INSERT/UPDATE/DELETE.

Use EXPLAIN ANALYZE to check if the optimizer is using an index. For more details — Query processing.

Index Integrity Check

To perform an offline check of B+tree index integrity, the validate() function is available:

SELECT angara_index_validate('idx_name');

Recommended to run after a crash or recovery from backup.

Related Sections

Concepts (What to read next)

• Query Processing — how the optimizer selects and combines indexes.
• Storage Engine — how B+tree pages map onto the tablespace.
• Transactions and MVCC — why updating indexes under load requires MVCC visibility.

How-to (What to do)

• DDL: CREATE/DROP INDEX — syntax for creating and dropping indexes.
• Diagnostics — how to use EXPLAIN ANALYZE and sys.* to see if an index is used.

Reference

• Data Types — which types are supported as index keys.
• System Views sys.* — sys.indexes, sys.column_stats for coverage analysis.

Query Processing

Goal

Explain how AngaraBase processes SQL queries: from text to result. Useful for understanding EXPLAIN plans and performance diagnostics.

Pipeline Overview

Each SQL query goes through four stages:

SQL text ──▸ Parsing ──▸ Planning ──▸ Optimization ──▸ Execution ──▸ Result

1. Parsing: SQL → AST

The parser converts the query text into an Abstract Syntax Tree (AST). AngaraBase uses a PostgreSQL-compatible SQL dialect.

If the syntax is not supported, the server returns SQLSTATE 0A000 (feature_not_supported) with a description of the unsupported construct.

2. Planning: AST → Logical Plan

During the planning stage:

• Name resolution — matching table, column, and function names to catalog objects.
• Type checking — verifying types and applying automatic coercion if needed.

The result is a logical plan describing what needs to be done, but not how.

3. Optimization (AngaraPlan)

The AngaraPlan optimizer converts the logical plan into a physical one, choosing the most efficient execution strategy.

Cost-based optimizer (CBO): decisions are made based on statistics (AngaraStat) — row counts, value distributions, index availability.

Key optimizer decisions:

Robust planning: the optimizer is resilient to estimation errors — with a significant discrepancy between estimated and actual rows, the plan remains viable (avoids worst-case behavior).

LEO (Learning Optimizer): feedback loop — after query execution, actual statistics are used to improve future estimates.

4. Execution (AngaraFlow)

The AngaraFlow executor runs the physical plan in an iterator/streaming model (Volcano): each operator requests the next row from its child operator.

Core Operators:

AngaraStat (Statistics)

The optimizer uses statistics from system tables to estimate plan costs.

sys.table_stats

sys.column_stats

Managing Statistics Level

ALTER TABLE t SET (stats_level_max = 2);

A higher level gives the optimizer more information but increases statistics gathering time.

Query Diagnostics

EXPLAIN

Viewing the execution plan without executing the query:

EXPLAIN SELECT * FROM orders WHERE customer_id = 42;

Variants:

EXPLAIN ANALYZE SELECT ...; -- executes the query, shows actual rows/time
EXPLAIN (BUFFERS) SELECT ...; -- + page read statistics
EXPLAIN (FORMAT JSON) SELECT ...; -- JSON output

System Views

Slow Query Log

Enabled via environment variables:

ANGARABASE_LOG_MIN_DURATION_MS=100 # log queries taking longer than 100 ms
ANGARABASE_LOG_QUERY_TEXT=1 # include the query text in the log

More details — Diagnostics.

Future

Planned query processing improvements:

Related Sections

Concepts (What to read next)

• Indexes — how the optimizer chooses an index for SELECT/UPDATE.
• Catalog and Metadata — where statistics used by the planner are stored.
• Transactions and MVCC — how the isolation level affects the execution plan.

How-to (What to do)

• Diagnostics — EXPLAIN ANALYZE, slow-query log, how to read a plan.
• Tracing — distributed tracing of parse → plan → execute phases.
• Logging — ANGARABASE_LOG_QUERY_TEXT, ANGARABASE_LOG_MIN_DURATION_MS.

Reference

• SQL Overview — what parts of the standard are supported at the planner level.
• System Views sys.* — sys.queries, sys.column_stats, sys.health.

Catalog and Metadata

AngaraBase stores metadata for all database objects in a central registry — the SysCatalog. For users, access to metadata is provided via the sys.* system views and introspection functions.

Object Hierarchy

Database objects are organized in a strict hierarchy:

Instance → Database → Schema → Table → Column

Each level has a unique identifier in the SysCatalog. Tables belong to schemas, schemas to databases, and databases to the instance.

SysCatalog

SysCatalog is the central metadata registry. It stores information about:

• tables, columns, and their data types
• indexes and constraints
• users, roles, and privileges
• functions and aggregates
• security policies (RLS)
• statistics for the query optimizer

The SysCatalog is updated during DDL operations (CREATE, ALTER, DROP) and when statistics are gathered.

System Views (sys.*)

System views are read-only and do not require special privileges (with the exception of security-related views).

Instance Identification and State

Data Structure

Statistics

Security and Access

Introspection Functions

AngaraBase provides a set of built-in functions for programmatic access to metadata.

Roles and Privileges

Security (RLS, audit, break-glass)

Diagnostics and Performance

Practical Examples

Instance Information

SELECT * FROM sys.identity;

Checking Server Health

SELECT * FROM sys.health;

List All Tables

SELECT * FROM sys.tables;

Checking Security Settings

SELECT name, value FROM sys.settings WHERE name LIKE 'security.%';

Viewing Table Column Statistics

SELECT column_name, ndv_approx, null_count
FROM sys.column_stats
WHERE table_name = 'orders';

Checking Current User Privileges

SELECT * FROM sys.my_privileges;
SELECT angara_has_privilege('orders', 'SELECT');

Related Sections

• Quickstart (sys.* examples) — first steps with system views
• Security — security introspection functions
• Diagnostics — monitoring and troubleshooting
• Query Processing — how the optimizer uses catalog metadata

Instance Lifecycle

This document explains the conceptual model of AngaraBase instance identity, lifecycle, and the Instance Lease system that enables safe crash recovery and storage portability.

Instance Identity

Each AngaraBase instance has a unique identity established during initialization:

Core Identity Components

• cluster_id: UUID identifying the logical database cluster
• instance_id: UUID identifying this specific instance
• Data directory: Physical location of database files
• Transaction log directory: Physical location of WAL files

Identity Persistence

Identity is stored in two places:

1. VERSION marker: Binary file with format version and IDs
2. System catalog pages: In base.adb reserved pages with full metadata

Instance Lease System

The Instance Lease prevents multiple instances from accessing the same data files simultaneously, which would cause corruption.

Lease Structure

#![allow(unused)]
fn main() {
pub struct InstanceLeaseV0 {
 pub holder_id: String, // UUID of owning instance
 pub acquired_at_unix_s: u64, // When lease was taken
 pub expires_at_unix_s: u64, // When lease expires (TTL)
 pub holder_pid: u32, // Process ID (diagnostic)
 pub holder_hostname: String, // Hostname (diagnostic)
}
}

Lease State Machine

 [None] ──acquire──> [Held] ──heartbeat──> [Held]
 ↑ │ │
 │ │ │
 └──expired/release──┘ │
 │
 [Expired] <──────────timeout─────────────────┘
 │
 └──takeover──> [Held by new instance]

State Transitions

1. None → Held: First instance startup or after graceful shutdown
2. Held → Held: Periodic heartbeat updates (every 10s by default)
3. Held → None: Graceful shutdown releases lease immediately
4. Held → Expired: Heartbeat stops (crash, network partition)
5. Expired → Held: New instance takes over after TTL expiration

Lease Storage

• Location: Stored in SysCatalogMetaV0 within base.adb pages
• Persistence: Atomic updates with full page images
• Reliability: Works on NFS/SAN where flock() is unreliable

Startup Sequence

Phase 1: Pre-flight Checks

1. Verify data directory exists and is initialized
2. Check VERSION marker compatibility
3. Validate page size matches compiled binary

Phase 2: Lease Acquisition

1. Load system catalog from base.adb
2. Check existing lease status:

• No lease: Acquire immediately
• Expired lease: Take over with warning
• Active lease: Fail with informative error
• Force takeover: Override active lease (dangerous)

Phase 3: Recovery

1. WAL Recovery: Replay transaction log (file_bin backend)
2. MVCC Recovery: Restore in-memory transaction state
3. Heartbeat Start: Begin periodic lease renewal

Phase 4: Ready for Connections

1. Start protocol listeners (pgwire, admin)
2. Begin accepting client connections
3. Continue heartbeat until shutdown

Recovery Modes

AngaraBase tracks the recovery mode for operational visibility:

Normal Startup

• Clean start on existing, properly shut down data
• No WAL replay required
• recovery_mode = "normal"

Crash Recovery

• Previous instance terminated unexpectedly
• WAL replay recovers committed transactions
• MVCC state rebuilt from transaction log
• recovery_mode = "crash_recovery"

Forced Takeover

• Operator used ANGARABASE_FORCE_LEASE_TAKEOVER=1
• May indicate emergency recovery scenario
• recovery_mode = "forced_takeover"

Shared Storage Scenarios

The Instance Lease system enables AngaraBase to work correctly on shared storage where multiple hosts can access the same files.

NFS/SAN Deployment

Host A ──┐
 ├── NFS/SAN ──> [data/] [txlog/]
Host B ──┘ [base.adb with lease]

Benefits

• Failover: Host B can take over if Host A crashes
• Maintenance: Move instance between hosts without dump/restore
• Testing: Run against production data copies safely

Limitations

• Single writer: Only one instance can write at a time
• Network partitions: May cause false lease expiration
• Performance: Network storage latency affects throughput

File Copy Scenarios

For non-shared storage, manual file copy enables:

• Backup testing: Verify backup integrity on different host
• Development: Use production data copy for debugging
• Migration: Move to new hardware without downtime

Configuration

Lease Timing

• ANGARABASE_LEASE_TTL_S: How long lease lasts (default: 30s)
• ANGARABASE_LEASE_HEARTBEAT_S: Renewal frequency (default: 10s)

Safety Controls

• ANGARABASE_FORCE_LEASE_TAKEOVER: Emergency override (default: false)

Recommended Settings

# Production: Longer TTL for network stability
export ANGARABASE_LEASE_TTL_S=60
export ANGARABASE_LEASE_HEARTBEAT_S=20

# Development: Shorter TTL for faster iteration 
export ANGARABASE_LEASE_TTL_S=15
export ANGARABASE_LEASE_HEARTBEAT_S=5

Monitoring and Observability

Instance Status

-- Check current lease holder
SELECT lease_holder_id, lease_holder_hostname, 
 lease_expires_at, recovery_mode 
FROM sys.identity;

-- Check system health
SELECT uptime_seconds, txn_commit_epoch_current 
FROM sys.health;

Lease Events

AngaraBase logs lease events to stderr:

Instance lease acquired: holder=abc123...
Instance lease taken over: holder=def456...
Warning: lease heartbeat failed: I/O error
Instance lease released: holder=abc123...

Metrics Integration

Future versions will expose lease metrics via:

• Prometheus metrics endpoint
• sys.metrics virtual table
• Structured logging output

Security Considerations

Access Control

• Lease system does NOT provide authentication
• File system permissions still required
• Network access controls recommended for shared storage

Audit Trail

• Lease changes logged with timestamps
• Instance identity tracked in sys.identity
• Recovery mode visible for forensics

Troubleshooting

Common Issues

“Cannot start: database files are owned by another instance”

• Diagnosis: Active lease prevents startup
• Resolution: Wait for expiration or verify other instance is dead

Frequent lease takeovers

• Diagnosis: Network instability or resource contention
• Resolution: Increase TTL, check network/disk performance

“MVCC recovery failed”

• Diagnosis: Corrupted transaction log
• Resolution: Check filesystem, restore from backup if needed

Debug Information

-- Instance identity and lease
SELECT * FROM sys.identity;

-- Recent recovery statistics 
SELECT * FROM sys.health;

-- Transaction log status
SELECT * FROM sys.settings WHERE name LIKE 'transaction_log.%';

Related Sections

Concepts (What to read next)

• Storage Engine — datadir that protects the instance lease.
• Transactions and MVCC — what happens to active transactions during forced_takeover.

How-to (What to do)

• Crash recovery — operational procedures for recovery after a crash.
• Configuration — variables instance_lease.*, recovery.*.
• Backup and Restore — how to protect datadir and take a snapshot.

Reference

• System views sys.* — sys.identity, sys.health for lease state diagnostics.
• Known issues and SQLSTATE — INSTANCE_* error section.

SQL compatibility overview

Goal

Understand AngaraBase’s compatibility model with PostgreSQL and quickly interpret errors during testing.

Approach

AngaraBase implements a limited subset of PostgreSQL SQL via the pgwire protocol. Unsupported constructs return an explicit SQLSTATE (most often 0A000 feature_not_supported), rather than a silent incorrect result.

Principle: fail-closed — if a feature is not implemented, the client receives a deterministic error.

Source of truth

Practical advice

• ORM/tooling (DBeaver, psql, Hibernate, etc.) often run pg_catalog queries upon connection. Rely on the compat suite results in --dbeaver-smoke / --nightly modes.
• If you observe a hang/stall — this is a P0/P1 bug. Collect artifacts following the instructions in Support.

SQLSTATE quick reference

Full list with context: Known issues.

SQL reference pages

Current status of SQL subsystems on the 0.6.x branch. Badges reflect how well the subsystem is covered by pinned compat-suite tests and whether it can be relied upon in production scenarios.

SQL Functions (RM-0.6.5.5)

AngaraBase implements a subset of PostgreSQL built-in functions.

Vector execution visibility

When vector execution mode allows vector path (ANGARABASE_SQL_EXECUTION_MODE=auto for supported plans, or force_vector), EXPLAIN surfaces vector operator names:

• VectorSeqScan
• VectorFilter
• VectorProject
• VectorHashJoin
• VectorAgg

If the plan shape is not supported by AngaraVector, planner/executor keeps row operators in EXPLAIN.

Execution mode behavior:

• auto (default) — Stable use vector only for fully supported plan shapes, otherwise deterministic row fallback.
• force_row — Stable always use row executor.
• force_vector — Experimental fail-closed with feature_not_supported if vector execution is not possible. Suitable for targeted testing of the vector path; not recommended for production workloads in 0.6.x.

AngaraMemory table options

CREATE TABLE ... WITH (...) now supports bounded memory-table surfaces:

• storage='memory' — Baseline selects AngaraMemory table engine.
• durability='none'|'logged'|'snapshotted' — Baseline volatile/durable behavior.
• max_rows=<n> — Stable hard row-cap; overflow is fail-closed (54023).
• eviction_policy='error'|'fifo' — Experimental default error; fifo is opt-in.
• checkpoint_interval_ms=<n> — Baseline valid only with durability='snapshotted'.

If max_rows is omitted for memory tables, bounded default is applied from instance policy.

Behavior clarification for durability='snapshotted':

• DML hot path does not perform immediate page persistence.
• Checkpoint worker applies per-table scheduling using each table’s checkpoint_interval_ms (with bounded fallback).

Links

• Known issues: Known issues
• Support: Support
• Security model: Security / authorization

Data types

Goal

Reference for supported AngaraBase data types and type casting rules.

Supported types

Text-backed temporal types

TIMESTAMP and DATE are stored in text-backed compatibility mode. This means:

• Comparisons are performed as text (lexicographical); ISO 8601 format guarantees correct ordering.
• Serialization (RM-0.6.5.5): TIMESTAMP is always serialized in UTC without offset (e.g., 2026-05-07 14:30:00.123). Trailing microsecond zeros are truncated.
• Arithmetic (INTERVAL etc.) is not supported — 0A000.
• BRIN indexes on date / timestamp / timestamptz columns are supported (using text min/max).

Planned types

Attempting to use an unsupported type will result in a parser error or 0A000 feature_not_supported.

NULL handling

• All types allow NULL unless the column is declared as NOT NULL.
• In ORDER BY ASC, NULL values are treated as the largest (appear last).
• Explicit NULLS FIRST / NULLS LAST control is not supported — 0A000.

Type casting

AngaraBase supports PostgreSQL-syntax type casting:

SELECT '42'::INTEGER;
SELECT id::TEXT FROM t;

Casting between incompatible types results in a runtime error with the corresponding SQLSTATE.

Expected SQLSTATE

Links

• SQL compatibility overview: overview.md
• DDL (CREATE TABLE with types): ddl.md
• Known issues: Known issues

DDL — Data Definition Language

Goal

Reference for supported DDL operations: creating, altering, and dropping tables, indexes, and constraints.

CREATE TABLE

Basic form

CREATE TABLE t (
 id INTEGER PRIMARY KEY,
 name VARCHAR(100) NOT NULL,
 v BIGINT
);

With table options

CREATE TABLE events (
 id INTEGER PRIMARY KEY,
 ts TIMESTAMP NOT NULL,
 data TEXT
) WITH (append_only = true);

CREATE TABLE ledger (
 id INTEGER PRIMARY KEY,
 amount BIGINT,
 ref_id INTEGER
) WITH (mutation_policy = 'no_delete');

Constraints

• PRIMARY KEY — required for every table (exactly one).
• NOT NULL — column does not accept NULL.
• FOREIGN KEY ... NOT ENFORCED — declarative FK without runtime checking.

CREATE TABLE orders (
 id INTEGER PRIMARY KEY,
 parent_id INTEGER NOT NULL,
 FOREIGN KEY (parent_id) REFERENCES parents (id) NOT ENFORCED
);

Enforced foreign keys are not supported — attempting to create an FK without NOT ENFORCED will return 0A000.

ALTER TABLE

Table-level options

ALTER TABLE t SET (append_only = true);
ALTER TABLE t SET (append_only = false);
ALTER TABLE t SET (mutation_policy = 'no_delete');
ALTER TABLE t SET (mutation_policy = 'unrestricted');

ALTER TABLE t SET (stats_level_max = 2);
ALTER TABLE t SET (stats_reservoir_size = 1000);

Column-level options

ALTER TABLE t ALTER COLUMN v SET (stats_level_max = 1);

Table options reference

append_only = true is equivalent to mutation_policy = 'append_only'.

DROP TABLE

DROP TABLE t;

With DROP TABLE, owned sequences (SERIAL/IDENTITY) are cascadedly dropped, even without CASCADE (PostgreSQL behavior).

CREATE / ALTER / DROP SEQUENCE

AngaraBase supports first-class sequence objects (SEQUENCE) — RM-0.6.3.7, RFC-2026-497. They are persisted in sys_catalog, survive server restart, and back SERIAL/BIGSERIAL and GENERATED [ALWAYS|BY DEFAULT] AS IDENTITY.

CREATE SEQUENCE

CREATE SEQUENCE s1;                                                -- start=1, inc=1, no upper bound
CREATE SEQUENCE s2 START WITH 100 INCREMENT BY 5;
CREATE SEQUENCE s3 MINVALUE 1 MAXVALUE 999 CYCLE;                  -- cyclic counter
CREATE SEQUENCE IF NOT EXISTS s1;                                  -- idempotent

Options (any order): START WITH n, INCREMENT BY n, MINVALUE n / NO MINVALUE, MAXVALUE n / NO MAXVALUE, CYCLE / NO CYCLE.

ALTER SEQUENCE

ALTER SEQUENCE s1 INCREMENT BY 10;
ALTER SEQUENCE s1 RESTART WITH 1;                                  -- reset counter
ALTER SEQUENCE s1 MAXVALUE 1000 NO CYCLE;
ALTER SEQUENCE t_id_seq OWNED BY t.id;                             -- bind to column
ALTER SEQUENCE s1 OWNED BY NONE;                                   -- unbind

DROP SEQUENCE

DROP SEQUENCE s1;
DROP SEQUENCE IF EXISTS s1;
DROP SEQUENCE s1 CASCADE;

Function behavior

See dml.md — “Sequence functions” section (nextval, currval, setval).

SQLSTATE

CREATE INDEX

AngaraBase supports single-column indexes of two types: btree (default) and brin.

-- btree (default)
CREATE INDEX idx_t_v ON t (v);

-- btree (explicit)
CREATE INDEX idx_t_v ON t USING btree (v);

-- brin
CREATE INDEX idx_events_ts ON events USING brin (ts);

BRIN supported key types

BRIN remains an accelerator path with heap fetch + MVCC predicate recheck.

Current bounds

• Single-column indexes only.
• Composite and expression indexes are not supported — 0A000.
• Unsupported index methods (GIN, GiST, etc.) — 0A000.

Expected SQLSTATE

Links

• Data types: data-types.md
• DML (INSERT/UPDATE/DELETE): dml.md
• Partitioning (CREATE TABLE … PARTITION BY): partitioning.md
• Known issues: Known issues

DML — Data Manipulation Language

Goal

Reference for supported DML operations and mutation policy behavior.

INSERT

INSERT INTO t (id, v) VALUES (1, 100);
INSERT INTO t (id, v) VALUES (2, 200), (3, 300);

For partitioned tables, an INSERT into the parent table automatically routes the row to the appropriate partition. If no suitable partition (and no DEFAULT) is found — 23514 check_violation.

INSERT … SELECT

RM-0.6.3.7 (RFC-2026-497, S8): the source of an INSERT can be an arbitrary SELECT query. The result columns are mapped to the target columns positionally (or by explicit column list); SERIAL/IDENTITY/DEFAULT are resolved independently for each row.

INSERT INTO archive (id, v) SELECT id, v FROM t WHERE created_at < '2025-01-01';
INSERT INTO log (note) SELECT 'rebuilt:' || name FROM rebuilt_items;

INSERT … ON CONFLICT (UPSERT)

RM-0.6.3.7 (RFC-2026-497, S9): single-column conflict target (or default PK) is supported. EXCLUDED.<col> refers to the proposed row.

-- DO NOTHING (earlier behavior, now accepts an explicit target):
INSERT INTO t (id, v) VALUES (1, 100) ON CONFLICT (id) DO NOTHING;

-- DO UPDATE SET ... [WHERE ...]
INSERT INTO counters (key, n)
VALUES ('hits', 1)
ON CONFLICT (key) DO UPDATE SET n = counters.n + EXCLUDED.n;

-- WHERE-filter on existing row (PG-semantics: conflict + WHERE-false = silent no-op):
INSERT INTO inventory (sku, qty)
VALUES ('A', 5)
ON CONFLICT (sku) DO UPDATE SET qty = EXCLUDED.qty
WHERE inventory.qty < EXCLUDED.qty;

Not supported:

• multi-column target (ON CONFLICT (a, b)) → 0A000;
• ON CONFLICT ON CONSTRAINT <name> → 0A000.

UPDATE

UPDATE t SET v = 999 WHERE id = 1;

Since RM-0.6.5.5, UPDATE SET supports functional expressions and type casting:

• UPDATE t SET write_date = NOW() WHERE id = 1;
• UPDATE t SET ts = '2026-05-07 14:30:00'::timestamp WHERE id = 1;
• UPDATE t SET day = date_trunc('day', NOW()) WHERE id = 1;

Supported are NOW(), CURRENT_TIMESTAMP, CURRENT_DATE, date_trunc(), and explicit CAST (syntax ::).

DELETE

DELETE FROM t WHERE id = 2;

RETURNING

RM-0.6.3.7 (RFC-2026-497, S10): RETURNING is supported for multi-row INSERT, UPDATE, DELETE. The projection allows *, an explicit column list, and expressions. For INSERT ... ON CONFLICT DO UPDATE, the post-update row is returned (like in PostgreSQL).

INSERT INTO t (v) VALUES (100), (200), (300) RETURNING id, v;
UPDATE t SET v = v + 1 WHERE v > 0 RETURNING id, v AS new_v;
DELETE FROM t WHERE v IS NULL RETURNING id;

Sequence functions

RM-0.6.3.7 (RFC-2026-497): nextval / currval / setval for SEQUENCE objects (see ddl.md — “CREATE / ALTER / DROP SEQUENCE” section). Non-transactional: gap-on-rollback is the correct behavior.

SELECT nextval('s1');                  -- 1, 2, 3, ...; overflow without CYCLE → 2200H
SELECT currval('s1');                  -- last value issued in THIS session
SELECT setval('s1', 100);              -- last_value=100, is_called=true; next nextval = 101
SELECT setval('s1', 50, false);        -- next nextval = 50

currval is session-bound: it returns the value of the last nextval (or setval(_, _, true)) in the same pgwire session. If nextval has not yet been called in the current session — 55000 object_not_in_prerequisite_state, regardless of whether nextval was called in other sessions.

SERIAL / IDENTITY

SERIAL / BIGSERIAL and GENERATED [ALWAYS|BY DEFAULT] AS IDENTITY automatically create an owned sequence <table>_<col>_seq, which is dropped when the table is dropped.

CREATE TABLE users (id SERIAL PRIMARY KEY, name TEXT);
INSERT INTO users (name) VALUES ('alice'), ('bob') RETURNING id;
-- id is populated via nextval('users_id_seq')

CREATE TABLE orders (
  id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  amount NUMERIC
);
INSERT INTO orders (id, amount) VALUES (42, 100);
-- → 428C9 generated_always: id column is GENERATED ALWAYS

CREATE TABLE invoices (
  id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY,
  total NUMERIC
);
INSERT INTO invoices (id, total) VALUES (DEFAULT, 250) RETURNING id;
INSERT INTO invoices (id, total) VALUES (1000, 250);     -- explicit OK

SELECT

SELECT id, v FROM t WHERE v > 50 ORDER BY id;

For details on queries (CTE, JOIN, GROUP BY, etc.) — see queries.md.

TRUNCATE

TRUNCATE TABLE t;

TRUNCATE is a DDL reset of the table. For append-only tables, this is the only way to delete data.

Mutation policy enforcement

Mutation policy controls the allowed DML operations on a table.

append_only

no_delete

unrestricted

All DML operations are allowed (default behavior).

Autocommit vs explicit transactions

By default, every DML query runs in autocommit mode (an implicit transaction, committed immediately).

For explicit transactions:

BEGIN;
INSERT INTO t (id, v) VALUES (10, 1000);
UPDATE t SET v = 2000 WHERE id = 10;
COMMIT;

ROLLBACK aborts all changes of the current transaction.

Expected SQLSTATE

Links

• DDL (CREATE TABLE, mutation policy options): ddl.md
• Queries (SELECT details): queries.md
• Partitioning: partitioning.md
• Known issues: Known issues

Queries

Goal

Reference for supported query forms: CTE, JOIN, aggregation, sorting, and statistical surfaces.

Non-recursive WITH (CTE)

AngaraBase supports limited (non-recursive) CTEs as a derived source in FROM:

WITH recent AS (
 SELECT id, v FROM t WHERE v > 100
)
SELECT r.id, r.v
FROM recent r
ORDER BY r.id;

Supported CTE projection forms

WITH c AS (SELECT id, v FROM t)
SELECT * FROM c WHERE id < 10;

Explicitly unsupported

• WITH RECURSIVE — 0A000 feature_not_supported
• Data-modifying CTEs (WITH ... INSERT/UPDATE/DELETE) — 0A000 feature_not_supported

JOINs

SELECT a.id, b.name
FROM orders a
INNER JOIN customers b ON a.customer_id = b.id;

SELECT a.id, b.name
FROM orders a
LEFT JOIN customers b ON a.customer_id = b.id;

Supported are: INNER JOIN, LEFT JOIN.

ORDER BY

SELECT * FROM t ORDER BY v;
SELECT * FROM t ORDER BY v ASC;
SELECT * FROM t ORDER BY v DESC;

ORDER BY <expr> supports limited scalar expressions already supported by the engine.

Support for aliases

ORDER BY can reference aliases defined in the SELECT list using the AS keyword. This works for both regular columns and aggregation results:

-- Alias for an expression
SELECT x * 2 AS doubled FROM t ORDER BY doubled;

-- Alias for aggregation
SELECT grp, sum(amount) AS total 
FROM t 
GROUP BY grp 
ORDER BY total DESC;

Column ordinals

Sorting by the ordinal number of a column in the SELECT list (1-based) is supported, according to the SQL:2011 standard:

-- Sort by the first column (a)
SELECT a, b FROM t ORDER BY 1;

-- Sort by the second column (b) in descending order
SELECT a, b FROM t ORDER BY 2 DESC;

If the specified ordinal is out of range for the number of columns in the query (or equals 0), a 42P10 (invalid_column_reference) error is returned.

NULL ordering

• In ASC order, NULL is treated as the largest value (appears last).
• In DESC order, NULL appears first.
• Explicit NULLS FIRST / NULLS LAST is not supported — 0A000.

GROUP BY / HAVING

SELECT v, COUNT(*) AS cnt
FROM t
GROUP BY v
HAVING COUNT(*) > 1;

LIMIT / OFFSET

SELECT * FROM t ORDER BY id LIMIT 10;
SELECT * FROM t ORDER BY id LIMIT 10 OFFSET 20;

Expression Errors

When incompatible types are used in arithmetic expressions (for example, attempting to add a string to a number 'string' + 1), AngaraBase returns a 42883 (undefined_function / operator not found) error.

Note: string literals containing numbers (e.g., '123') may be automatically cast to a numeric type depending on the context.

Unsupported query forms

AngaraStat surfaces

AngaraBase provides built-in statistical views.

sys.table_stats

sys.column_stats

SELECT * FROM sys.table_stats WHERE table_name = 'events';
SELECT * FROM sys.column_stats WHERE table_name = 'events' AND column_name = 'ts';

Expected SQLSTATE

Links

• DML operations: dml.md
• DDL (stats options): ddl.md
• SQL compatibility overview: overview.md
• Known issues: Known issues

Table partitioning

Goal

Reference for table partitioning: RANGE, LIST, partition management, and runtime behavior.

PARTITION BY RANGE

CREATE TABLE events (
 id INTEGER PRIMARY KEY,
 ts DATE NOT NULL,
 data TEXT
) PARTITION BY RANGE (ts);

Range partitions

CREATE TABLE events_2025 PARTITION OF events
 FOR VALUES FROM ('2025-01-01') TO ('2026-01-01');

CREATE TABLE events_2026 PARTITION OF events
 FOR VALUES FROM ('2026-01-01') TO ('2027-01-01');

PARTITION BY LIST

CREATE TABLE metrics (
 id INTEGER PRIMARY KEY,
 region VARCHAR(20) NOT NULL,
 value BIGINT
) PARTITION BY LIST (region);

List partitions

CREATE TABLE metrics_eu PARTITION OF metrics
 FOR VALUES IN ('eu-west', 'eu-east');

CREATE TABLE metrics_us PARTITION OF metrics
 FOR VALUES IN ('us-east', 'us-west');

DEFAULT partition

CREATE TABLE events_other PARTITION OF events DEFAULT;

Rows that do not fit into any partition are routed to DEFAULT. If DEFAULT does not exist and no suitable partition is found — 23514 check_violation.

ALTER TABLE — attach / detach

ALTER TABLE events ATTACH PARTITION events_2027
 FOR VALUES FROM ('2027-01-01') TO ('2028-01-01');

ALTER TABLE events DETACH PARTITION events_2025;

DROP PARTITION

DROP PARTITION events_2025;

Runtime behavior

INSERT routing

An INSERT into the parent table automatically routes the row to the appropriate child partition:

INSERT INTO events (id, ts, data) VALUES (1, '2026-06-15', 'test');
-- → events_2026

If no partition matches and DEFAULT is missing:

ERROR: 23514: new row for relation "events" violates check constraint

Append-only inheritance

If the parent table is declared as append_only = true:

• All attached partitions inherit the append-only mode.
• A partition cannot disable append_only while the parent remains append-only — 42809.

CREATE TABLE events (
 id INTEGER PRIMARY KEY,
 ts DATE NOT NULL
) PARTITION BY RANGE (ts) WITH (append_only = true);

-- partitions inherit append_only
CREATE TABLE events_2026 PARTITION OF events
 FOR VALUES FROM ('2026-01-01') TO ('2027-01-01');

Current v0 bounds

Unsupported forms return 0A000 feature_not_supported.

Expected SQLSTATE

Links

• DDL reference: ddl.md
• DML (INSERT routing): dml.md
• SQL compatibility overview: overview.md
• Known issues: Known issues

Security model (overview)

Goal

Understand the layered security architecture of AngaraBase: which controls exist, how they interact, and how to verify your instance is running in a secure configuration.

Prerequisites

• A running AngaraBase instance (local or staging).
• SQL session access (pgwire).
• Basic understanding of roles and tables in your database.

Security model (layers)

AngaraBase uses a layered defence model. Each layer is independent and composable — no single layer bypass compromises the whole.

Layer 1 — Transport and identity

• TLS protects the wire protocol.
• Auth modes (trust, scram, cert) control how clients prove identity.
• Fail-closed: remote bind without TLS is rejected when tls.require_on_remote_bind = true.

See authentication.md for setup and verification.

Layer 2 — Authorization and data visibility

• RBAC (roles, grants, privileges) decides whether an operation is allowed at all.
• RLS (row-level security policies) decides which rows are visible or modifiable.
• Deny-by-default: enabling RLS without policies blocks all rows, including for the table owner.

See authorization.md for SQL surface and introspection.

Layer 3 — Controlled privilege escalation

• Break-glass is the only way to bypass RLS — even SUPERUSER cannot.
• Activation requires a mandatory REASON and TTL.
• Every query during break-glass generates a dedicated audit entry.

See break-glass.md for the full lifecycle.

Layer 4 — Audit and accountability

• Audit chain is append-only and tamper-evident (SHA-256 chain hash).
• Scope: auth, DDL, DCL, policy changes, break-glass lifecycle.
• DML audit policy: configurable off|allowlist|denylist per table.

See audit.md for configuration and verification.

Layer 5 — Data-at-rest protection

• TDE (Transparent Data Encryption) covers pages, WAL, and audit sink.
• Fail-closed: missing or invalid key material prevents startup and audit I/O.

See encryption.md for TDE setup and key management.

Layer 6 — Client-encrypted columns (v0)

• Server stores ciphertext + metadata (alg, mode, key_id) but never the keys.
• DETERMINISTIC mode allows equality predicates; RANDOMIZED rejects server-side predicates (0A000).

See encryption.md for the SQL surface and operator rules.

How features work together

Quick security verification

Step 1 — Check effective settings

SELECT name, value
FROM sys.settings
WHERE name LIKE 'tls.%'
 OR name LIKE 'security.%'
 OR name LIKE 'audit.%'
ORDER BY name;

Returns effective security knobs without exposing secrets.

Step 2 — Check security surfaces

SELECT * FROM angara_user_roles() LIMIT 20;
SELECT * FROM angara_table_policies('public.users');
SELECT * FROM angara_break_glass_status();
SELECT * FROM angara_audit_verify_chain();

Validates that key introspection/verification functions are available and responsive.

Step 3 — Validate RLS explanation surface

SELECT * FROM angara_effective_rls_predicate('public.users');

Returns the effective predicate and helps explain row-visibility behaviour.

Expected result

• sys.settings shows security knobs without secrets.
• Security functions return data (or empty results) without internal errors.
• Unsupported operations terminate with an explicit SQLSTATE (0A000, 42501, or 22023) — never a silent bypass.

Troubleshooting

• 42501 insufficient_privilege on security DDL/ops Check user roles and session context; see authorization.md.
• 0A000 feature_not_supported in policy/encrypted path This is a bounded contract (not a bug) — use the supported syntax or mode.
• TDE enabled but audit/data I/O fails Verify master key presence and correctness; fail-closed is expected. See encryption.md.
• Need a bug-report artifact? Follow the bundle steps in ../reference/support.md.

Links

• Security knobs registry: angarabook/src/operations/security-operations.md

• Authentication: authentication.md

• Authorization: authorization.md

• Audit: audit.md

• Encryption: encryption.md

• Break-glass: break-glass.md

• Hardening runbook: hardening.md

Authentication

Goal

Configure and verify transport security (TLS) and client authentication modes so that only identified clients can connect to AngaraBase.

Prerequisites

• Access to the server configuration (TOML config and/or environment variables).
• TLS certificate and key files (for scram or cert modes with remote clients).
• Ability to restart the server after configuration changes.

Auth modes

AngaraBase supports three authentication modes, set via ANGARABASE_AUTH_MODE or security.auth_mode in the config:

Default: trust (requires explicit opt-in flag for remote bind — see Startup safety below).

Steps

1) Configure TLS

TLS is controlled in the [tls] section of angarabase.conf:

[tls]
enabled = true
cert_path = "/etc/angarabase/tls/server.crt"
key_path = "/etc/angarabase/tls/server.key"
require_on_remote_bind = true

Or via environment variables:

export ANGARABASE_TLS_ENABLED=1
export ANGARABASE_TLS_CERT_PATH=/etc/angarabase/tls/server.crt
export ANGARABASE_TLS_KEY_PATH=/etc/angarabase/tls/server.key
export ANGARABASE_TLS_REQUIRE_ON_REMOTE_BIND=1

require_on_remote_bind = true enforces fail-closed behaviour: if the server binds to a non-loopback address and TLS is not enabled, startup is refused.

2) Set auth mode

export ANGARABASE_AUTH_MODE=scram

Or in angarabase.conf:

[security]
auth_mode = "scram"

3) SCRAM setup

When auth_mode = scram, the superuser must be bootstrapped with a password at init time:

angarabase-server --init /var/lib/angarabase \
 --superuser admin \
 --superuser-password 'strong-password' \
 --auth-mode scram

The password is converted to a SCRAM-SHA-256 verifier before it touches disk — plaintext is never stored.

Additional users are created via SQL:

CREATE USER app_reader WITH PASSWORD 'change-me';

Password storage format: SCRAM-SHA-256$<iterations>:<salt>$<StoredKey>:<ServerKey>.

4) Startup safety

To protect against accidental production use of trust mode:

• trust mode on a non-loopback bind address requires the explicit flag --allow-insecure-no-auth.
• Without this flag the server refuses to start (fail-closed).
• scram or cert modes do not require the flag.

5) Verify effective config from SQL

After startup, confirm the active settings:

SELECT name, value
FROM sys.settings
WHERE name IN (
 'tls.enabled',
 'tls.require_on_remote_bind',
 'security.auth_mode'
)
ORDER BY name;

No secret material (keys, passwords, verifiers) appears in sys.settings.

Expected result

• In scram mode, unauthenticated connections are rejected.
• In cert mode, clients without a valid certificate are rejected.
• trust on remote bind without --allow-insecure-no-auth prevents startup.
• sys.settings confirms the effective auth mode and TLS state without leaking secrets.

Troubleshooting

• Server refuses to start in trust mode You are binding to a non-loopback address. Either switch to scram/cert or pass --allow-insecure-no-auth for development.
• authentication failed for user "..." on connect Verify the password or certificate; check that security.auth_mode matches the client’s auth method.
• TLS handshake failure Confirm cert_path and key_path point to valid, non-expired files; verify the client’s sslmode setting.
• Init refused: scram requires password Provide --superuser-password (or --superuser-password-file) when --auth-mode scram is set.
• Need a bug-report artifact? See ../reference/support.md.

Links

• Security model overview: overview.md
• Hardening runbook: hardening.md
• Configuration reference: ../operations/configuration.md
• Security knobs registry: angarabook/src/operations/security-operations.md

Authorization (RBAC and RLS)

Goal

Set up role-based access control and row-level security policies so that users see and modify only the data they are permitted to.

Prerequisites

• SQL session with a user that has SUPERUSER or SECURITY_ADMIN privileges.
• A test table (e.g. public.users) with representative data.

RBAC — users, roles, privileges

Create a user and grant a role

CREATE USER app_reader WITH PASSWORD 'change-me';
GRANT reader TO app_reader;
GRANT SELECT ON public.users TO reader;

Object-level grants

GRANT SELECT ON TABLE public.orders TO analyst;
GRANT INSERT, UPDATE ON TABLE public.orders TO writer_role;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO analyst;

Default policy is deny-by-default: a user without an explicit GRANT cannot access another user’s tables.

Built-in role hierarchy

SUPERUSER
├── SECURITY_ADMIN — policy, audit, key, and break-glass management
├── DBA — ops (shutdown, backup, restore, settings, diagnostics)
├── CREATEROLE — CREATE/ALTER/DROP USER and ROLE
└── CREATEDB — CREATE DATABASE

SUPERUSER does not bypass RLS — only BREAK_GLASS can (see break-glass.md).

RLS — row-level security

Enable RLS on a table

ALTER TABLE public.users ENABLE ROW LEVEL SECURITY;

After this, the table follows deny-by-default: no rows are visible to anyone (including the owner) until at least one policy is created.

Create a security policy

CREATE SECURITY POLICY p_users_tenant ON public.users
 USING (tenant_id = current_setting('app.tenant_id')::int);

The USING predicate is injected as an automatic WHERE clause on SELECT, INSERT, UPDATE, and DELETE.

RLS enforcement rules

Multiple policies on one table use AND semantics — all must pass.

RLS v1 masking metadata

Policies can include a MASK clause for user-facing field masking:

ALTER SECURITY POLICY p_users_tenant ON public.users
 USING (tenant_id = current_setting('app.tenant_id')::int)
 MASK (email USING 'partial');

Supported mask types in v1:

Unsupported mask expressions return 0A000 feature_not_supported.

Unsupported predicates — fail-closed

In the IR/planner mode, complex predicates that cannot be safely rewritten (subqueries, arbitrary function calls, JOINs) are rejected:

ERROR: 0A000 feature_not_supported
 unsupported RLS predicate form

This is a bounded contract, not a bug. Use the supported predicate language (column references, current_setting(), current_user, literals, comparisons, AND/OR/NOT, IN, IS [NOT] NULL, type casts).

Introspection

SELECT * FROM angara_table_policies('public.users');
SELECT * FROM angara_effective_rls_predicate('public.users');
SELECT * FROM sys.users;
SELECT * FROM sys.roles;
SELECT * FROM sys.user_roles;
SELECT * FROM sys.role_privileges;
SELECT * FROM sys.object_grants;
SELECT * FROM sys.my_privileges;
SELECT * FROM sys.security_policies;
SELECT * FROM angara_user_roles('alice');
SELECT * FROM angara_user_privileges('alice');
SELECT angara_has_privilege('alice', 'SELECT', 'TABLE', 'public.orders');
SELECT angara_is_rls_active('public.users');

angara_effective_rls_predicate() shows the combined predicate, provenance, and mask metadata for a table — useful for explaining row-visibility behaviour.

Expected result

• RBAC grants control object-level access; users without grants are denied.
• RLS predicates filter rows transparently on SELECT/INSERT/UPDATE/DELETE.
• angara_table_policies reflects active policies, mask metadata, and provenance.
• Unsupported paths return a deterministic SQLSTATE, never a silent bypass.

Troubleshooting

• 42501 insufficient_privilege The current user lacks the required role or grant. Check sys.my_privileges and angara_user_roles().
• 0A000 feature_not_supported on RLS policy or mask The predicate or mask expression is outside the supported v1 syntax. Simplify the expression.
• Rows unexpectedly invisible after enabling RLS Deny-by-default is working correctly. Create a SECURITY POLICY with a USING predicate to allow the intended rows.
• Need a bug-report artifact? See ../reference/support.md.

Links

• Security model overview: overview.md
• Break-glass (RLS bypass): break-glass.md
• Audit: audit.md
• SQL compatibility: ../sql-reference/overview.md
• Known issues: ../reference/known-issues.md

Audit

Goal

Understand and configure AngaraBase’s tamper-evident audit subsystem: chain verification, DML audit policy, and JSON export.

Prerequisites

• A running AngaraBase instance with audit enabled (default).
• SQL session with SECURITY_ADMIN or SUPERUSER privileges (for configuration changes).
• Write-accessible path for the audit log file.

Audit chain concept

AngaraBase maintains an append-only, tamper-evident audit trail:

• Every security-relevant event (auth, DDL, DCL, policy changes, break-glass lifecycle) produces an AuditEvent.
• Each event contains prev_hash — a SHA-256 hash of the preceding entry, forming a hash chain.
• Breaking or modifying any entry invalidates all subsequent hashes, making tampering detectable.
• The chain is separate from the transactional data path — audit events are recorded even for rolled-back transactions.

Audit scope

Steps

1) Verify chain integrity

SELECT * FROM angara_audit_verify_chain();

Returns is_valid, first_broken_seq, and details. A healthy chain returns is_valid = true.

2) Query the audit log

SELECT * FROM sys.audit_log
WHERE event_type = 'break_glass_query'
 AND timestamp > now() - INTERVAL '24 hours'
ORDER BY seq DESC
LIMIT 50;

Columns: seq, timestamp, event_type, user_name, auth_method, client_ip, database, session_claims, payload, prev_hash.

3) Configure audit v1 DML policy

DML audit is controlled by three knobs:

export ANGARABASE_AUDIT_DML_MODE=allowlist
export ANGARABASE_AUDIT_DML_ALLOWLIST=public.users,public.payments

Use ANGARABASE_AUDIT_DML_DENYLIST for the denylist.

Malformed policy or ambiguous object references cause a startup/config-apply rejection (fail-closed).

4) Configure audit log path

export ANGARABASE_AUDIT_LOG_PATH=/var/lib/angarabase/audit/audit.jsonl

The path must be writable. If the path is inaccessible, audit writes fail-closed.

5) Configure JSON export

export ANGARABASE_AUDIT_EXPORT_JSON_ENABLED=1
export ANGARABASE_AUDIT_EXPORT_RATE_LIMIT_RPS=50

Export is bounded and rate-limited. Export failures are reported but never expose secret payload fragments in error text.

6) TDE interaction

When TDE is enabled (ANGARABASE_TDE_ENABLE=1), the audit sink on disk is encrypted transparently:

• Key: audit_dek = KDF(master_key, domain="audit-v0", key_id).
• sys.audit_log decrypts on read when the correct key is available.
• Without the key, audit read/write is impossible (fail-closed).
• Encrypted audit data remains encrypted in backups and copied artefacts.

See encryption.md for TDE configuration.

Expected result

• angara_audit_verify_chain() returns is_valid = true for an intact chain.
• sys.audit_log shows auth, DDL, DCL, policy, and break-glass events.
• With DML policy set to allowlist or denylist, matching DML operations appear in the audit log.
• TDE-encrypted audit files are unreadable without the master key.

Troubleshooting

• angara_audit_verify_chain() returns is_valid = false The chain has been tampered with or corrupted. Note the first_broken_seq and investigate the audit file. The audit subsystem can self-repair by truncating to the last valid entry.
• DML events not appearing in audit log Check audit.dml_mode — default is off. Verify that the target table is in the allowlist (or not in the denylist).
• Audit write failures after enabling TDE Verify ANGARABASE_TDE_MASTER_KEY_HEX and key validity. Fail-closed behaviour is expected when key material is missing.
• Break-glass activation fails with “audit subsystem unavailable” Break-glass requires a healthy audit subsystem. Fix the audit path or key material first.
• Need a bug-report artifact? See ../reference/support.md.

Links

• Security model overview: overview.md
• Encryption (TDE): encryption.md
• Break-glass: break-glass.md
• Configuration reference: ../operations/configuration.md

Encryption

Goal

Configure data-at-rest protection with TDE and understand the server-side contract for client-encrypted columns.

Prerequisites

• Access to server environment variables or configuration file.
• A 256-bit master key (64 hex characters) for TDE.
• Ability to restart the server after TDE changes.

TDE — Transparent Data Encryption (v0)

TDE encrypts data at rest without application changes. When enabled, the following are encrypted:

• Storage pages (the .adb data file).
• WAL (write-ahead log) entries.
• Audit sink on disk (JSONL trail).

Configure TDE

export ANGARABASE_TDE_ENABLE=1
export ANGARABASE_TDE_MASTER_KEY_HEX=<64-hex-characters>
export ANGARABASE_TDE_MASTER_KEY_ID=master-prod-2026q1
export ANGARABASE_TDE_LAST_ROTATION_UNIX=1760000000

Fail-closed behaviour

• If ANGARABASE_TDE_ENABLE=1 and the key is missing, malformed, or wrong length, the server refuses to start.
• If the key is invalid for the existing data directory, page/WAL decryption fails at startup — also fail-closed.
• The audit sink follows the same rule: without a valid key, audit read/write is impossible.

Verify from SQL

SELECT name, value
FROM sys.settings
WHERE name IN (
 'security.tde_enabled',
 'security.tde_master_key_id'
)
ORDER BY name;

Only non-secret metadata (key_id, enabled flag) is exposed. The hex key itself never appears.

Client-encrypted columns

Client-encrypted columns allow applications to store ciphertext in the database while the server never holds key material.

SQL surface

ALTER TABLE customers
 ALTER COLUMN tax_id
 SET ENCRYPTED WITH (TYPE=DETERMINISTIC, KEY_ID='cust-key-01');

Full syntax:

ALTER TABLE <table>
 ALTER COLUMN <column>
 SET ENCRYPTED WITH (
 TYPE = DETERMINISTIC | RANDOMIZED,
 KEY_ID = '<opaque-id>',
 ALG = '<algorithm-id>' -- optional, defaults to AES-256-SIV / AES-256-GCM
 );

What the server stores

The server never accepts raw key material via SQL, environment variables, config, logs, or sys.* views.

Operator rules

Unsupported operations return 0A000 feature_not_supported (shape-stable error).

Fail-closed and observability

• Missing or invalid encryption metadata on a column causes DML to be rejected.
• sys.settings and diagnostics expose only key_id, mode, and algorithm — never ciphertext blobs or key-like material.
• Error messages never include ciphertext fragments or key material.

Expected result

• With TDE enabled, the data directory contains no plaintext data, WAL, or audit payloads.
• sys.settings shows security.tde_enabled = true and the key_id without exposing the key.
• Client-encrypted columns store ciphertext; deterministic mode allows equality queries; randomized mode rejects server-side predicates.
• All fail-closed paths produce deterministic errors, not silent fallbacks.

Troubleshooting

• Server refuses to start after enabling TDE Check ANGARABASE_TDE_MASTER_KEY_HEX — must be exactly 64 hex characters. If the data directory was previously unencrypted, you cannot enable TDE retroactively on existing data (see migration docs).
• 0A000 feature_not_supported on encrypted column query The query uses an unsupported operator for the column’s encryption mode. For RANDOMIZED columns, only read/write is allowed. For DETERMINISTIC, only equality predicates work.
• Audit read/write fails with TDE enabled The audit DEK derives from the master key. Verify the master key matches the one used when the audit file was created.
• sys.settings does not show TDE knobs Ensure ANGARABASE_TDE_ENABLE=1 is set in the environment before server startup.
• Need a bug-report artifact? See ../reference/support.md.

Links

• Security model overview: overview.md
• Audit (TDE interaction): audit.md
• Hardening runbook: hardening.md
• Configuration reference: ../operations/configuration.md
• Security knobs registry: angarabook/src/operations/security-operations.md

Break-glass

Goal

Understand and use controlled privilege escalation (break-glass) to temporarily bypass RLS policies with full auditability.

Prerequisites

• SQL session with a user who has been granted the BREAK_GLASS capability.
• A healthy audit subsystem (break-glass cannot activate if audit is unavailable).

What is break-glass?

Break-glass is AngaraBase’s mechanism for controlled, time-limited, fully audited bypass of row-level security policies. It exists because SUPERUSER alone does not bypass RLS — this is a deliberate design choice.

Key properties:

• Activation requires a mandatory reason and a mandatory TTL.
• Every query executed during break-glass generates a dedicated break_glass_query audit entry.
• There is no “silent” bypass — the audit chain always records break-glass activity.
• This is a first-in-class database feature: neither PostgreSQL nor MS SQL Server have a built-in break-glass mechanism with TTL + reason + mandatory audit.

Steps

1) Grant the break-glass capability

A SECURITY_ADMIN grants the BREAK_GLASS role to a user or role:

GRANT BREAK_GLASS TO dba_team;

2) Activate break-glass

The user who has been granted BREAK_GLASS activates it with a reason and duration:

SET BREAK_GLASS REASON='INCIDENT-789: data corruption investigation' TTL='2h';

Duration format: '15m', '2h', '1d' etc. Maximum TTL is controlled by the server configuration (see below).

3) Check status

SELECT * FROM angara_break_glass_status();

Returns: is_active, reason, expires_at, activated_at.

4) Work under break-glass

While break-glass is active, RLS policies are bypassed. Every query in this session generates an audit entry with event_type = 'break_glass_query', including the full (sanitized) SQL text.

5) Deactivate (manual or automatic)

RESET BREAK_GLASS;

If not deactivated manually, break-glass auto-expires when the TTL elapses. After expiry, RLS applies again immediately.

6) Revoke the capability

REVOKE BREAK_GLASS FROM dba_team;

Configuration

Also exposed as security.break_glass_max_ttl in sys.settings.

Audit trail

All break-glass lifecycle events are recorded:

Invariants

1. Audit must be healthy. If the audit subsystem is down or corrupted, break-glass activation fails (fail-closed).
2. TTL is mandatory. SET BREAK_GLASS without TTL → error.
3. Reason is mandatory. SET BREAK_GLASS without REASON → error.
4. Max TTL is server-enforced. Exceeding security.break_glass_max_ttl → 22023 invalid_parameter_value.
5. No refresh. A client cannot extend the TTL — deactivate and re-activate with a new reason/TTL instead.
6. SUPERUSER ≠ RLS bypass. Only BREAK_GLASS bypasses RLS.

Expected result

• SET BREAK_GLASS with valid reason and TTL activates bypass; angara_break_glass_status() confirms.
• All queries during break-glass appear in sys.audit_log with event_type = 'break_glass_query'.
• After TTL expiry or RESET BREAK_GLASS, RLS enforcement resumes.
• Invalid TTL returns 22023; missing reason or TTL returns an error.

Troubleshooting

• 22023 invalid_parameter_value on SET BREAK_GLASS The TTL exceeds security.break_glass_max_ttl or is in an invalid format. Check the max TTL setting and use a supported duration format ('15m', '2h', '1d').
• 42501 insufficient_privilege on SET BREAK_GLASS The current user has not been granted BREAK_GLASS. A SECURITY_ADMIN must run GRANT BREAK_GLASS TO <user>.
• Break-glass activation fails with “audit unavailable” The audit subsystem must be healthy. Check ANGARABASE_AUDIT_LOG_PATH and audit key material if TDE is enabled.
• Break-glass expired unexpectedly TTL is server-enforced and cannot be refreshed. Deactivate and re-activate with a new reason and TTL.
• Need a bug-report artifact? See ../reference/support.md.

Links

• Security model overview: overview.md
• Authorization (RLS policies): authorization.md
• Audit: audit.md
• Known issues: ../reference/known-issues.md

Deployment hardening runbook

Goal

Walk through a step-by-step process to launch AngaraBase in a production-secure configuration, and verify the result from SQL.

Prerequisites

• Access to server configuration (TOML config and/or environment variables).
• TLS certificate and key files.
• A 256-bit master key (64 hex characters) for TDE.
• Ability to restart the server.

Security hardening checklist

Before starting, confirm each item applies to your deployment:

• TLS enabled for all remote connectivity.
• Auth mode set explicitly (scram or cert) — not left as trust for production.
• TDE enabled with valid key material.
• Audit log path set and writable.
• DML audit policy chosen deliberately (off, allowlist, or denylist).
• Break-glass max TTL configured appropriately.
• angara_audit_verify_chain() runs clean.

Steps

Step 1 — Enable TLS

[tls]
enabled = true
cert_path = "/etc/angarabase/tls/server.crt"
key_path = "/etc/angarabase/tls/server.key"
require_on_remote_bind = true

This protects the wire protocol and enforces fail-closed on non-loopback bind without TLS.

See authentication.md for TLS details.

Step 2 — Set auth mode

export ANGARABASE_AUTH_MODE=scram

Disables trust-only behaviour for production. The superuser must have been bootstrapped at --init time with a SCRAM password.

See authentication.md for auth modes and SCRAM setup.

Step 3 — Enable TDE

export ANGARABASE_TDE_ENABLE=1
export ANGARABASE_TDE_MASTER_KEY_HEX=<64-hex-secret>
export ANGARABASE_TDE_MASTER_KEY_ID=master-prod-2026q1
export ANGARABASE_TDE_LAST_ROTATION_UNIX=1760000000

Enables at-rest encryption for pages, WAL, and audit sink. Without a valid key the server refuses to start (fail-closed).

See encryption.md for TDE configuration and key management.

Step 4 — Configure audit policy

export ANGARABASE_AUDIT_LOG_PATH=/var/lib/angarabase/audit/audit.jsonl
export ANGARABASE_AUDIT_DML_MODE=allowlist
export ANGARABASE_AUDIT_DML_ALLOWLIST=public.users,public.payments
export ANGARABASE_AUDIT_EXPORT_JSON_ENABLED=1
export ANGARABASE_AUDIT_EXPORT_RATE_LIMIT_RPS=50

Sets targeted DML audit coverage and bounded JSON export.

See audit.md for audit policy options and chain verification.

Step 5 — Verify effective config from SQL

After starting the server, confirm all settings from a SQL session:

SELECT name, value
FROM sys.settings
WHERE name IN (
 'tls.enabled',
 'tls.require_on_remote_bind',
 'security.auth_mode',
 'security.tde_enabled',
 'security.tde_master_key_id',
 'audit.dml_mode',
 'audit.export_json_enabled',
 'audit.export_rate_limit_rps'
)
ORDER BY name;

Only non-secret metadata appears. The TDE hex key, passwords, and SCRAM verifiers are never exposed.

Step 6 — Verify audit chain

SELECT * FROM angara_audit_verify_chain();

A healthy deployment returns is_valid = true.

Step 7 — Verify security surfaces

SELECT * FROM angara_user_roles() LIMIT 20;
SELECT * FROM angara_break_glass_status();

Confirm that introspection functions are responsive and return expected data.

Expected result

• Server starts only in fail-closed mode for unsafe configurations.
• sys.settings shows only non-secret metadata.
• Disk contains no plaintext audit payloads when TDE is enabled.
• Audit chain is intact; DML events appear for tables in the allowlist.
• Authentication rejects unauthenticated connections in scram/cert mode.

Troubleshooting

• Server does not start after enabling TDE Verify ANGARABASE_TDE_MASTER_KEY_HEX is exactly 64 hex characters and matches the data directory’s key. Fail-closed is expected.
• Auth/TLS conflict on remote bind If binding to a non-loopback address, tls.enabled must be true or --allow-insecure-no-auth must be passed (dev only). Check tls.require_on_remote_bind and server.host.
• DML audit events missing Verify audit.dml_mode is not off and that the target table matches the allowlist/denylist entries. Table names must be fully qualified (schema.table).
• angara_audit_verify_chain() returns is_valid = false The audit chain is corrupted. Note the first_broken_seq and investigate the audit file.
• Break-glass cannot activate The audit subsystem must be healthy. Fix the audit path or TDE key material first.
• Need a bug-report artifact? See ../reference/support.md.

Links

• Security model overview: overview.md
• Authentication: authentication.md
• Authorization: authorization.md
• Audit: audit.md
• Encryption: encryption.md
• Break-glass: break-glass.md
• Configuration reference: ../operations/configuration.md
• Security knobs registry: angarabook/src/operations/security-operations.md

GOST Security Compliance & Testing Guide

Status: TLS implemented, TDE planned Target Audience: Security Auditors, DevOps, QA

1. GOST Security Ecosystem in AngaraBase

AngaraBase implements a layered approach to Russian national cryptographic standards (GOST).

1.1. Transport Layer (TLS) — Available

Protection of data in transit using GOST R 34.10-2012 (Public Key) and GOST 28147-89 (Cipher suites).

• Implementation: Provider-based abstraction (OpenSSL Engine / Rustls).
• Policy: Fail-closed (server refuses to start if configured GOST provider is missing).
• Configuration: tls.gost_enabled, tls.gost_cipher_suites.

1.2. Data-at-Rest (TDE) — Planned

Protection of data on disk (Pages, WAL, Audit Logs) using block ciphers Kuznyechik (GOST 34.12-2015) or Magma.

• Scope: Transparent Data Encryption (TDE) for storage files.
• Key Management: Integration with external KMS supporting GOST keys.
• Status: Roadmap item.

1.3. Integrity & Authentication — Future

• Hashing: Migration from SHA-256 to Streebog (GOST R 34.11-2012) for data checksums and SCRAM authentication.
• Audit Signing: Digital signature of audit logs to ensure non-repudiation.

2. Testing GOST TLS Support

This guide describes how to verify that AngaraBase is correctly using GOST cipher suites and strictly enforcing the fail-closed policy.

Prerequisites

You need a Linux environment with OpenSSL configured for GOST.

# Debian/Ubuntu
sudo apt-get install openssl libssl-dev libengines-gost

# Verify engine availability
openssl engine gost -t
# Output should contain: [gost] Reference implementation of GOST engine -> [ available ]

Step 1: Generate GOST Certificates

Standard RSA/ECDSA certificates will not work with GOST cipher suites. You must generate keys using GOST algorithms.

# 1. Generate a private key using GOST R 34.10-2012 (256 bit)
openssl genpkey -algorithm gost2012_256 -pkeyopt paramset:A -out gost_server.key

# 2. Generate a self-signed certificate
openssl req -new -x509 -days 365 \
 -key gost_server.key \
 -out gost_server.crt \
 -subj "/CN=localhost"
 
# 3. Verify the certificate algorithm
openssl x509 -in gost_server.crt -text -noout | grep "Signature Algorithm"
# Expected: Signature Algorithm: GOST R 34.10-2012 with GOST R 34.11-2012 (256 bit)

Step 2: Configure AngaraBase

Enable TLS and GOST mode. Ensure allow_insecure is OFF to test strict mode.

export ANGARABASE_TLS_ENABLE=1
export ANGARABASE_TLS_CERT_PATH=$(pwd)/gost_server.crt
export ANGARABASE_TLS_KEY_PATH=$(pwd)/gost_server.key
export ANGARABASE_TLS_GOST_ENABLED=1
export ANGARABASE_TLS_GOST_CIPHER_SUITES="GOST2012-GOST8912-GOST8912"

# Start the server
./angarabase-server

Step 3: Verification (Positive Test)

Connect using a client that supports GOST (e.g., openssl s_client or a patched psql).

Using OpenSSL s_client:

openssl s_client -connect localhost:5152 -servername localhost

Verification Checklist:

1. Look for Cipher : GOST2012-GOST8912-GOST8912 (or similar GOST suite) in the output.
2. Look for Protocol : TLSv1.2.
3. Ensure the handshake completes successfully.

Using SQL (if psql supports it):

SELECT name, value FROM sys.settings WHERE name LIKE 'tls.%';
-- Verify tls.gost_enabled is 'true'

Step 4: Fail-Closed Verification (Negative Test)

Verify that the server refuses to start if the environment is broken.

1. Scenario A: Missing Provider. Temporarily disable the GOST engine (e.g., by renaming the library or changing OpenSSL config) and try to start AngaraBase with ANGARABASE_TLS_GOST_ENABLED=1.

• Expected Result: Server panic/exit with “GOST provider not available”.

2. Scenario B: Invalid Cipher Suite. Set ANGARABASE_TLS_GOST_CIPHER_SUITES="INVALID-CIPHER".

• Expected Result: Server panic/exit with configuration error.

3. Scenario C: RSA Certificate with GOST Ciphers. Try to start with ANGARABASE_TLS_GOST_ENABLED=1 but provide standard RSA certificates.

• Expected Result: Handshake failures (OpenSSL error: “no shared cipher” or “wrong signature type”).

3. Troubleshooting

Next steps

Once you have determined which GOST scenarios you need:

• GOST crypto setup — step-by-step installation of the cryptographic provider and profile switching.
• Encryption (TDE + client) — general contract for TDE and client encryption.
• Audit — how to close GOST signatures on the audit-chain.
• Hardening runbook — final checklist before production.

Installation

AngaraBase is distributed through package repositories. Public access is free and requires no registration.

Supported operating systems

• RHEL-compatible (8+): Red Hat Enterprise Linux, CentOS Stream, AlmaLinux, Rocky Linux, Oracle Linux, Fedora.
• Gentoo Linux: supported via the official binhost.

RPM (RHEL / CentOS / Alma / Fedora)

Installation is performed using the standard dnf package manager.

1. Import the GPG key

Import the public key used to sign the packages:

rpm --import https://rpm.angarabase.dev/release-key.asc

2. Add the repository

Create the repository configuration file:

cat > /etc/yum.repos.d/angarabase.repo << 'EOF'
[angarabase]
name=AngaraBase
baseurl=https://rpm.angarabase.dev/el$releasever/stable/
enabled=1
gpgcheck=1
gpgkey=https://rpm.angarabase.dev/release-key.asc
EOF

3. Install the package

Install the server package:

dnf install angarabase-server

Gentoo Linux

Gentoo users can install from a pre-built binhost with compiled packages.

1. Import the GPG key

gpg --fetch-keys https://rpm.angarabase.dev/release-key.asc

2. Add the binhost

Add the repository to your Portage configuration:

cat >> /etc/portage/binrepos.conf << 'EOF'
[angarabase]
priority = 9999
sync-uri = https://rpm.angarabase.dev/gentoo/stable/
EOF

3. Install the package

Install the package using the binary build:

emerge --getbinpkg angarabase

Enterprise repository

Enterprise subscribers get access to a private repository with binary builds. Access is secured via mTLS (mutual certificate authentication).

Setup instructions are provided together with your invitation code when the subscription is activated. If you are a customer and do not have the instructions, contact us at team@angarabase.com.

More information: angarabase.dev/enterprise

Configuration

Goal

Configure a AngaraBase instance for local testing or production deployment using TOML config keys, data-directory conventions, and environment variable overrides.

Prerequisites

• Built angarabase-server binary (see Quickstart)

Minimal config keys

Server

[server]
addr = "127.0.0.1:5152" # host:port; use 0.0.0.0:PORT to bind all interfaces

Storage

[storage]
data_directory = "/var/lib/angarabase/data"
transaction_log_directory = "/var/lib/angarabase/txlog"

Logging

[logging]
log_level = "info"
log_directory = "/var/log/angarabase"

Ops (metrics / admin listeners)

[ops]
metrics_addr = "127.0.0.1:9898" # Prometheus metrics endpoint; empty = disabled
# admin_addr = "127.0.0.1:9999" # reserved, not yet active

Transaction log (durability)

[transaction_log]
backend = "noop"
durability = "strict"
fsync = true

TLS (optional)

[tls]
enabled = true
cert_path = "/etc/angarabase/tls/server.crt"
key_path = "/etc/angarabase/tls/server.key"

# If true and server.host is non-loopback, TLS is required (fail-closed).
require_on_remote_bind = true

WAL (Write-Ahead Log)

[wal]
vlf_enable = true
max_size_mb = 512
vlf_size_mb = 32
init_vlfs = 2
auto_shrink = true

SQL Execution

[execution]
mode = "auto" # auto | force_vector | force_row
vector_batch_size = 1024
query_memory_limit_mb = 256

Adaptive Query Processing (AQP)

[aqp]
enabled = true
mode = "conservative" # conservative | aggressive
min_query_time_ms = 100
learning_rate = 0.1
max_correction = 100.0
variance_threshold = 10.0
correction_cache_mb = 64
store_capacity_mb = 1024

Diagnostics

[diagnostics]
log_min_duration_ms = -1 # -1 = disabled
log_query_text = false
stat_statements_max = 0 # 0 = disabled

Initialization workflow

Run --init before the first normal start:

angarabase-server --config /path/to/angarabase.conf --init

• Directories data_directory and transaction_log_directory must be writable.
• When a valid logging.log_directory is set, the server writes angarabase-server.log.
• --init creates all required data-directory artifacts (see below).

For local dev/test shortcuts see Quickstart.

Data directory artifacts

Inside storage.data_directory the server creates and maintains:

Legacy text artifacts (identity_v0.txt, sys_catalog/*.txt) are no longer the source of truth and are not created by --init. If they remain from older versions they are ignored.

Environment variables

Environment variables override the corresponding config keys. When both are set, the env variable wins.

Precedence rule: default → config → env override

All parameters can be configured in angarabase.conf. Environment variables are intended for operational override without restart (for dynamic parameters) or for diagnostics. For permanent configuration, use the TOML config file.

Core server knobs

TLS knobs

Transaction log knobs

WAL (Write-Ahead Log) knobs

Diagnostics / slow-query knobs

Optimizer planning knobs

Vector execution knobs

Parallel execution governance knobs

Adaptive query processing knobs

OpenTelemetry tracing knobs

See also Diagnostics for query performance analysis.

Authentication knobs

Startup safety: trust/no-auth mode now requires explicit --allow-insecure-no-auth; without this flag the server refuses to start when ANGARABASE_AUTH_MODE resolves to trust/no-auth.

Security / break-glass knobs

Audit knobs

AngaraIO v2 I/O Scheduler knobs

Note: High-priority queue is unbounded per RFC invariant to ensure critical operations never block.

TDE (Transparent Data Encryption) knobs

Security note: when ANGARABASE_TDE_ENABLE=1 and ANGARABASE_AUDIT_LOG_PATH is set, the audit sink is encrypted at rest; without valid key material audit read/write is fail-closed (no plaintext fallback).

Expected result

After a successful --init and start:

curl -sS http://127.0.0.1:9898/health/ready # port matches [ops].metrics_addr

Returns {"status":"ready"}. Effective configuration is visible via:

SELECT * FROM sys.settings;

Troubleshooting

For unresolved issues see Known issues and Support.

Links

• Security/ops knobs registry (full defaults, fail-closed gates, sys.settings contract): angarabook/src/operations/security-operations.md
• Operator runbook: angarabook/src/operations/troubleshooting.md
• Security overview: Security model
• Quickstart: Quickstart

Container Deployment Quickstart

Operator quickstart for image-first AngaraBase startup:

• local docker run with a cgroup-aware startup probe;
• minimal k8s deployment (single-node, without HA);
• smoke checks for readiness and basic diagnostics.

0) 30-second evaluator path (image-only)

If you need a quick evaluator/DPP smoke without a Rust toolchain:

tools/compat_suite/image_smoke.sh

The script starts a container from the canonical Dockerfile, waits for /health/ready, checks the startup log (deployment probe resolved effective budgets, cgroup_version), and verifies probe metrics in /metrics.

1) Local container smoke (docker run)

Build the image:

docker build -t angarabase:local .

Run with a memory limit (probe contract):

docker run --rm --name angarabase-local \
  --memory=512m \
  -e ANGARABASE_DEPLOYMENT_PROFILE=container \
  -p 5152:5152 \
  -p 9898:9898 \
  angarabase:local

Expected signal in logs:

• the deployment probe resolved effective budgets line is present;
• deployment_profile=container;
• memory_source/cpu_source reflect cgroup_v1 or cgroup_v2 (or proc_fallback with fallback_reason).

Check readiness in the container:

curl -fsS http://127.0.0.1:9898/health/ready

The container HEALTHCHECK uses the same endpoint, so it becomes green only after health_readiness_reason_v0() passes.

Check metrics:

curl -fsS http://127.0.0.1:9898/metrics | rg "deployment_probe|deployment_profile"

2) Resource-limit examples

CPU + memory cap:

docker run --rm --name angarabase-capped \
  --memory=1g \
  --cpus=1.5 \
  -e ANGARABASE_DEPLOYMENT_PROFILE=container \
  -p 5152:5152 \
  -p 9898:9898 \
  angarabase:local

Explicit override (the probe must not overwrite it):

docker run --rm \
  --memory=512m \
  -e ANGARABASE_DEPLOYMENT_PROFILE=container \
  -e ANGARABASE_MEMORY_BUDGET=256MB \
  -e ANGARABASE_CPU_BUDGET=1000m \
  -p 5152:5152 \
  -p 9898:9898 \
  angarabase:local

Expected: the budget source in metrics/logs becomes config_override.

3) Kubernetes minimal smoke

Apply manifests:

kubectl apply -f tools/deploy/kubernetes/minimal/namespace.yaml
kubectl apply -f tools/deploy/kubernetes/minimal/configmap.yaml
kubectl apply -f tools/deploy/kubernetes/minimal/service.yaml
kubectl apply -f tools/deploy/kubernetes/minimal/statefulset.yaml

Verify:

kubectl -n angarabase get pods
kubectl -n angarabase get svc angarabase
kubectl -n angarabase logs statefulset/angarabase
kubectl -n angarabase get statefulset angarabase -o yaml | rg "resources:|health/"

Important: resources.requests/limits are set in statefulset.yaml. If limits are removed, the probe may see “unlimited” and switch to a host-level fallback for budgets.

Delete:

kubectl delete namespace angarabase

4) Diagnostics and backup entrypoints

For the packaged/operator path, use angara-cli instead of direct tools/... calls:

angara-cli diagnostics bundle --root artifacts/diagnostics/container-smoke --json
angara-cli backup full --config /etc/angarabase/angarabase.conf --out /tmp/base_full.abk

Related runbooks:

• Diagnostics bundle runbook
• Backup and restore

DPP alignment:

• for resource-limit safety rails and fail-closed mode, see business_strategy/PILOT_CHECKLIST.md (§5 and §6).

Backup and restore

Goal

Create and restore backups of a AngaraBase instance — from cold/offline baselines through online FULL + LOG chain with point-in-time recovery (PITR).

Prerequisites

• Initialized AngaraBase instance (angarabase-server --init; see Configuration)
• Built angara-cli binary (see Quickstart)
• For online backups (phase 1b): a running server with WAL (transaction_log backend ≠ noop)

Cold / offline backup

The original supported method. The server must be stopped (or will be stopped by the runner).

1. Copy storage.data_directory and storage.transaction_log_directory.
2. After restore, run the txlog-level oracle.
3. The source must be a correctly initialized data directory.

Pinned commands (legacy shell)

Backup:

tools/backup_restore/run.sh backup \
 --data-dir /var/lib/angarabase/data \
 --txlog-dir /var/lib/angarabase/transaction_log \
 --out /tmp/angarabase-backup.tar.gz \
 --root artifacts/backup_restore/backup

Restore:

tools/backup_restore/run.sh restore \
 --archive /tmp/angarabase-backup.tar.gz \
 --dest /tmp/angarabase-restore \
 --force \
 --root artifacts/backup_restore/restore

Oracle (post-restore verification):

tools/backup_restore/oracle.sh --root artifacts/backup_restore_oracle/run_1

Remote Admin Backup Flow (Online Backup via Admin API)

AngaraBase supports triggering backups remotely via the Admin TCP endpoint. This is the recommended approach for production deployments, as it does not require direct file system access to the database node.

Requirements

• The angarabase-server must be running with the Admin API enabled (configured via admin.listen_address in angarabase.conf or ANGARABASE_ADMIN_ADDR environment variable).
• angara-cli must be executed with the --addr <host:port> parameter instead of --config.
• The user executing the backup must have network access to the Admin API port (default: 9899).

How it works

When using --addr, angara-cli connects to the running server’s Admin API and issues a RUN backup_full command. The server coordinates the backup process internally (Fuzzy Copy, WAL Inclusion) and streams the resulting .angarabk file back to the client over the network.

This allows operators to take backups from a central management node without SSH access to the database servers.

Storage layout note

User tables use the page-based .adb path. Legacy heap_store/*.bin is migrated to .adb on first access.

For backup/restore and triage:

• Both files per database are critical: <db>.adb and <db>.atl.
• heap_store/ is not the source of truth after migration.
• Upgrading old instances may take time proportional to heap_store volume.
• If the backup was taken with TDE enabled (ANGARABASE_TDE_ENABLE=1), restore requires the same valid key material; without it, startup after restore is fail-closed.

Backup/Restore v2 — phase 1a: one-file backupset

Minimal offline pipeline producing a single *.angarabk file (manifest-first format).

Capabilities

Pinned commands (phase 1a)

# FULL (offline/local baseline)
angara-cli backup full \
 --config /path/to/angarabase.conf \
 --out /tmp/full_0001.angarabk \
 --db-id base

# FULL (remote execution via Admin API)
angara-cli backup full \
 --addr 127.0.0.1:9899 \
 --out /tmp/full_0001.angarabk \
 --db-id base

# INSPECT (manifest-only)
angara-cli backup inspect --file /tmp/full_0001.angarabk

# VERIFY v0 (struct/hash)
angara-cli backup verify --file /tmp/full_0001.angarabk --json

# RESTORE (local) into a new directory
angara-cli backup restore \
 --config /path/to/angarabase.conf \
 --file /tmp/full_0001.angarabk \
 --target-dir /tmp/restore_0001 \
 --overwrite

Evidence pack (N=3)

tools/backup_restore/evidence_v2_phase1a.sh --runs 3

Inspect vs verify

• backup inspect: manifest-only, no payload scan.
• backup verify: struct/hash integrity; proves backupset consistency, not SQL-level correctness.

Backup/Restore v2 — phase 1b: online FULL + LOG chain + PITR

Online pipeline with point-in-time recovery support.

Capabilities

Pinned commands (phase 1b)

# ONLINE FULL (best-effort, with backup fence)
# Local execution:
angara-cli backup full-online \
 --config /path/to/angarabase.conf \
 --out /tmp/full_online_0001.angarabk \
 --db-id base

# Remote execution (via Admin API):
angara-cli backup full-online \
 --addr 127.0.0.1:9899 \
 --out /tmp/full_online_0001.angarabk \
 --db-id base

# LOG chunk (LSN boundaries must be contiguous with FULL/prev LOG)
angara-cli backup log \
 --config /path/to/angarabase.conf \
 --out /tmp/log_0001.angarabk \
 --start-lsn <u64> --end-lsn <u64> \
 --db-id base

# Validate chain contiguity
angara-cli backup chain-validate \
 --chain /tmp/full_online_0001.angarabk,/tmp/log_0001.angarabk \
 --json

# PITR restore to target_lsn
angara-cli backup restore-chain \
 --config /path/to/angarabase.conf \
 --target-dir /tmp/restore_chain_0001 \
 --target-lsn <u64> \
 --chain /tmp/full_online_0001.angarabk,/tmp/log_0001.angarabk \
 --overwrite

Evidence pack (N=3)

tools/backup_restore/evidence_v2_phase1b.sh --runs 3

Expected result

After a successful restore the server should start cleanly and previously committed data should be present:

SELECT * FROM sys.health;
SELECT * FROM sys.identity;

For PITR restores, data reflects the state at target_lsn.

Troubleshooting

For unresolved issues see Known issues and Support.

Host migration (without dump/restore)

An alternative to backup/restore for moving AngaraBase to another host is direct data-file copying using the Instance Lease system.

When applicable

• Single-node deployment — one AngaraBase instance
• Shared storage — NFS, SAN, or manual file copy
• Same version — the same AngaraBase version on source and target
• Maintenance window — the instance can be stopped during copying

Step-by-step instructions

1. Stop the source instance:

# Graceful shutdown to release the lease
kill -TERM <angarabase-pid>

# Wait for completion
ps aux | grep angarabase-server

2. Verify lease release (optional):

-- On another instance or via backup connection
SELECT lease_holder_id FROM sys.identity;

3. Copy data files:

# Copy data directory
rsync -av /source/data/ /target/data/

# Copy transaction log directory
rsync -av /source/txlog/ /target/txlog/

# Verify integrity
find /target/data -name "*.adb" -exec ls -la {} \;

4. Start on the target host:

# Update configuration if needed
vim /target/angarabase.conf

# Start AngaraBase
angarabase-server --config /target/angarabase.conf

5. Verify migration:

-- Check lease holder
SELECT lease_holder_hostname, recovery_mode FROM sys.identity;

-- Verify data integrity
SELECT COUNT(*) FROM your_tables;

-- Check system state
SELECT * FROM sys.health;

What to check after startup

• lease_holder_hostname changed to the new host
• recovery_mode shows the recovery type
• All tables are accessible and contain the expected data
• No errors in server logs
• Client connections work correctly

Limitations

• Downtime required — requires stopping the service during copying
• File consistency — files must be copied atomically
• Version compatibility — AngaraBase versions must match
• Configuration — paths in the configuration may need updates

Comparison with backup/restore

See also

Detailed host migration instructions: Crash Recovery — Host Migration

Links

• Canonical operator runbook: angarabook/src/operations/backup-restore.md

• Configuration reference: Configuration

• Diagnostics (post-restore health check): Diagnostics

• Host migration details: Crash Recovery

Crash Recovery and Storage Portability

This guide covers crash recovery scenarios and how to safely restart AngaraBase instances on existing data files, including migration to different hosts.

Overview

AngaraBase includes an Instance Lease system that prevents dual-write corruption while enabling safe crash recovery and storage portability. The system works on any filesystem, including NFS and SAN where traditional file locking is unreliable.

What Happens During Crash Recovery

When AngaraBase starts on existing data files, it performs these recovery phases:

Phase A: WAL Recovery (file_bin backend only)

• Scans transaction log files for incomplete entries
• Truncates partial tail records to maintain consistency
• Replays committed page deltas (redo operations)

Phase B: MVCC History Restore

• Recovers in-memory MVCC state from transaction log
• Marks uncommitted transactions as aborted
• Restores visibility information for concurrent reads

Phase C: Instance Lease Check

• Checks for active instance lease in base.adb
• Prevents dual instance startup with fail-closed error
• Automatically takes over expired leases (crash recovery)

Restarting on Existing Files (Same Host)

For normal restart scenarios on the same machine:

1. Stop the server (if running):

# Graceful shutdown releases the lease automatically
kill -TERM <angarabase-pid>

2. Start normally:

angarabase-server --config angarabase.conf

3. Verify recovery:

SELECT recovery_mode, lease_holder_id FROM sys.identity;

The instance lease will be automatically acquired after the TTL expires (default: 30 seconds).

Host Migration (Without dump/restore)

To move data files to a different host:

Prerequisites

• Both hosts have the same AngaraBase version
• Same page size (checked automatically)
• Shared storage OR manual file copy

Step-by-Step Process

1. Stop the source instance:

# Graceful shutdown to release lease
kill -TERM <angarabase-pid>

# Verify shutdown completed
ps aux | grep angarabase-server

2. Verify lease is released:

# Check that no process holds the lease
# (Optional: use another AngaraBase instance to query sys.identity)

3. Copy data files (if not using shared storage):

# Copy entire data directory
rsync -av /old/host/data/ /new/host/data/

# Copy transaction log directory 
rsync -av /old/host/txlog/ /new/host/txlog/

4. Start on new host:

angarabase-server --config angarabase.conf

5. Verify migration:

SELECT lease_holder_hostname, recovery_mode FROM sys.identity;
SELECT COUNT(*) FROM your_tables; -- Verify data integrity

Force Lease Takeover

If the previous instance crashed and the lease hasn’t expired, use force takeover:

When to Use

• Previous instance confirmed dead (host crashed, process killed)
• Lease shows expired time but takeover not automatic
• Emergency recovery scenarios

How to Use

# Set environment variable before starting
export ANGARABASE_FORCE_LEASE_TAKEOVER=1
angarabase-server --config angarabase.conf

Safety Checks

Before forcing takeover, verify:

• Previous instance process is definitely terminated
• No other AngaraBase processes accessing the same files
• Network partitions resolved (if applicable)

Warning: Force takeover with a running instance will cause data corruption.

Diagnostics

Check Lease Status

SELECT 
 lease_holder_id,
 lease_holder_hostname,
 lease_expires_at,
 lease_acquired_at,
 recovery_mode
FROM sys.identity;

Check Recovery Metrics

SELECT * FROM sys.health;

Lease Configuration

Environment variables (set before startup):

• ANGARABASE_LEASE_TTL_S: Lease duration in seconds (default: 30)
• ANGARABASE_LEASE_HEARTBEAT_S: Heartbeat interval (default: 10)
• ANGARABASE_FORCE_LEASE_TAKEOVER: Force takeover flag (default: false)

Limitations

WAL Backend Requirements

• Full recovery: Requires transaction_log.backend = "file_bin"
• Partial recovery: noop backend has no WAL replay (data loss possible)

Filesystem Considerations

• Local filesystems: Full support (ext4, xfs, btrfs, etc.)
• NFS/SAN: Instance lease works; verify file copy consistency
• Network partitions: May cause false lease expiration

Version Compatibility

• Same major.minor version required for host migration
• Page size must match (checked automatically)
• Configuration compatibility recommended

Troubleshooting

“Cannot start: database files are owned by another instance”

• Cause: Active lease held by another instance
• Solution: Wait for lease expiration or use force takeover (if safe)

“MVCC recovery failed”

• Cause: Corrupted transaction log files
• Solution: Check disk space, filesystem errors; may need backup restore

“VERSION decode failed”

• Cause: Corrupted version marker or incompatible format
• Solution: Restore from backup; check filesystem integrity

Performance After Recovery

• First queries may be slower (cold buffer pool)
• MVCC state rebuilds incrementally
• Statistics may need refresh (ANALYZE TABLE)

See Also

• Instance Lifecycle - Conceptual overview
• Backup and Restore - Host migration alternatives
• Configuration - Lease settings reference

Binary release upgrade

This scenario covers upgrade/downgrade within the current release line without changing the on-disk format.

Invariants

• The upgrade changes binaries and unit/scripts, but does not touch user data.
• The data directory /var/lib/angarabase is not deleted.
• The config /etc/angarabase/angarabase.conf is preserved (package noreplace policy).

Upgrade via DEB/RPM

DEB

sudo dpkg -i angarabase-server_<VERSION>_amd64.deb
sudo systemctl status angarabase --no-pager

RPM

sudo rpm -Uvh angarabase-server-<VERSION>-1.x86_64.rpm
sudo systemctl status angarabase --no-pager

Upgrade from tarball

sudo systemctl stop angarabase
sudo install -m 0755 bin/angarabase-server /usr/bin/angarabase-server
sudo install -m 0755 bin/angara-cli /usr/bin/angara-cli
sudo systemctl daemon-reload
sudo systemctl start angarabase

Post-upgrade check

angarabase-server --version
sudo systemctl is-active angarabase

Smoke-check:

• the server starts without panic;
• config and data dir are accessible;
• the basic SQL health check passes.

Rollback

Rollback is performed by installing the previous package/archive.

Important: rollback is allowed only between compatible storage-contract versions.

Next

After a successful upgrade and post-upgrade checklist:

• Crash recovery — what to do if the new version does not start.
• Backup and restore — take a control backup immediately after the upgrade.
• Verify release artifacts — make sure the next version is downloaded and verified the same way.
• Operator deep-dive: Upgrade and migration — full playbook with rollback scenarios.

Monitoring

Goal

Connect Prometheus and Grafana to AngaraBase metrics and get a working dashboard for ops/triage monitoring.

Prerequisites

• Running angarabase-server (see Quickstart)
• Access to the AngaraBase metrics listener (ANGARABASE_METRICS_ADDR)
• Running Prometheus instance
• Running Grafana instance

Step 1: enable the metrics endpoint

Set the listener address in the config file:

[ops]
metrics_addr = "127.0.0.1:9898"

Or use the environment variable override (takes precedence over config):

export ANGARABASE_METRICS_ADDR=127.0.0.1:9898
angarabase-server --config /etc/angarabase/angarabase.conf

Verify the endpoint responds:

curl -sS http://127.0.0.1:9898/metrics | rg '^angarabase_' -m 5
curl -sS http://127.0.0.1:9898/health/live
curl -sS http://127.0.0.1:9898/health/ready
curl -sS http://127.0.0.1:9898/health/startup

See Configuration — Ops knobs for the full [ops] section reference.

OpenTelemetry tracing

AngaraBase also exposes opt-in OTel-style span export for query lifecycle triage.

Minimal setup example:

export ANGARABASE_OTEL_ENABLED=1
export ANGARABASE_OTEL_EXPORTER=file
export ANGARABASE_OTEL_ENDPOINT=artifacts/otel/spans.jsonl
export ANGARABASE_OTEL_SAMPLE_RATE_PPM=1000000

Spans are emitted for bounded stage names (accept, auth, session, parse, plan, execute, storage_io, commit/rollback) and are intended for diagnostics evidence, not for raw SQL capture.

Step 2: configure the Prometheus scrape

Add a job to prometheus.yml:

scrape_configs:
 - job_name: angarabase
 scrape_interval: 15s
 metrics_path: /metrics
 static_configs:
 - targets: ["127.0.0.1:9898"]

Restart Prometheus and verify in the UI (/targets) that the angarabase target is UP.

Step 3: add the Prometheus data source in Grafana

1. Open the Grafana UI.
2. Navigate to Connections → Data sources.
3. Add Prometheus.
4. Set the URL to your Prometheus instance (e.g., http://127.0.0.1:9090).
5. Click Save & test and confirm the connection succeeds.

Step 4: import the AngaraBase dashboard

1. Go to Dashboards → New → Import.
2. Upload the file tools/observability/grafana/angarabase-overview-v2.json.
3. Select the Prometheus data source added in the previous step.
4. Save the dashboard.

Alternative — download the JSON directly from the server:

curl -fsS http://127.0.0.1:9898/grafana/angarabase-overview.json -o angarabase-overview.json

Then import angarabase-overview.json in Grafana.

Expected result

After import, the dashboard should display the following panels:

If panels are empty, see Troubleshooting below.

Troubleshooting

No data in panels

• Check http://<prometheus>/targets: the angarabase target must be UP.
• Ensure ANGARABASE_METRICS_ADDR matches the targets value in Prometheus.
• Verify metrics use the angarabase_ prefix:

curl -sS http://127.0.0.1:9898/metrics | rg '^angarabase_' -m 20

Data source test fails in Grafana

• Verify the data source URL and network reachability from Grafana to Prometheus.
• For Docker/K8s use the service DNS/hostname (not 127.0.0.1) when Grafana and Prometheus run in different containers.

Readiness probe is not ready

• GET /health/ready returns a reason in JSON — this is the primary triage signal.
• Collect a diagnostics bundle and attach summary.json:

tools/diagnostics_bundle/run.sh \
 --root artifacts/diagnostics/<incident-id> \
 --config <path> \
 --metrics-url http://<metrics-host>/metrics

• For further investigation see Diagnostics and Support.

Links

• Dashboard JSON: tools/observability/grafana/angarabase-overview-v2.json
• Dashboard JSON from server: GET /grafana/angarabase-overview.json (on ANGARABASE_METRICS_ADDR / [ops].metrics_addr)
• Dashboard import guide: tools/observability/README.md
• Operator runbook (metrics + probes): Troubleshooting guide
• Metrics checklist (contract): Observability metrics checklist
• Configuration reference: Configuration
• Support flow: Support

Diagnostics

Goal

Analyze query performance, inspect server state, and triage issues using AngaraBase’s built-in diagnostic tools.

Prerequisites

• Running angarabase-server (see Quickstart)
• psql or another pgwire-compatible client
• For slow-query logging: env variables set before server start (see Configuration)

EXPLAIN variants

Basic query plan

EXPLAIN SELECT * FROM t WHERE id = 1;

Shows the planned execution path without running the query.

EXPLAIN ANALYZE (with actual timing)

EXPLAIN ANALYZE SELECT * FROM t WHERE id > 100;

Executes the query and reports actual row counts and timing alongside the plan.

For parallel join plans, EXPLAIN ANALYZE also reports best-effort join accounting counters:

• join_build_rows — rows processed by join build phase,
• join_probe_rows — rows processed by join probe phase.

EXPLAIN ANALYZE for DML (dry-run)

EXPLAIN ANALYZE over INSERT, UPDATE, or DELETE runs the statement inside an isolated dry-run transaction that rolls back automatically — no data is modified.

EXPLAIN ANALYZE INSERT INTO t (id, v) VALUES (999, 42);
EXPLAIN ANALYZE UPDATE t SET v = v + 1 WHERE id = 1;
EXPLAIN ANALYZE DELETE FROM t WHERE id = 1;

Buffer statistics

EXPLAIN (BUFFERS) SELECT * FROM t WHERE id = 1;

Adds buffer-pool hit/miss counters to the plan output.

JSON output

EXPLAIN (FORMAT JSON) SELECT * FROM t WHERE id = 1;

Returns the plan as a JSON document — useful for programmatic analysis.

Runtime diagnostic views

angara_stat_activity

Shows currently executing queries and their wait state.

SELECT pid, state, wait_event_type, query
FROM angara_stat_activity;

wait_event_type values: Lock, IO, Net, CPU — provides coarse wait categorization for triage.

angara_stat_statements

Aggregated per-query statistics (call count, total time, rows, etc.).

SELECT query, calls, total_time, rows
FROM angara_stat_statements
ORDER BY total_time DESC
LIMIT 10;

Reset accumulated stats:

SELECT angara_stat_statements_reset();

The maximum number of tracked statements is controlled by ANGARABASE_STAT_STATEMENTS_MAX (LRU-bounded). See Configuration — Diagnostics knobs.

angara_top_queries

Convenience wrapper returning the top-N queries by cumulative time:

SELECT * FROM angara_top_queries(10);

Slow-query log

Capture queries that exceed a duration threshold to the server log.

Configuration

Set before server start (env variables override config):

Example:

export ANGARABASE_LOG_MIN_DURATION_MS=500
export ANGARABASE_LOG_QUERY_TEXT=1
angarabase-server --config /path/to/angarabase.conf

Slow queries appear in the server log at logging.log_directory.

Saturation / backpressure (operability)

When p99/p99.9 degrades under load, you usually want to distinguish:

• lock/contention waits,
• IO/scheduler saturation,
• admission/queue rejects vs “random” timeouts.

Practical entry points:

• angara_stat_activity.wait_event_type (coarse wait classification)
• operator runbook: angarabook/src/operations/troubleshooting.md

System introspection views

sys.health

Overall server health status:

SELECT * FROM sys.health;

sys.settings

Effective configuration (config + env overrides resolved):

SELECT * FROM sys.settings;
SELECT * FROM sys.settings WHERE name LIKE 'storage.%';

sys.identity

Instance identity metadata:

SELECT * FROM sys.identity;

Expected result

• EXPLAIN variants return plan text (or JSON) without errors.
• angara_stat_activity shows at least the current session.
• angara_stat_statements accumulates entries after queries are executed.
• sys.health returns {"status":"ready"} on a healthy instance.

Troubleshooting

For unresolved issues see Known issues and Support.

Links

• SQL compatibility reference: SQL overview
• Monitoring setup: Monitoring
• Configuration (env knobs): Configuration
• Diagnostics bundle tool: tools/diagnostics_bundle/run.sh
• Operator runbook: angarabook/src/operations/troubleshooting.md
• Incident runbook (10 minutes): Error debug runbook
• Support flow: Support

Structured Logging

Status: ✅ Active Scope: Production operations, diagnostics, troubleshooting

Overview

AngaraBase uses structured logging for production observability and diagnostics. This replaces ad-hoc println! calls with consistent, level-based log messages using the Rust log crate and env_logger backend.

Key Benefits

• Consistent Format: All log messages follow structured key=value format
• Level Control: Runtime-configurable verbosity (error/warn/info/debug/trace)
• Production Ready: Suitable for log aggregation systems (ELK, Splunk, etc.)
• Performance: Low overhead when disabled, structured context when enabled

Log Levels

Level Policy

Production Recommendations

• Production: info (default) - captures operational events without performance impact
• Troubleshooting: debug - detailed diagnostics for performance analysis
• Development: trace - full verbosity for debugging

Configuration

Static Configuration (angarabase.conf)

[logging]
log_level = "info"

Supported values: error, warn, info, debug, trace Default: info Restart required: No (dynamic setting)

Environment Variable Override

export ANGARABASE_LOG_LEVEL=debug
./angarabase-server --config /etc/angarabase/angarabase.conf

Environment variables take precedence over config file settings during bootstrap.

Runtime Configuration

Change log level without restart using SQL:

-- Check current setting
SELECT * FROM sys.settings WHERE name = 'logging.log_level';

-- Change to debug level
SELECT sys.set_setting('logging.log_level', 'debug');

-- Verify change
SELECT * FROM sys.settings WHERE name = 'logging.log_level';

Note: Runtime changes are volatile and reset on restart. Update angarabase.conf for persistent changes.

Log Format

Structure

All log messages follow this format:

2026-03-09T10:30:45.123Z INFO [angarabase_server::stats] stats_scheduler: auto_analyze_triggered table=users staleness_score=2.45

Components:

• Timestamp: ISO 8601 with millisecond precision
• Level: ERROR/WARN/INFO/DEBUG/TRACE
• Module: Rust module path (e.g., angarabase_server::stats)
• Context: Structured key=value pairs with operation context

Context Format

Log messages use consistent key=value structured context:

#![allow(unused)]
fn main() {
// Good: structured context
log::info!("instance_lease: acquired holder={} expires_at={}", holder_id, expires_at);

// Good: operation context
log::debug!("micro_rescan: completed table={} scanned_rows={} duration={:?}", table, rows, duration);

// Good: error context 
log::warn!("audit_sink: write_failed path={} err={}", path, error);
}

Context Guidelines:

• Use snake_case for keys
• Include operation prefix (e.g., micro_rescan:, instance_lease:)
• Provide enough context for troubleshooting
• Avoid sensitive data in logs (use IDs, not content)

Production Deployment

Log Aggregation

AngaraBase structured logs integrate well with log aggregation systems:

ELK Stack:

# logstash.conf
filter {
 if [program] == "angarabase-server" {
 grok {
 match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:module}\] %{GREEDYDATA:context}" }
 }
 kv {
 source => "context"
 field_split => " "
 value_split => "="
 }
 }
}

Splunk:

[angarabase]
EXTRACT-level = (?<level>ERROR|WARN|INFO|DEBUG|TRACE)
EXTRACT-module = \[(?<module>[^\]]+)\]
EXTRACT-operation = (?<operation>\w+):

Log Rotation

Configure log rotation for production deployments:

# /etc/logrotate.d/angarabase
/var/log/angarabase/*.log {
 daily
 rotate 30
 compress
 delaycompress
 missingok
 notifempty
 create 0644 angarabase angarabase
 postrotate
 systemctl reload angarabase-server
 endscript
}

Monitoring

Key log patterns to monitor:

# Error rate monitoring
grep "ERROR" /var/log/angarabase/server.log | wc -l

# Instance lease issues
grep "instance_lease.*failed" /var/log/angarabase/server.log

# Performance degradation
grep "stats_scheduler.*io_backpressure" /var/log/angarabase/server.log

# MVCC recovery events
grep "mvcc_recovery:" /var/log/angarabase/server.log

Troubleshooting

Common Issues

High log volume at debug level:

-- Reduce to info level
SELECT sys.set_setting('logging.log_level', 'info');

Missing log output:

# Check RUST_LOG environment variable
echo $RUST_LOG

# Override with explicit filter
export RUST_LOG=angarabase_server=debug,rustls=warn

Log format parsing issues:

• Ensure timestamp parsing handles millisecond precision
• Context parsing should handle key=value pairs with spaces
• Module names contain :: separators

Performance Impact

Log level performance characteristics:

Recommendation: Use info for production, temporarily increase to debug for troubleshooting specific issues.

Migration Notes

From println! to log::*!

Current implementation migrated all production println! calls to structured logging:

• 60+ calls across 11 files replaced
• Backward compatibility: No breaking changes to existing functionality
• Enhanced context: More structured information for operations

Future Enhancements (Phase 2)

Planned for next releases:

• Tracing integration: Distributed tracing with spans
• OpenTelemetry: OTLP export for APM systems
• Metrics correlation: Link logs with performance metrics
• Sampling: Adaptive sampling for high-volume environments

See Also

• RFC-2026-360: Structured Logging & Tracing v0
• Instance Lifecycle - Instance lease logging
• Crash Recovery - MVCC recovery logging
• Settings Management - Runtime configuration

AngaraBase Tracing Operations Guide

Audience: Database operators, SREs, performance engineers

Overview

AngaraBase includes structured tracing based on the tracing crate with OpenTelemetry support. This system replaces legacy env_logger and provides end-to-end visibility for the query execution pipeline.

Key Benefits:

• End-to-end spans: accept → authenticate → parse → plan → execute → storage → response
• Each query has a unique trace_id
• JSON-structured logs for machine parsing
• Integration with Jaeger/Tempo through OpenTelemetry
• Automatic tracing-context propagation across async/sync boundaries

Configuration

Basic Setup

In angarabase.conf:

[diagnostics]
# Tracing output format: "text" (human-readable) or "json" (structured)
tracing_format = "json"

# Log level filtering (same as RUST_LOG)
log_level = "info"

Environment Variables

JSON vs Text Format

Text format (human-readable):

2026-03-12T10:30:45.123Z INFO angarabase::query::executor: query_start session_id=12345 sql="SELECT * FROM users"
2026-03-12T10:30:45.125Z DEBUG angarabase::query::planner: plan_created plan_hash=abc123 estimated_rows=1000

JSON format (machine-parseable):

{"timestamp":"2026-03-12T10:30:45.123Z","level":"INFO","target":"angarabase::query::executor","fields":{"session_id":12345,"sql":"SELECT * FROM users"},"span":{"name":"query_execution","trace_id":"abc123"}}

Tracing Architecture

Span Hierarchy

query_execution (root span)
├── parse (SQL → AST)
├── plan (AST → execution plan)
├── execute
│ ├── storage_io (page reads/writes)
│ ├── lock_acquisition
│ └── wal_flush
└── commit (autocommit finalization)

Span Propagation

AngaraBase automatically propagates tracing context through:

1. Async boundaries: tokio::task::spawn_blocking calls
2. Thread pool: Worker thread execution
3. Network layer: AngaraNet io_uring/tokio integration

Implementation pattern:

#![allow(unused)]
fn main() {
let span = tracing::Span::current();
tokio::task::spawn_blocking(move || {
 let _enter = span.enter();
 // Sync code here inherits tracing context
 engine.execute(&query)
}).await
}

Operational Procedures

Enabling Tracing

1. Development/Debug:

RUST_LOG=angarabase=trace ./angarabase-server

2. Production (JSON logs):

# angarabase.conf
[diagnostics]
tracing_format = "json"
log_level = "info"

3. OpenTelemetry Export:

export ANGARABASE_OTLP_ENDPOINT=http://jaeger:14268/api/traces
export ANGARABASE_TRACE_SAMPLE_RATE=0.05 # 5% sampling
./angarabase-server

Monitoring Query Performance

1. Slow Query Detection

Text logs:

grep "query_execution.*duration_ms" /var/log/angarabase.log | \
 awk '$NF > 1000' | head -10 # Queries > 1 second

JSON logs:

jq 'select(.span.name == "query_execution" and .fields.duration_ms > 1000)' \
 /var/log/angarabase.log

2. Per-Phase Timing Analysis

Look for spans with names: parse, plan, execute, commit

Example JSON query:

jq 'select(.span.name == "execute" and .fields.duration_ms > 500)' \
 /var/log/angarabase.log | \
 jq '.fields | {trace_id, duration_ms, estimated_rows}'

3. Lock Contention Detection

# Find queries waiting on locks
jq 'select(.fields.wait_event_type == "Lock")' /var/log/angarabase.log

Troubleshooting Common Issues

High Parse Time

# Find queries with slow parsing
jq 'select(.span.name == "parse" and .fields.duration_ms > 100)' \
 /var/log/angarabase.log | jq '.fields.sql'

Common causes:

• Complex SQL with many JOINs
• Large IN clauses
• Deeply nested subqueries

High Plan Time

# Find queries with slow planning
jq 'select(.span.name == "plan" and .fields.duration_ms > 200)' \
 /var/log/angarabase.log

Common causes:

• Missing statistics
• Complex join ordering
• Large number of tables

High Execute Time

# Correlate with storage I/O
jq 'select(.span.name == "storage_io" and .fields.duration_ms > 1000)' \
 /var/log/angarabase.log

Common causes:

• Sequential scans
• I/O bottlenecks
• Lock contention

Integration with External Tools

Jaeger Integration

1. Setup Jaeger:

docker run -d --name jaeger \
-p 14268:14268 -p 16686:16686 \
jaegertracing/all-in-one:latest

2. Configure AngaraBase:

export ANGARABASE_OTLP_ENDPOINT=http://localhost:14268/api/traces

3. View traces: http://localhost:16686

Grafana/Tempo Integration

# tempo.yaml
server:
 http_listen_port: 3200

distributor:
 receivers:
 otlp:
 protocols:
 http:
 endpoint: 0.0.0.0:4318

storage:
 trace:
 backend: local
 local:
 path: /tmp/tempo/traces

Log Aggregation (ELK Stack)

Logstash configuration:

input {
 file {
 path => "/var/log/angarabase.log"
 codec => "json"
 }
}

filter {
 if [span][name] {
 mutate {
 add_field => { "trace_operation" => "%{[span][name]}" }
 }
 }
}

output {
 elasticsearch {
 hosts => ["elasticsearch:9200"]
 index => "angarabase-traces-%{+YYYY.MM.dd}"
 }
}

Performance Impact

Overhead Measurements

Production Recommendations

1. Use JSON format for structured log processing
2. Set appropriate log levels: INFO for production, DEBUG for troubleshooting
3. Configure OTLP sampling: 1-10% for high-traffic systems
4. Monitor log volume: JSON logs are ~2x larger than text

Alerting and Monitoring

Key Metrics to Monitor

1. Query Duration Distribution:

histogram_quantile(0.95, 
rate(angarabase_query_duration_seconds_bucket[5m])
)

2. Slow Query Count:

increase(angarabase_slow_queries_total[5m])

3. Tracing Overhead:

rate(angarabase_tracing_events_total[5m])

Alerting Rules

groups:
- name: angarabase_tracing
 rules:
 - alert: HighQueryLatency
 expr: |
 histogram_quantile(0.95, 
 rate(angarabase_query_duration_seconds_bucket[5m])
 ) > 1.0
 for: 2m
 labels:
 severity: warning
 annotations:
 summary: "AngaraBase query latency is high"
 
 - alert: TracingVolumeHigh
 expr: |
 rate(angarabase_tracing_events_total[5m]) > 1000
 for: 5m
 labels:
 severity: info
 annotations:
 summary: "AngaraBase tracing volume is high"

Security Considerations

Sensitive Data in Logs

⚠️ WARNING: Tracing logs may contain SQL queries with sensitive data.

Mitigation strategies:

1. Query parameter redaction:

[diagnostics]
redact_query_params = true # Replace literals with ?

2. Log rotation and retention:

# logrotate configuration
/var/log/angarabase.log {
daily
rotate 7
compress
delaycompress
missingok
notifempty
}

3. Access control:

chmod 640 /var/log/angarabase.log
chown angarabase:angarabase-ops /var/log/angarabase.log

OTLP Export Security

# Use TLS for OTLP export
export ANGARABASE_OTLP_ENDPOINT=https://jaeger.internal:14268/api/traces
export ANGARABASE_OTLP_TLS_CERT=/etc/ssl/certs/angarabase.crt
export ANGARABASE_OTLP_TLS_KEY=/etc/ssl/private/angarabase.key

Troubleshooting

Common Issues

1. No Tracing Output

Symptoms: No tracing spans in logs Causes:

• RUST_LOG not set or too restrictive
• tracing_format misconfigured
• Tracing not enabled in binary

Solution:

# Check tracing is compiled in
./angarabase-server --version | grep tracing

# Enable debug tracing
RUST_LOG=angarabase=debug ./angarabase-server

2. Missing Span Context

Symptoms: Broken trace chains, missing parent-child relationships Causes:

• Missing span.enter() in spawn_blocking calls
• Incorrect async context propagation

Solution: Check for proper span propagation pattern in code

3. High Log Volume

Symptoms: Disk space issues, performance degradation Causes:

• Log level too verbose (TRACE in production)
• No log rotation
• High query volume

Solution:

# Reduce log level
export RUST_LOG=angarabase=info

# Enable log rotation
logrotate -f /etc/logrotate.d/angarabase

4. OTLP Export Failures

Symptoms: Traces not appearing in Jaeger/Tempo Causes:

• Network connectivity issues
• Incorrect endpoint configuration
• Authentication failures

Solution:

# Test OTLP endpoint
curl -v $ANGARABASE_OTLP_ENDPOINT/health

# Check AngaraBase logs for export errors
grep "otlp.*error" /var/log/angarabase.log

Advanced Topics

Custom Span Attributes

AngaraBase automatically adds these attributes to spans:

Correlation with System Metrics

CPU correlation:

# Find high-CPU queries
jq 'select(.span.name == "execute" and .fields.cpu_time_ms > 1000)' \
 /var/log/angarabase.log

I/O correlation:

# Find I/O-heavy queries 
jq 'select(.fields.io_reads > 1000 or .fields.io_writes > 100)' \
 /var/log/angarabase.log

Custom Dashboards

Grafana query examples:

1. Query throughput by operation:

sum(rate(angarabase_queries_total[5m])) by (operation)

2. Average query duration by phase:

avg(angarabase_query_phase_duration_seconds) by (phase)