ArcherDB Documentation

ArcherDB is a distributed geospatial database built for real-time location tracking at scale. It combines the consistency guarantees of Viewstamped Replication (VSR) with a high-performance S2-based spatial index, enabling sub-millisecond queries across billions of location events.

Security and continuity controls are infrastructure-managed by default: authn/authz, TLS/mTLS, encryption at rest, and backup orchestration are expected to be enforced outside of ArcherDB itself.

Quick Links

For Developers

Tutorials

Learn ArcherDB step-by-step:

• Quickstart - Insert and query your first location in 5 minutes
• Getting Started - Comprehensive setup, SDK installation, and usage patterns

How-To Guides

Goal-oriented guides:

• SDK Retry Semantics - Configure retry behavior and handle errors
• Error Codes - Understand and troubleshoot errors
• Testing Guide - Run tests locally for all 5 SDKs
• Benchmark Guide - Run and interpret performance benchmarks
• Release Checklist - Must-do go/no-go checklist before calling ArcherDB released
• Java Publish Checklist - Separate maintainer checklist for Maven Central publication

Reference

Complete API and configuration documentation:

• API Reference - All operations with request/response details
• OpenAPI Specification - Machine-readable API definition
• Tier Profiles - Runtime/capacity presets and release artifact guidance
• Hardware Requirements - Minimum and recommended specs
• LSM Tuning - Storage engine configuration
• Journal Sizing - Write-ahead log configuration

SDK Documentation

Comprehensive guides for each language:

• SDK Overview - Choosing an SDK, feature matrix, common patterns
• SDK Comparison Matrix - Feature parity and code examples
• SDK Limitations - Known issues and workarounds
• Parity Matrix - Cross-SDK verification status
• SDK Comprehensive Test Report - Current SDK validation evidence

For Operators

Deployment

• Operations Runbook - Day-to-day operational procedures
• Capacity Planning - Sizing clusters for your workload
• Multi-Region Deployment - Design/reference guide; server runtime is not GA

Recovery & Continuity

• Backup Operations - External snapshot/backup integration guidance
• Disaster Recovery - Replica recovery and external restore procedures
• Upgrade Guide - Rolling upgrades and external rollback planning

Troubleshooting

• Troubleshooting Guide - Diagnose and resolve common issues
• Error Codes - Error reference with troubleshooting guidance

Alert Runbooks

Per-alert response guides linked from Prometheus alerts:

• Replica Down - When a replica is unreachable
• View Changes - Frequent leader elections
• Index Degraded - Index performance issues
• High Read Latency - Read latency triage
• High Write Latency - Write latency triage
• Disk Capacity - Disk capacity and fill-rate response
• Compaction Backlog - Storage compaction backlog response
• Kernel Crash Durability - Crash-durability harness guidance

Performance

• Benchmarks - Benchmark framework and results
• Benchmark Guide - Running and interpreting benchmarks
• Profiling - Performance profiling workflows
• Performance Tuning - Query and workload optimization
• LSM Tuning - Storage engine optimization

Testing & CI

• Testing Guide - Run all tests locally
• CI Tiers - Smoke, PR, nightly, weekly tiers
• curl Examples - Raw HTTP examples for all operations
• Protocol Reference - Wire format and data types

Understanding ArcherDB

Architecture and design:

• Architecture - System design, data flow, and component interactions
• VSR Understanding - How Viewstamped Replication provides consensus
• Durability Verification - How ArcherDB ensures data durability

Security

• Security Best Practices - Infrastructure security and trust-boundary model
• Encryption Guide - External encryption-at-rest controls
• Encryption Security - Data-protection model and non-goals

Internals

For contributors:

• Message Bus Errors - Network layer error handling
• Upstream Backports - Local patches carried from upstream dependencies

Release Notes

• CHANGELOG.md - Release history and notable changes
• FINALIZATION_PLAN.md - Release hardening and exit criteria

Documentation Coverage

All documentation requirements (DOCS-01 through DOCS-08) are complete:

See .planning/REQUIREMENTS.md for full traceability.

Quickstart

Time to complete: ~5 minutes

Get from zero to your first spatial query in 5 minutes. For comprehensive setup, see the Getting Started Guide.

Step 1: Download Binary (~1 min)

curl -L https://github.com/ArcherDB-io/archerdb/releases/latest/download/archerdb-linux-x86_64.tar.gz | tar xz
sudo mv archerdb /usr/local/bin/

curl -L https://github.com/ArcherDB-io/archerdb/releases/latest/download/archerdb-macos-aarch64.tar.gz | tar xz
sudo mv archerdb /usr/local/bin/

curl -L https://github.com/ArcherDB-io/archerdb/releases/latest/download/archerdb-macos-x86_64.tar.gz | tar xz
sudo mv archerdb /usr/local/bin/

Step 2: Start Server (~1 min)

# Format data file and start server
archerdb format --cluster=0 --replica=0 --replica-count=1 data.archerdb
archerdb start --addresses=3000 data.archerdb

You should see: info: server ready on 127.0.0.1:3000

Open a new terminal for the next steps.

Step 3: Install SDK (~1 min)

pip install archerdb

npm install archerdb-node

go get github.com/ArcherDB-io/archerdb/src/clients/go

./zig/zig build clients:java -Drelease
(cd src/clients/java && mvn --batch-mode --quiet install)

Step 4: Insert a Location (~1 min)

Insert a delivery vehicle location in San Francisco:

import archerdb

client = archerdb.GeoClientSync(archerdb.GeoClientConfig(
    cluster_id=0,
    addresses=['127.0.0.1:3000']
))

# Insert vehicle at SF city center (37.7749, -122.4194)
event = archerdb.create_geo_event(
    entity_id=archerdb.id(),
    latitude=37.7749,
    longitude=-122.4194,
    group_id=1,
)
client.insert_events([event])
print(f"Inserted vehicle: {event.entity_id}")

const { createGeoClient, createGeoEvent, id } = require('archerdb-node')

const client = createGeoClient({
  cluster_id: 0n,
  addresses: ['127.0.0.1:3000'],
})

// Insert vehicle at SF city center (37.7749, -122.4194)
const event = createGeoEvent({
  entity_id: id(),
  latitude: 37.7749,
  longitude: -122.4194,
  group_id: 1n,
})

await client.insertEvents([event])
console.log(`Inserted vehicle: ${event.entity_id}`)
client.destroy()

package main

import (
    "fmt"
    "log"
    archerdb "github.com/ArcherDB-io/archerdb/src/clients/go"
    "github.com/ArcherDB-io/archerdb/src/clients/go/pkg/types"
)

func main() {
    client, _ := archerdb.NewGeoClient(archerdb.GeoClientConfig{
        ClusterID: types.ToUint128(0),
        Addresses: []string{"127.0.0.1:3000"},
    })
    defer client.Close()

    // Insert vehicle at SF city center (37.7749, -122.4194)
    event, _ := types.NewGeoEvent(types.GeoEventOptions{
        EntityID:  types.ID(),
        Latitude:  37.7749,
        Longitude: -122.4194,
        GroupID:   1,
    })
    client.InsertEvents([]types.GeoEvent{event})
    fmt.Printf("Inserted vehicle: %s\n", event.EntityID)
}

import com.archerdb.geo.*;

public class Quickstart {
    public static void main(String[] args) throws Exception {
        try (GeoClient client = GeoClient.create(0L, "127.0.0.1:3000")) {
            // Insert vehicle at SF city center (37.7749, -122.4194)
            UInt128 entityId = UInt128.random();
            GeoEvent event = new GeoEvent.Builder()
                .setEntityId(entityId)
                .setLatitude(37.7749)
                .setLongitude(-122.4194)
                .setGroupId(1L)
                .build();

            GeoEventBatch batch = client.createBatch();
            batch.add(event);
            batch.commit();
            System.out.println("Inserted vehicle: " + entityId);
        }
    }
}

# Insert vehicle at SF city center (37.7749, -122.4194)
curl -X POST http://127.0.0.1:3000/insert \
  -H "Content-Type: application/json" \
  -d '{
    "events": [{
      "entity_id": "00000000-0000-0000-0000-000000000001",
      "lat_nano": 37774900000,
      "lon_nano": -122419400000,
      "group_id": 1
    }]
  }'

Step 5: Query by Radius (~1 min)

Find all vehicles within 1km of SF city center:

# Find vehicles within 1km of SF city center
results = client.query_radius(
    center_lat=37.7749,
    center_lon=-122.4194,
    radius_m=1000,
)
print(f"Found {len(results.events)} vehicles within 1km")

// Find vehicles within 1km of SF city center
const results = await client.queryRadius({
  latitude: 37.7749,
  longitude: -122.4194,
  radius_m: 1000,
})
console.log(`Found ${results.events.length} vehicles within 1km`)

// Find vehicles within 1km of SF city center
filter, _ := types.NewRadiusQuery(37.7749, -122.4194, 1000, 100)
results, _ := client.QueryRadius(filter)
fmt.Printf("Found %d vehicles within 1km\n", len(results.Events))

// Find vehicles within 1km of SF city center
QueryResult results = client.queryRadius(
    QueryRadiusFilter.create(37.7749, -122.4194, 1000, 100)
);
System.out.println("Found " + results.getEvents().size() + " vehicles within 1km");

# Find vehicles within 1km of SF city center
curl -X POST http://127.0.0.1:3000/query/radius \
  -H "Content-Type: application/json" \
  -d '{
    "center_lat_nano": 37774900000,
    "center_lon_nano": -122419400000,
    "radius_m": 1000,
    "limit": 100
  }'

Congratulations! You just completed your first spatial query.

You inserted a location and found it with a radius query - the core of real-time location tracking.

Next Steps

• Getting Started Guide - Batching, polygon queries, error handling
• API Reference - Complete operation documentation
• Operations Runbook - Production deployment

Getting Started with ArcherDB

Time to First Query

Total time: ~10 minutes

This guide gets you to your first spatial query quickly, then shows you the full capabilities.

Prerequisites

• Operating System: Linux (kernel >= 5.6), macOS, or Windows
• For SDKs: Python 3.9+, Node.js 18+, Go 1.21+, or Java 11+

Security Boundary Note

ArcherDB expects security controls at the infrastructure boundary:

• Authentication/authorization at your API or gateway layer
• TLS/mTLS in gateway/service mesh or private network transport
• Encryption at rest and key management in storage/cloud platform
• Backup: ArcherDB has built-in upload to S3, GCS, and Azure Blob (see Backup Operations); platform snapshots remain a defense-in-depth option

Installation (~2 min)

Download the pre-built binary for your platform:

curl -L https://github.com/ArcherDB-io/archerdb/releases/latest/download/archerdb-linux-x86_64.tar.gz | tar xz
sudo mv archerdb /usr/local/bin/
archerdb --version

curl -L https://github.com/ArcherDB-io/archerdb/releases/latest/download/archerdb-macos-aarch64.tar.gz | tar xz
sudo mv archerdb /usr/local/bin/
archerdb --version

curl -L https://github.com/ArcherDB-io/archerdb/releases/latest/download/archerdb-macos-x86_64.tar.gz | tar xz
sudo mv archerdb /usr/local/bin/
archerdb --version

git clone https://github.com/ArcherDB-io/archerdb.git
cd archerdb
./zig/download.sh
./zig/zig build
# Binary at ./zig-out/bin/archerdb

Choose a Tier

ArcherDB tier presets are open-source runtime and capacity profiles:

• lite (recommended for demos/evaluation): fastest first-run experience, low footprint, intentionally storage-limited.
• standard: baseline production profile.
• pro: higher-performance mainstream profile.
• enterprise: high-end production profile.
• ultra: top-end profile.

If building from source, choose a tier explicitly:

./zig/zig build -Dconfig=lite
# or: standard, pro, enterprise, ultra

Starting a Cluster (~1 min)

Single-Node Development

# Format data file (cluster=0 for dev)
archerdb format --cluster=0 --replica=0 --replica-count=1 data.archerdb

# Start server on port 3000
archerdb start --addresses=3000 data.archerdb

You should see: info: server ready on 127.0.0.1:3000

Production Three-Node Cluster

For fault tolerance (survives 1 node failure):

# Node 1 (192.168.1.1)
archerdb format --cluster=12345 --replica=0 --replica-count=3 /data/archerdb.db
archerdb start --addresses=192.168.1.1:3000,192.168.1.2:3000,192.168.1.3:3000 /data/archerdb.db

# Node 2 (192.168.1.2)
archerdb format --cluster=12345 --replica=1 --replica-count=3 /data/archerdb.db
archerdb start --addresses=192.168.1.1:3000,192.168.1.2:3000,192.168.1.3:3000 /data/archerdb.db

# Node 3 (192.168.1.3)
archerdb format --cluster=12345 --replica=2 --replica-count=3 /data/archerdb.db
archerdb start --addresses=192.168.1.1:3000,192.168.1.2:3000,192.168.1.3:3000 /data/archerdb.db

SDK Installation (~1 min)

pip install archerdb

npm install archerdb-node

go get github.com/ArcherDB-io/archerdb/src/clients/go

./zig/zig build clients:java -Drelease
(cd src/clients/java && mvn --batch-mode --quiet install)

<dependency>
    <groupId>com.archerdb</groupId>
    <artifactId>archerdb-java</artifactId>
    <version>0.1.0-SNAPSHOT</version>
</dependency>

implementation 'com.archerdb:archerdb-java:0.1.0-SNAPSHOT'

Backup Configuration (Optional)

ArcherDB can stream every closed LSM block to S3, GCS, or Azure Blob storage as the cluster runs. Append the relevant flags to archerdb start:

# S3 / S3-compatible (MinIO, R2, Backblaze, LocalStack)
archerdb start \
  --backup-enabled \
  --backup-provider=s3 \
  --backup-region=us-east-1 \
  --backup-bucket=my-archerdb-backups \
  data.archerdb

# Azure Blob (SharedKey)
archerdb start \
  --backup-enabled \
  --backup-provider=azure \
  --backup-bucket=my-archerdb-container \
  --backup-access-key-id=<account-name> \
  --backup-secret-access-key=<base64-account-key> \
  data.archerdb

S3 credentials are picked up from AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY when not provided on the command line. GCS uses the same flags as S3 with --backup-provider=gcs and an HMAC key issued via the Cloud Storage Interoperability console. Restore via archerdb restore from any of the supported providers; see Backup Operations and Disaster Recovery.

Hello World: Vehicle Tracking (~3 min)

This example demonstrates the core value of ArcherDB: track vehicles and find nearby pickups.

Scenario: A delivery vehicle needs to find pickup locations within 1km.

Step 1: Create Client

import archerdb

client = archerdb.GeoClientSync(archerdb.GeoClientConfig(
    cluster_id=0,
    addresses=['127.0.0.1:3000']
))

const { createGeoClient, createGeoEvent, id } = require('archerdb-node')

const client = createGeoClient({
  cluster_id: 0n,
  addresses: ['127.0.0.1:3000'],
})

import (
    archerdb "github.com/ArcherDB-io/archerdb/src/clients/go"
    "github.com/ArcherDB-io/archerdb/src/clients/go/pkg/types"
)

client, err := archerdb.NewGeoClient(archerdb.GeoClientConfig{
    ClusterID: types.ToUint128(0),
    Addresses: []string{"127.0.0.1:3000"},
})
if err != nil {
    log.Fatal(err)
}
defer client.Close()

import com.archerdb.geo.*;

GeoClient client = GeoClient.create(0L, "127.0.0.1:3000");

# No client setup needed - use curl directly
BASE_URL="http://127.0.0.1:3000"

Step 2: Insert Delivery Vehicle

Insert the vehicle at SF city center (37.7749, -122.4194):

vehicle_id = archerdb.id()
vehicle = archerdb.create_geo_event(
    entity_id=vehicle_id,
    latitude=37.7749,      # SF city center
    longitude=-122.4194,
    group_id=1,            # Fleet ID
)
client.insert_events([vehicle])
print(f"Vehicle {vehicle_id} inserted")

const vehicleId = id()
const vehicle = createGeoEvent({
  entity_id: vehicleId,
  latitude: 37.7749,      // SF city center
  longitude: -122.4194,
  group_id: 1n,           // Fleet ID
})
await client.insertEvents([vehicle])
console.log(`Vehicle ${vehicleId} inserted`)

vehicleID := types.ID()
vehicle, _ := types.NewGeoEvent(types.GeoEventOptions{
    EntityID:  vehicleID,
    Latitude:  37.7749,      // SF city center
    Longitude: -122.4194,
    GroupID:   1,            // Fleet ID
})
client.InsertEvents([]types.GeoEvent{vehicle})
fmt.Printf("Vehicle %s inserted\n", vehicleID)

UInt128 vehicleId = UInt128.random();
GeoEvent vehicle = new GeoEvent.Builder()
    .setEntityId(vehicleId)
    .setLatitude(37.7749)      // SF city center
    .setLongitude(-122.4194)
    .setGroupId(1L)            // Fleet ID
    .build();

GeoEventBatch batch = client.createBatch();
batch.add(vehicle);
batch.commit();
System.out.println("Vehicle " + vehicleId + " inserted");

curl -X POST $BASE_URL/insert \
  -H "Content-Type: application/json" \
  -d '{
    "events": [{
      "entity_id": "00000000-0000-0000-0000-000000000001",
      "lat_nano": 37774900000,
      "lon_nano": -122419400000,
      "group_id": 1
    }]
  }'

Step 3: Insert Nearby Pickup Locations

Add two pickup locations near the vehicle:

# Pickup 1: 200m east of vehicle
pickup1 = archerdb.create_geo_event(
    entity_id=archerdb.id(),
    latitude=37.7751,
    longitude=-122.4180,
    group_id=2,  # Pickups group
)

# Pickup 2: 150m north of vehicle
pickup2 = archerdb.create_geo_event(
    entity_id=archerdb.id(),
    latitude=37.7760,
    longitude=-122.4200,
    group_id=2,
)

client.insert_events([pickup1, pickup2])
print("2 pickup locations inserted")

// Pickup 1: 200m east of vehicle
const pickup1 = createGeoEvent({
  entity_id: id(),
  latitude: 37.7751,
  longitude: -122.4180,
  group_id: 2n,  // Pickups group
})

// Pickup 2: 150m north of vehicle
const pickup2 = createGeoEvent({
  entity_id: id(),
  latitude: 37.7760,
  longitude: -122.4200,
  group_id: 2n,
})

await client.insertEvents([pickup1, pickup2])
console.log('2 pickup locations inserted')

// Pickup 1: 200m east of vehicle
pickup1, _ := types.NewGeoEvent(types.GeoEventOptions{
    EntityID:  types.ID(),
    Latitude:  37.7751,
    Longitude: -122.4180,
    GroupID:   2,  // Pickups group
})

// Pickup 2: 150m north of vehicle
pickup2, _ := types.NewGeoEvent(types.GeoEventOptions{
    EntityID:  types.ID(),
    Latitude:  37.7760,
    Longitude: -122.4200,
    GroupID:   2,
})

client.InsertEvents([]types.GeoEvent{pickup1, pickup2})
fmt.Println("2 pickup locations inserted")

// Pickup 1: 200m east of vehicle
GeoEvent pickup1 = new GeoEvent.Builder()
    .setEntityId(UInt128.random())
    .setLatitude(37.7751)
    .setLongitude(-122.4180)
    .setGroupId(2L)  // Pickups group
    .build();

// Pickup 2: 150m north of vehicle
GeoEvent pickup2 = new GeoEvent.Builder()
    .setEntityId(UInt128.random())
    .setLatitude(37.7760)
    .setLongitude(-122.4200)
    .setGroupId(2L)
    .build();

GeoEventBatch batch = client.createBatch();
batch.add(pickup1);
batch.add(pickup2);
batch.commit();
System.out.println("2 pickup locations inserted");

curl -X POST $BASE_URL/insert \
  -H "Content-Type: application/json" \
  -d '{
    "events": [
      {
        "entity_id": "00000000-0000-0000-0000-000000000002",
        "lat_nano": 37775100000,
        "lon_nano": -122418000000,
        "group_id": 2
      },
      {
        "entity_id": "00000000-0000-0000-0000-000000000003",
        "lat_nano": 37776000000,
        "lon_nano": -122420000000,
        "group_id": 2
      }
    ]
  }'

Step 4: Find Pickups Within 1km of Vehicle

Query for pickups near the vehicle’s location:

# Find all entities within 1km of vehicle
results = client.query_radius(
    center_lat=37.7749,
    center_lon=-122.4194,
    radius_m=1000,  # 1 kilometer
    limit=100,
)

print(f"Found {len(results.events)} entities within 1km of vehicle:")
for event in results.events:
    print(f"  Entity {event.entity_id} at ({event.latitude:.4f}, {event.longitude:.4f})")

Found 3 entities within 1km of vehicle:
  Entity 1234... at (37.7749, -122.4194)  # Vehicle
  Entity 5678... at (37.7751, -122.4180)  # Pickup 1
  Entity 9012... at (37.7760, -122.4200)  # Pickup 2

// Find all entities within 1km of vehicle
const results = await client.queryRadius({
  latitude: 37.7749,
  longitude: -122.4194,
  radius_m: 1000,  // 1 kilometer
  limit: 100,
})

console.log(`Found ${results.events.length} entities within 1km of vehicle:`)
for (const event of results.events) {
  console.log(`  Entity ${event.entity_id}`)
}

// Find all entities within 1km of vehicle
filter, _ := types.NewRadiusQuery(37.7749, -122.4194, 1000, 100)
results, _ := client.QueryRadius(filter)

fmt.Printf("Found %d entities within 1km of vehicle:\n", len(results.Events))
for _, event := range results.Events {
    fmt.Printf("  Entity %s at (%.4f, %.4f)\n",
        event.EntityID, event.Latitude(), event.Longitude())
}

// Find all entities within 1km of vehicle
QueryResult results = client.queryRadius(
    QueryRadiusFilter.create(37.7749, -122.4194, 1000, 100)
);

System.out.println("Found " + results.getEvents().size() + " entities within 1km of vehicle:");
for (GeoEvent event : results.getEvents()) {
    System.out.println("  Entity " + event.getEntityId());
}

curl -X POST $BASE_URL/query/radius \
  -H "Content-Type: application/json" \
  -d '{
    "center_lat_nano": 37774900000,
    "center_lon_nano": -122419400000,
    "radius_m": 1000,
    "limit": 100
  }'

Congratulations! You’ve completed the core ArcherDB workflow: insert locations and find nearby entities.

Additional Operations

Polygon Queries (Geofencing)

Find entities within a geographic boundary:

# Define a polygon (counter-clockwise winding)
polygon = [
    (37.78, -122.42),  # NW corner
    (37.78, -122.40),  # NE corner
    (37.76, -122.40),  # SE corner
    (37.76, -122.42),  # SW corner
]

results = client.query_polygon(vertices=polygon, limit=1000)
print(f"Found {len(results.events)} entities in polygon")

const polygon = [
  { lat: 37.78, lon: -122.42 },
  { lat: 37.78, lon: -122.40 },
  { lat: 37.76, lon: -122.40 },
  { lat: 37.76, lon: -122.42 },
]

const results = await client.queryPolygon({ vertices: polygon, limit: 1000 })
console.log(`Found ${results.events.length} entities in polygon`)

vertices := [][]float64{
    {37.78, -122.42},
    {37.78, -122.40},
    {37.76, -122.40},
    {37.76, -122.42},
}

filter, _ := types.NewPolygonQuery(vertices, 1000)
results, _ := client.QueryPolygon(filter)
fmt.Printf("Found %d entities in polygon\n", len(results.Events))

QueryPolygonFilter filter = new QueryPolygonFilter.Builder()
    .addVertex(37.78, -122.42)
    .addVertex(37.78, -122.40)
    .addVertex(37.76, -122.40)
    .addVertex(37.76, -122.42)
    .setLimit(1000)
    .build();

QueryResult results = client.queryPolygon(filter);
System.out.println("Found " + results.getEvents().size() + " entities in polygon");

curl -X POST $BASE_URL/query/polygon \
  -H "Content-Type: application/json" \
  -d '{
    "vertices": [
      {"lat_nano": 37780000000, "lon_nano": -122420000000},
      {"lat_nano": 37780000000, "lon_nano": -122400000000},
      {"lat_nano": 37760000000, "lon_nano": -122400000000},
      {"lat_nano": 37760000000, "lon_nano": -122420000000}
    ],
    "limit": 1000
  }'

Polygon with Holes

Exclude regions (parks, lakes, restricted zones):

# Outer boundary (counter-clockwise)
boundary = [
    (37.79, -122.42),
    (37.79, -122.39),
    (37.76, -122.39),
    (37.76, -122.42),
]

# Hole to exclude (clockwise winding)
park_hole = [
    (37.78, -122.41),
    (37.775, -122.41),
    (37.775, -122.40),
    (37.78, -122.40),
]

results = client.query_polygon(
    vertices=boundary,
    holes=[park_hole],  # Up to 100 holes
    limit=1000,
)

const boundary = [
  { lat: 37.79, lon: -122.42 },
  { lat: 37.79, lon: -122.39 },
  { lat: 37.76, lon: -122.39 },
  { lat: 37.76, lon: -122.42 },
]

const parkHole = [
  { lat: 37.78, lon: -122.41 },
  { lat: 37.775, lon: -122.41 },
  { lat: 37.775, lon: -122.40 },
  { lat: 37.78, lon: -122.40 },
]

const results = await client.queryPolygon({
  vertices: boundary,
  holes: [parkHole],
  limit: 1000,
})

boundary := [][]float64{
    {37.79, -122.42},
    {37.79, -122.39},
    {37.76, -122.39},
    {37.76, -122.42},
}

parkHole := [][]float64{
    {37.78, -122.41},
    {37.775, -122.41},
    {37.775, -122.40},
    {37.78, -122.40},
}

filter, _ := types.NewPolygonQuery(boundary, 1000, parkHole)
results, _ := client.QueryPolygon(filter)

QueryPolygonFilter filter = new QueryPolygonFilter.Builder()
    .addVertex(37.79, -122.42)
    .addVertex(37.79, -122.39)
    .addVertex(37.76, -122.39)
    .addVertex(37.76, -122.42)
    .startHole()
    .addHoleVertex(37.78, -122.41)
    .addHoleVertex(37.775, -122.41)
    .addHoleVertex(37.775, -122.40)
    .addHoleVertex(37.78, -122.40)
    .finishHole()
    .setLimit(1000)
    .build();

curl -X POST $BASE_URL/query/polygon \
  -H "Content-Type: application/json" \
  -d '{
    "vertices": [
      {"lat_nano": 37790000000, "lon_nano": -122420000000},
      {"lat_nano": 37790000000, "lon_nano": -122390000000},
      {"lat_nano": 37760000000, "lon_nano": -122390000000},
      {"lat_nano": 37760000000, "lon_nano": -122420000000}
    ],
    "holes": [[
      {"lat_nano": 37780000000, "lon_nano": -122410000000},
      {"lat_nano": 37775000000, "lon_nano": -122410000000},
      {"lat_nano": 37775000000, "lon_nano": -122400000000},
      {"lat_nano": 37780000000, "lon_nano": -122400000000}
    ]],
    "limit": 1000
  }'

Polygon constraints:

• Outer boundary: counter-clockwise winding
• Holes: clockwise winding
• Maximum 100 holes per polygon
• Holes must be fully contained within boundary

Get Latest Position

event = client.get_latest(entity_id)
if event:
    print(f"Last seen at: ({event.latitude:.4f}, {event.longitude:.4f})")

# Batch lookup
events = client.get_latest_batch([entity_id1, entity_id2, entity_id3])

const event = await client.getLatest(entityId)
if (event) {
  console.log(`Last seen at: (${event.latitude}, ${event.longitude})`)
}

// Batch lookup
const events = await client.getLatestBatch([entityId1, entityId2, entityId3])

event, err := client.GetLatest(entityID)
if err == nil && event != nil {
    fmt.Printf("Last seen at: (%.4f, %.4f)\n", event.Latitude(), event.Longitude())
}

// Batch lookup
events, _ := client.GetLatestBatch([]types.UInt128{entityID1, entityID2})

Optional<GeoEvent> event = client.getLatest(entityId);
event.ifPresent(e ->
    System.out.println("Last seen at: (" + e.getLatitude() + ", " + e.getLongitude() + ")")
);

// Batch lookup
List<GeoEvent> events = client.getLatestBatch(List.of(entityId1, entityId2));

curl -X POST $BASE_URL/query/latest \
  -H "Content-Type: application/json" \
  -d '{"entity_ids": ["00000000-0000-0000-0000-000000000001"]}'

Delete Entities (GDPR Compliance)

result = client.delete_entities([entity_id1, entity_id2])
print(f"Deleted: {result.deleted_count}")

const result = await client.deleteEntities([entityId1, entityId2])
console.log(`Deleted: ${result.deleted_count}`)

result, _ := client.DeleteEntities([]types.UInt128{entityID1, entityID2})
fmt.Printf("Deleted: %d\n", result.DeletedCount)

DeleteResult result = client.deleteEntities(List.of(entityId1, entityId2));
System.out.println("Deleted: " + result.getDeletedCount());

curl -X POST $BASE_URL/delete \
  -H "Content-Type: application/json" \
  -d '{"entity_ids": ["00000000-0000-0000-0000-000000000001"]}'

Error Handling

All SDKs provide typed errors:

from archerdb import (
    ArcherDBError,
    InvalidCoordinates,
    ClusterUnavailable,
    OperationTimeout,
)

try:
    client.insert_events(events)
except InvalidCoordinates as e:
    print(f"Bad coordinates: {e}")
except ClusterUnavailable:
    print("Cluster down, retry later")
except OperationTimeout:
    print("Timeout - operation may have committed")

import {
  InvalidCoordinates,
  ClusterUnavailable,
  OperationTimeout,
} from 'archerdb-node'

try {
  await client.insertEvents(events)
} catch (error) {
  if (error instanceof InvalidCoordinates) {
    console.error('Bad coordinates:', error.message)
  } else if (error instanceof ClusterUnavailable) {
    console.error('Cluster down, retry later')
  } else if (error instanceof OperationTimeout) {
    console.error('Timeout - operation may have committed')
  }
}

import "github.com/ArcherDB-io/archerdb/src/clients/go/pkg/errors"

_, err := client.InsertEvents(events)
if err != nil {
    switch {
    case errors.IsInvalidCoordinates(err):
        log.Printf("Bad coordinates: %v", err)
    case errors.IsClusterUnavailable(err):
        log.Println("Cluster down, retry later")
    case errors.IsOperationTimeout(err):
        log.Println("Timeout - operation may have committed")
    }
}

import com.archerdb.geo.exceptions.*;

try {
    client.insertEvents(events);
} catch (InvalidCoordinatesException e) {
    System.err.println("Bad coordinates: " + e.getMessage());
} catch (ClusterUnavailableException e) {
    System.err.println("Cluster down, retry later");
} catch (OperationTimeoutException e) {
    System.err.println("Timeout - operation may have committed");
}

# HTTP status codes:
# 200 - Success
# 400 - Invalid request (bad coordinates, malformed JSON)
# 503 - Cluster unavailable
# 504 - Request timeout

Configuration Reference

Client Options

Retry Options

See SDK Retry Semantics for detailed retry behavior.

What’s Next (~1 min)

SDK Documentation

• Python SDK
• Node.js SDK
• Go SDK
• Java SDK

SDK Retry Semantics

This document describes the retry behavior of ArcherDB client SDKs, including multi-batch retry patterns, idempotency guarantees, and configuration options.

Overview

ArcherDB SDKs implement automatic retry with exponential backoff for transient failures. Retries are enabled by default and preserve idempotency guarantees, making it safe to retry operations without risk of duplicate execution.

Retry Configuration

All SDKs support the following configuration parameters:

Node.js

import { createGeoClient, RetryConfig } from 'archerdb-node'

const client = createGeoClient({
  cluster_id: 0n,
  addresses: ['127.0.0.1:3000'],
  retry: {
    enabled: true,
    max_retries: 5,
    base_backoff_ms: 100,
    max_backoff_ms: 1600,
    total_timeout_ms: 30000,
    jitter: true,
  }
})

Python

import archerdb

client = archerdb.GeoClientSync(archerdb.GeoClientConfig(
    cluster_id=0,
    addresses=['127.0.0.1:3000'],
    retry=archerdb.RetryConfig(
        enabled=True,
        max_retries=5,
        base_backoff_ms=100,
        max_backoff_ms=1600,
        total_timeout_ms=30000,
        jitter=True,
    )
))

Go

import (
    "github.com/ArcherDB-io/archerdb/src/clients/go/pkg/retry"
)

config := retry.DefaultConfig()
config.MaxRetries = 5
config.BaseBackoffMs = 100
config.MaxBackoffMs = 1600
config.TotalTimeoutMs = 30000
config.Jitter = true

Backoff Schedule

The SDK uses exponential backoff with the following schedule:

Jitter Formula: actual_delay = base_delay + random(0, base_delay / 2)

Jitter prevents the “thundering herd” problem where many clients retry simultaneously after a transient failure.

Error Classification

Retryable Errors (Transient)

These errors are automatically retried:

• timeout - Operation timed out
• view_change_in_progress - Leader election in progress
• not_primary - Connected to non-primary replica
• cluster_unavailable - No quorum available
• session_expired - Session needs re-registration
• Network errors (connection reset, refused, timeout)

Non-Retryable Errors (Permanent)

These errors fail immediately without retry:

• invalid_coordinates - Coordinates out of valid range
• polygon_too_complex - Too many vertices
• batch_too_large - Batch exceeds maximum size
• query_result_too_large - Query limit exceeded
• invalid_entity_id - Zero or malformed entity ID
• Secure-transport boundary errors (gateway/proxy certificate failures)

Multi-Batch Retry Pattern

When a large batch operation times out, the SDK cannot determine which events succeeded vs. failed. The split_batch helper enables safe retry by dividing the batch into smaller chunks.

The Problem

[Event 0] [Event 1] [Event 2] ... [Event 9999]
                         ^
                         | Timeout occurs here
                         | Events 0-2000 committed
                         | Events 2001-9999 unknown

After a timeout, you don’t know which events committed. Simply retrying the entire batch risks duplicate processing (though server-side idempotency may prevent this for some operations).

The Solution: split_batch

Split the batch into smaller chunks and retry each chunk individually:

Original:  [0..9999] → TIMEOUT

Split:     [0..999] → SUCCESS (or already committed)
           [1000..1999] → SUCCESS
           [2000..2999] → SUCCESS
           ...

Node.js

import { splitBatch, OperationTimeout } from 'archerdb-node'

const events = generateLargeEventList()

try {
  const batch = client.createBatch()
  for (const event of events) {
    batch.add(event)
  }
  await batch.commit()
} catch (error) {
  if (error instanceof OperationTimeout) {
    // Split into smaller chunks and retry
    const chunks = splitBatch(events, 500)

    for (const chunk of chunks) {
      const retryBatch = client.createBatch()
      for (const event of chunk) {
        retryBatch.add(event)
      }
      try {
        await retryBatch.commit()
      } catch (retryError) {
        if (retryError instanceof OperationTimeout) {
          // Retry with even smaller chunks
          const smallerChunks = splitBatch(chunk, 100)
          // ... continue recursively
        }
      }
    }
  }
}

Python

from archerdb import split_batch, OperationTimeout

events = generate_large_event_list()

try:
    batch = client.create_batch()
    for event in events:
        batch.add(event)
    batch.commit()
except OperationTimeout:
    # Split into smaller chunks and retry
    chunks = split_batch(events, 500)

    for chunk in chunks:
        retry_batch = client.create_batch()
        for event in chunk:
            retry_batch.add(event)
        try:
            retry_batch.commit()
        except OperationTimeout:
            # Retry with even smaller chunks
            smaller_chunks = split_batch(chunk, 100)
            # ... continue recursively

Go

import "github.com/ArcherDB-io/archerdb/src/clients/go/pkg/types"

events := generateLargeEventList()

// Original batch failed - split and retry
chunks := types.SplitGeoEventBatch(events, 500)

for _, chunk := range chunks {
    _, err := client.InsertEvents(chunk)
    if errors.Is(err, errors.ErrTimeout) {
        // Retry with smaller chunks
        smallerChunks := types.SplitGeoEventBatch(chunk, 100)
        // ... continue recursively
    }
}

Idempotency Guarantees

Server-Side Deduplication

The server maintains client sessions and deduplicates requests based on client_id + request_number. When retrying:

1. SDK uses the same request_number for all retry attempts
2. If the request already executed, server returns the cached response
3. No double-execution occurs for idempotent operations

Idempotent vs Non-Idempotent Operations

Recommendation: Prefer upsert_events over insert_events for safer retry behavior.

Session Recovery

If the client crashes between sending a request and receiving a response:

• New client instance generates a new client_id
• Cannot rely on deduplication from the old session
• Application may need to handle potential duplicates

Retry Exhaustion

When all retry attempts are exhausted, the SDK returns a RetryExhausted error with:

• Number of attempts made
• The last error from the final attempt

Node.js

import { RetryExhausted } from 'archerdb-node'

try {
  await batch.commit()
} catch (error) {
  if (error instanceof RetryExhausted) {
    console.log(`Failed after ${error.attempts} attempts`)
    console.log(`Last error: ${error.lastError.message}`)
  }
}

Python

from archerdb import RetryExhausted

try:
    batch.commit()
except RetryExhausted as e:
    print(f"Failed after {e.attempts} attempts")
    print(f"Last error: {e.last_error}")

Go

import "github.com/ArcherDB-io/archerdb/src/clients/go/pkg/retry"

events := generateLargeEventList()

err := retry.Do(func() error {
    _, err := client.InsertEvents(events)
    return err
}, config)

var exhausted retry.ErrRetryExhausted
if errors.As(err, &exhausted) {
    fmt.Printf("Failed after %d attempts\n", exhausted.Attempts)
    fmt.Printf("Last error: %v\n", exhausted.LastError)
}

Best Practices

1. Use upsert over insert - Upsert operations are idempotent and safe to retry.

2. Keep batches reasonably sized - Smaller batches (500-1000 events) have better retry characteristics than maximum-sized batches.

3. Handle RetryExhausted - Always catch retry exhaustion and implement application-level fallback.

4. Use split_batch for large imports - When importing large datasets, proactively split into chunks rather than waiting for timeouts.

5. Monitor retry metrics - Track retry counts in production to detect cluster issues early.

6. Don’t disable retry without reason - The default retry configuration handles most transient failures automatically.

Related Documentation

• Source: src/error_codes.zig

ArcherDB Error Codes Reference

This document provides a complete reference for all ArcherDB error codes.

Error Code Ranges

Retry Semantics

Errors are classified into three categories:

• Retryable: Transient errors that may succeed on retry (e.g., leader election, network issues)
• Client Error: Invalid request that will always fail (fix the request, don’t retry)
• Fatal: Server-side bugs (open an issue with logs and reproduction details)

Distributed Error Codes

Multi-Region Errors (213-218)

These errors occur in multi-region deployments with async replication.

Usage Notes:

• Code 213: Writes must go to the primary region. SDKs automatically route writes to primary.
• Code 214: The follower hasn’t caught up with replication. Wait and retry, or use a fresher replica.
• Code 215: The primary region is down. Wait for failover or recovery.
• Code 216: Cross-region replication is slow. Retry with backoff.
• Code 217: Concurrent writes to the same entity detected in active-active replication. Application needs conflict resolution.
• Code 218: Entity’s geo-shard doesn’t match the region handling the request. Check shard routing configuration.

Sharding Errors (220-224)

These errors occur in sharded cluster deployments.

Usage Notes:

• Code 220: SDKs automatically refresh topology and retry. No application action needed.
• Code 221: Wait for shard recovery. The cluster may be experiencing failures.
• Code 222: Wait for resharding to complete. Operations will succeed after.
• Code 223: The requested shard count is not valid (e.g., must be power of 2).
• Code 224: A resharding operation failed. Check cluster health.

Security Boundary Errors (410-414, reserved/legacy)

These codes are reserved for deployments that layer external security controls around ArcherDB.

Usage Notes:

• Code 410: Check external key management service availability and IAM/policy bindings.
• Code 411: Validate storage snapshot integrity and external decryption path.
• Code 412: Verify infrastructure policy requires encrypted storage/transport for this route.
• Code 413: Retry after external key rotation completes.
• Code 414: Align external tooling format/version with deployment standards.

SDK Error Handling

Python

from archerdb import (
    MultiRegionError,
    ShardingError,
    MultiRegionException,
    ShardingException,
    is_retryable,
)

try:
    result = client.query_radius(lat, lon, radius)
except ShardingException as e:
    if e.error == ShardingError.RESHARDING_IN_PROGRESS:
        # Wait and retry - cluster is resharding
        time.sleep(5)
        result = client.query_radius(lat, lon, radius)
    elif is_retryable(e.code):
        # Generic retry logic
        result = retry_with_backoff(lambda: client.query_radius(lat, lon, radius))
    else:
        raise  # Non-retryable error

Java

import com.archerdb.geo.ShardingError;
import com.archerdb.geo.ArcherDBException;

try {
    QueryResult result = client.queryRadius(lat, lon, radius);
} catch (ArcherDBException e) {
    ShardingError shardError = ShardingError.fromCode(e.getErrorCode());
    if (shardError != null && shardError.isRetryable()) {
        // Retry with backoff
    }
}

Go

import "github.com/ArcherDB-io/archerdb/src/clients/go/pkg/errors"

result, err := client.QueryRadius(lat, lon, radius)
if err != nil {
    if archerErr, ok := err.(*errors.ArcherDBError); ok {
        if errors.IsShardingError(int(archerErr.Code)) {
            if errors.IsRetryable(int(archerErr.Code)) {
                // Retry with backoff
            }
        }
    }
}

Node.js/TypeScript

import {
    ShardingError,
    ShardingException,
    isShardingError,
    isRetryable,
} from 'archerdb';

try {
    const result = await client.queryRadius(lat, lon, radius);
} catch (e) {
    if (e instanceof ShardingException) {
        if (e.error === ShardingError.RESHARDING_IN_PROGRESS) {
            // Wait and retry
            await sleep(5000);
            result = await client.queryRadius(lat, lon, radius);
        }
    }
}

Troubleshooting Guide

Multi-Region Issues

Sharding Issues

Security Boundary Issues (410-414)

Related Documentation

• SDK Retry Semantics - Detailed retry configuration
• Disaster Recovery - Recovery procedures
• Operations Runbook - Operational procedures

ArcherDB Testing Guide

Comprehensive guide for running ArcherDB tests locally across all 5 SDKs.

Overview

ArcherDB’s test suite covers:

• Unit tests: Per-SDK operation validation
• Integration tests: Multi-node cluster behavior
• Parity tests: Cross-SDK result consistency
• Edge case tests: Geographic boundary conditions
• Performance tests: Latency and throughput benchmarks

Prerequisites

Required Software

Installation

# Python test infrastructure
pip install -r test_infrastructure/requirements.txt

# Node.js SDK dependencies
cd src/clients/node && npm install

# Go SDK dependencies
cd src/clients/go && go mod download

# Java SDK dependencies
cd src/clients/java && mvn dependency:resolve

# C SDK - no external dependencies (header-only)
# Zig - bundled in repo at ./zig/zig (for server build/tests)

Quick Start

1. Build the Server

# Constrained build (recommended for most machines)
./zig/zig build -j4 -Dconfig=lite

# Full build (CI or dedicated machine)
./zig/zig build

2. Start a Local Server

# Single node for development
./zig/zig build run -- --port 3001

# Or run the pre-built binary
./zig-out/bin/archerdb --port 3001

3. Run SDK Tests

Each SDK has its own test suite. Run from the repository root:

Python:

cd src/clients/python
pip install pytest
pytest tests/ -v

Node.js:

cd src/clients/node
npm install
npm test

Go:

cd src/clients/go
go test ./... -v

Java:

cd src/clients/java
mvn test

C:

cd src/clients/c
make test

Server (Zig unit tests):

./zig/zig build -j4 -Dconfig=lite test:unit

4. Run Specific Test Filter

Most test frameworks support filtering:

# Python
pytest tests/ -v -k "insert"

# Go
go test ./... -v -run TestInsert

# Server (Zig)
./zig/zig build -j4 -Dconfig=lite test:unit -- --test-filter "insert"

Test Infrastructure

The test_infrastructure/ directory provides Python utilities for cluster management and test data generation.

Cluster Harness

Start and manage multi-node ArcherDB clusters programmatically:

from test_infrastructure.harness import ArcherDBCluster, ClusterConfig

# Start a 3-node cluster
config = ClusterConfig(node_count=3)
with ArcherDBCluster(config) as cluster:
    cluster.wait_for_ready(timeout=60)
    leader_addr = cluster.get_leader_address()
    # Run tests against leader_addr...

Data Generators

Generate test datasets with various distribution patterns:

from test_infrastructure.generators import generate_events, DatasetConfig

# Generate 1000 events concentrated around cities
events = generate_events(DatasetConfig(
    size=1000,
    pattern='city_concentrated',
    cities=['san_francisco', 'tokyo'],
    seed=42,  # Reproducible
))

See test_infrastructure/README.md for complete documentation.

Fixtures

Pre-defined test fixtures are in test_infrastructure/fixtures/v1/:

Environment Variables

Running Integration Tests

Integration tests require a running cluster and are gated by environment variable:

# Start cluster first
./zig/zig build run -- --port 3001

# Enable integration tests
export ARCHERDB_INTEGRATION=1
pytest tests/ -v -m integration

Parity Testing

Verify that all SDKs produce identical results:

# Run full parity suite
python tests/parity_tests/parity_runner.py

# Run specific operations
python tests/parity_tests/parity_runner.py --ops insert query-radius

# Run specific SDKs
python tests/parity_tests/parity_runner.py --sdks python node go

# Verbose output
python tests/parity_tests/parity_runner.py -v

Results are written to:

• reports/parity.json - Machine-readable
• docs/PARITY.md - Human-readable matrix

See docs/PARITY.md for methodology and current status.

Edge Case Testing

Geographic edge cases (poles, antimeridian, equator) are tested separately:

# Run edge case tests
pytest tests/edge_case_tests/ -v

# Run specific category
pytest tests/edge_case_tests/ -v -k "polar"
pytest tests/edge_case_tests/ -v -k "antimeridian"

Troubleshooting

Server Won’t Start

1. Check if binary exists:

   ls zig-out/bin/archerdb

2. Build if missing:

   ./zig/zig build -j4 -Dconfig=lite

3. Check for port conflicts:

   lsof -i :3001

Tests Fail with Connection Errors

1. Verify server is running:

   curl http://127.0.0.1:3001/ping
   # Should return: {"pong":true}

2. Check environment variables:

   echo $ARCHERDB_HOST $ARCHERDB_PORT

Python Import Errors

Ensure test infrastructure is in path:

export PYTHONPATH="${PYTHONPATH}:${PWD}/test_infrastructure"

Or install in development mode:

pip install -e test_infrastructure/

Out of Memory During Tests

Use constrained build configuration:

# Instead of full build
./zig/zig build -j4 -Dconfig=lite test:unit

# Or minimal for low-memory systems
./zig/zig build -j2 -Dconfig=lite test:unit

Preserving Test Data for Debugging

export PRESERVE_ON_FAILURE=1
pytest tests/
# Data preserved in /tmp/archerdb-test-*

Resource-Constrained Testing

For machines with limited resources (24GB RAM, 8 cores):

Use the helper script:

./scripts/test-constrained.sh unit              # Default: -j4, lite
./scripts/test-constrained.sh --minimal unit    # Minimal: -j2, lite
./scripts/test-constrained.sh --full unit       # Full resources
./scripts/test-constrained.sh check             # Quick compile check

CI Integration

Tests are run automatically in CI with tiered execution:

• Smoke (<5 min): Every push, basic connectivity
• PR (<15 min): Pull requests, full SDK suite
• Nightly (2h): Manual-dispatch comprehensive multi-node testing
• Weekly (3h): Manual-dispatch benchmark publication

See docs/testing/ci-tiers.md for tier details.

See Also

• CI Tier Structure - CI pipeline organization
• Performance Baselines - Regression thresholds and benchmark baselines
• Benchmark Guide - Performance testing
• SDK Comparison Matrix - SDK feature parity
• SDK Limitations - Known issues and workarounds
• Parity Matrix - Cross-SDK verification status

Last updated: 2026-02-01

CI Tier Structure

ArcherDB uses a tiered CI approach to balance fast feedback with comprehensive testing.

Overview

Tier 1: Smoke Tests

Trigger: Every push to main, every commit in PRs

Duration: <5 minutes

Scope:

• Build verification (compiles cleanly)
• Basic connectivity test per SDK
• Single operation per SDK (insert + query)
• Single-node topology only

Purpose:

• Immediate feedback on breakage
• Gate for PR merges
• Catch obvious regressions fast

Failure handling:

• Blocks PR merge
• Notifies author immediately
• Must fix before proceeding

Jobs:

All SDK jobs run in parallel using GitHub Actions matrix strategy.

Tier 2: PR Tests

Trigger: Pull request opened, synchronized, or reopened

Duration: <15 minutes

Scope:

• Full SDK test suite for all 5 SDKs
• Single-node topology
• All 14 operations tested
• Error handling verification
• Retry logic validation

Purpose:

• Comprehensive validation before merge
• Catch edge cases smoke tests miss
• Verify parity across SDKs

Failure handling:

• Blocks PR merge
• Detailed test report in PR comment
• Must fix all failures before merge

Jobs:

Tier 3: Nightly Tests

Trigger: Manual workflow_dispatch

Duration: ~2 hours

Scope:

• All Tier 2 tests plus:
• Multi-node topologies (1, 3, 5, 6 nodes)
• Geographic edge cases (poles, antimeridian, equator)
• Failure injection tests
• Recovery verification
• Long-running stability tests

Purpose:

• Catch topology-specific issues
• Verify multi-node consistency
• Test failure recovery paths
• Find rare race conditions

Failure handling:

• Does NOT block merges
• Used for release-candidate and deep validation runs
• Failures are triaged manually from workflow output and artifacts

Jobs:

Tier 4: Weekly Benchmarks

Trigger: Manual workflow_dispatch

Duration: ~3 hours

Scope:

• Full benchmark suite across topologies
• Throughput benchmarks (events/sec)
• Read latency (P50, P95, P99)
• Write latency (P50, P95, P99)
• Mixed workload benchmarks
• SDK parity benchmarks

Purpose:

• Detect performance regressions
• Track performance trends
• Compare SDK performance
• Validate scaling behavior

Failure handling:

• Used when maintainers want to refresh published benchmark history
• Results are reviewed manually before promotion into long-term history
• Does not block future merges

Performance Targets:

Regression Detection:

• Uses Welch’s t-test for statistical significance
• Compares against stored baseline (JSON)
• 95% confidence level (alpha=0.05)
• Requires consistent CV <10% before comparison

Jobs:

Local benchmark outputs live under reports/benchmarks/, reports/history/, and reports/baselines/. Published history can be promoted into benchmarks/history/YYYY-MM-DD.json by the manual benchmark publication workflow.

Hardware

Artifacts

All tiers upload artifacts for debugging:

Running Locally

Simulate each tier locally:

Smoke:

./scripts/test-constrained.sh check
pytest tests/ -v -m smoke --timeout=60

PR:

./scripts/test-constrained.sh unit
pytest tests/ -v --timeout=300

Nightly (requires cluster setup):

export ARCHERDB_INTEGRATION=1
python -m test_infrastructure.harness.cli start --nodes=3
pytest tests/ -v -m "integration or nightly"
python -m test_infrastructure.harness.cli stop

Weekly (requires cluster setup):

python3 test_infrastructure/benchmarks/cli.py run --full-suite

Workflow Files

CI workflows are defined in .github/workflows/:

Monitoring

CI health dashboard: Track test stability, flakiness, and duration trends.

Key metrics:

• Pass rate by tier (target: >99% for smoke/PR)
• Mean duration by tier
• Flaky test count (target: 0)
• Regression frequency (weekly benchmark)

See Also

• Testing Guide - Local test running
• Benchmark Guide - Performance testing details
• Parity Matrix - SDK verification status

Last updated: 2026-02-01

Performance Baseline Management

This document describes how ArcherDB manages performance baselines for regression detection in CI.

Overview

Performance baselines are locked reference points that CI uses to detect regressions. Each PR’s benchmark results are compared against the current main branch baseline. Regressions block merge to prevent shipping slow code.

Regression Thresholds

The following thresholds are used to detect performance regressions:

Throughput: If current mean execution time is >5% slower than baseline, the check fails.

• Formula: current_mean > baseline_mean * 1.05
• Example: Baseline 1000ns, current 1060ns = 6% slower = FAIL

Latency P99: If current P99 latency is >25% higher than baseline, the check fails.

• Formula: current_p99 > baseline_p99 * 1.25
• Example: Baseline P99 1200ns, current P99 1400ns = 16.7% higher = PASS
• Example: Baseline P99 1200ns, current P99 1600ns = 33.3% higher = FAIL

Baseline Lifecycle

main branch push
       |
       v
+------------------+
| Run benchmarks   |
| (full mode)      |
+------------------+
       |
       v
+------------------+
| Upload as        |
| benchmark-       |
| baseline         |
+------------------+
       |
       v
  (90 day retention)

PR created/updated
       |
       v
+------------------+
| Download         |
| baseline from    |
| main             |
+------------------+
       |
       v
+------------------+
| Run benchmarks   |
| (quick mode)     |
+------------------+
       |
       v
+------------------+
| Compare against  |
| baseline         |
+------------------+
       |
    +--+--+
    |     |
 PASS   FAIL
    |     |
    v     v
 Merge  Block
 OK     merge

Timeline

1. Main branch pushes upload new baseline artifact (full benchmark mode)
2. PRs download the current main baseline and run quick benchmarks
3. Comparison checks both throughput and P99 latency against thresholds
4. Regressions block merge until fixed or baseline is reset

Resetting the Baseline

Sometimes you need to reset the baseline after intentional performance changes:

When to Reset

• Intentional trade-off: You accepted slower writes for better consistency
• New feature overhead: Added necessary functionality that increases latency
• Algorithm change: Changed from O(n) to O(log n) with different constants

How to Reset

1. Delete the current baseline artifact:

   • Go to GitHub Actions > Select a recent main workflow run
   • Find “benchmark-baseline” artifact and delete it
   • Or use GitHub CLI: gh api -X DELETE /repos/{owner}/{repo}/actions/artifacts/{artifact_id}

2. Merge your PR (no comparison runs without baseline)

3. New baseline created on next main push

Alternative: Update Baseline Manually

If you don’t want to delete the artifact:

1. Merge to main (workflow runs)
2. New baseline automatically uploaded
3. Future PRs compare against new baseline

Troubleshooting

False Positives

Symptom: Benchmark fails but code hasn’t changed performance.

Possible causes:

• Stale baseline: If baseline is very old, machine differences may cause variance
• CI runner variance: GitHub Actions runners can have different performance
• Background load: Other jobs running on same machine

Resolution:

• Re-run the benchmark job
• If consistent, reset the baseline

Consistent Failures

Symptom: Multiple re-runs show same regression.

This likely indicates a real regression:

1. Review recent commits for performance-impacting changes
2. Profile locally to identify hot paths
3. Fix the performance issue
4. If intentional, reset the baseline and document why

No Baseline Available

Symptom: “Download baseline” step shows “Artifact not found”.

This is normal for:

• First run after repository setup
• After baseline was manually deleted
• Baseline artifact expired (90 day retention)

Resolution: Merge to main to create new baseline.

jq/bc Not Available

Symptom: “jq not installed” warning in comparison output.

Resolution: The workflow installs these dependencies. If you’re running locally, install them:

# Ubuntu/Debian
sudo apt-get install jq bc

# macOS
brew install jq bc

Configuration

Modifying Thresholds

Thresholds are defined in scripts/benchmark-ci.sh:

# Throughput: 5% threshold
throughput_threshold=$(echo "scale=0; $baseline_mean * 1.05 / 1" | bc)

# Latency P99: 25% threshold
latency_threshold=$(echo "scale=0; $baseline_p99 * 1.25 / 1" | bc)

To change thresholds:

1. Edit the multipliers in benchmark-ci.sh
2. Update this documentation
3. Update workflow header comments

Benchmark Modes

Related Files

• .github/workflows/benchmark.yml - CI workflow
• scripts/benchmark-ci.sh - Benchmark runner and comparison logic
• .planning/phases/09-testing-infrastructure/09-CONTEXT.md - Threshold decisions

References

• .planning/phases/09-testing-infrastructure/09-CONTEXT.md - Why these thresholds were chosen
• .planning/STATE.md - Observed 5% CV in benchmarks

ArcherDB Benchmark Guide

Guide for running, interpreting, and tracking ArcherDB performance benchmarks.

Overview

ArcherDB benchmarks measure:

• Throughput: Events processed per second
• Read latency: Query response time percentiles
• Write latency: Insert response time percentiles
• Mixed workload: Combined read/write performance
• Scaling: Performance across topologies (1/3/5/6 nodes)

Performance Targets

The repository uses the following comparison gates on comparable hardware profiles:

Running Benchmarks Locally

Prerequisites

# Install benchmark dependencies
pip install -r test_infrastructure/requirements.txt

# Build ArcherDB (lite config for testing)
./zig/zig build -j4 -Dconfig=lite

Quick Run (Single Topology)

# Run all benchmark types on 3-node cluster
python3 test_infrastructure/benchmarks/cli.py run --topology 3

# With time limit
python3 test_infrastructure/benchmarks/cli.py run --topology 3 --time-limit 60

# With operation count limit
python3 test_infrastructure/benchmarks/cli.py run --topology 3 --op-count 10000

Full Suite (All Topologies)

# Run complete benchmark suite (1/3/5/6 node topologies)
python3 test_infrastructure/benchmarks/cli.py run --full-suite

# Exclude mixed workload tests (faster)
python3 test_infrastructure/benchmarks/cli.py run --full-suite --no-mixed

Mixed Workload Benchmarks

Control the read/write ratio:

# 80% reads, 20% writes (default)
python3 test_infrastructure/benchmarks/cli.py run --topology 3 --read-write-ratio 0.8

# 50% reads, 50% writes
python3 test_infrastructure/benchmarks/cli.py run --topology 3 --read-write-ratio 0.5

# Write-heavy (20% reads, 80% writes)
python3 test_infrastructure/benchmarks/cli.py run --topology 3 --read-write-ratio 0.2

# Read-only
python3 test_infrastructure/benchmarks/cli.py run --topology 3 --read-write-ratio 1.0

Compare to Baseline

# Compare current run to stored baseline
python3 test_infrastructure/benchmarks/cli.py compare baseline.json current.json

Interpreting Results

Throughput

Events processed per second. Higher is better.

Throughput: <measured events/sec>
Compare against: the latest checked-in baseline for the same hardware/profile
Status: PASS when the run meets or exceeds the local comparison gate

Key factors:

• Batch size (larger batches = higher throughput)
• Network latency (lower = higher throughput)
• Node count (more nodes = higher total throughput, but overhead)

Latency Percentiles

Query response times. Lower is better.

Read Latency:
  P50: 0.3ms  (median)
  P95: 0.8ms  (95% of requests)
  P99: 4.2ms  (99% of requests)

Target: P95 <1ms, P99 <10ms
Status: PASS

Percentile meanings:

• P50 (median): Typical user experience
• P95: 95% of requests are this fast or faster
• P99: Captures tail latency, important for SLAs

Confidence Intervals

All means are reported with 95% confidence intervals: