Benchmark Framework

This document describes ArcherDB’s benchmark evidence model and where benchmark artifacts live.

Current Evidence

The latest measured single-node comparison (ArcherDB vs Valkey vs PostGIS on identical hardware, with methodology and reproduction steps) is benchmarks/single-node-2026-08.md. Raw run outputs: reports/benchmarks/evidence-20260803/.

Two methodology rules learned from that exercise are now binding:

1. Insert throughput requires sustained load. One full-size message carries ~81,916 events, so runs below a few hundred thousand events measure a handful of batches (startup-dominated), not throughput. The CI quick lane now inserts 500K events for exactly this reason.
2. Cross-store comparisons must state durability. ArcherDB replies after fsync + consensus; Valkey’s default replies from memory. Always publish the volatile ceiling and the durability-matched number (appendfsync always) alongside ArcherDB results.

Benchmark Layers

ArcherDB currently has three benchmark layers in the repository:

1. The Python benchmark harness under test_infrastructure/benchmarks/
2. GitHub Actions workflows for baseline comparison and published history under .github/workflows/benchmark.yml and .github/workflows/benchmark-weekly.yml
3. The correctness-gated external comparison suite under scripts/competitor-benchmarks/v2/

Local Outputs

Local benchmark runs write to:

• reports/benchmarks/ for detailed run outputs
• reports/history/ for local history
• reports/baselines/ for saved baselines

These are the authoritative local paths for development and release-candidate evidence.

Published History

The manual benchmark publication workflow stores promoted historical artifacts in:

• benchmarks/history/

Use that location for checked-in benchmark snapshots and long-term graph publication. Do not assume it is populated unless the publication workflow has been run.

The validated August 2026 ArcherDB Lite, PostgreSQL + PostGIS, and Valkey comparisons (two profiles: bulk 80,000-event batches and SDK-default 1,000-event batches, both regenerated after the client-SDK fixes; the August 2 run remains in history as the pre-fix baseline) are published at:

• benchmarks/history/2026-08-04-comparison-v2.json
• benchmarks/history/2026-08-03-comparison-v2.json
• benchmarks/comparison-v2/report.md
• benchmarks/comparison-v2/analysis.ipynb

Its public-claim gate passes only for equal-semantics full-history ingestion: every system durably retained and geospatially indexed all events while maintaining a separate latest-position index. Read/query results remain context-only where the five-trial coefficient of variation exceeds 10%.

CLI Surface

The benchmark CLI is the supported entry point for local execution:

# Single topology
python3 test_infrastructure/benchmarks/cli.py run --topology 3 --time-limit 60 --op-count 10000

# Full suite
python3 test_infrastructure/benchmarks/cli.py run --full-suite

# Full suite without mixed workload
python3 test_infrastructure/benchmarks/cli.py run --full-suite --no-mixed

The active harness now drives ArcherDB through the supported SDK/client surface. It does not send raw HTTP requests at replica/message-bus ports.

For larger local topologies, the harness also uses a machine-fit cluster profile so 5-node and 6-node runs do not reserve the same per-node memory budget as 1-node and 3-node runs on a shared development machine.

For the 6-node local topology, the harness now also:

• formats the cluster as 5 voters plus 1 standby
• passes --replica-count through both format and start
• staggers startup until each replica completes local init
• orders SDK endpoints as leader-first, then the remaining voters

Current checked-in quick artifacts from the April 9, 2026 evidence refresh live under:

• reports/benchmarks/release-20260409-sdkquick/20260409-055134-1node.json
• reports/benchmarks/release-20260409-sdkquick/20260409-055217-3node.json
• reports/benchmarks/release-20260409-sdkquick/20260409-063550-5node.json
• reports/benchmarks/release-20260409-sdkquick/20260409-082107-6node.json

Performance Targets

The repository currently uses these comparison gates on comparable hardware profiles:

Release Rule

Performance claims in release docs and announcements should only cite benchmark artifacts produced by the real benchmark harness. Synthetic proxies and stale historical summaries are not sufficient release evidence.