DNAV-1 "GPS-denied navigation" · D-HOWTEST-1 "How we test, explained" · 13 Oct 2026 · All documents

How we test Pathfinder

Every result comes from one pipeline: a written-down flight, flown in the ArduPilot simulator, scored against the simulator's true position, then judged by acceptance tests whose pass marks were locked before any result existed. A change reaches main only when the numbers improve.

Pages: Jury deck · Results explorer (deck slide 26) · Flight simulator · Results · Demo · Start here · How we test · Index · 14 Oct day summary

The pipeline

Scenario
A YAML file: route, wind, sensor noise, when GPS is jammed or faked, and a fixed random seed.
sim/scenarios/
Simulated flight
The real ArduPilot autopilot (SITL) flies the route closed loop; Pathfinder feeds it a position over MAVLink after the jam.
tests/harness/
Scoring
Pathfinder's estimate is compared with the simulator's true position once per second from the jam on.
results JSON per flight
Locked acceptance tests
AC-01 to AC-09. Pass marks written on 7 Oct; files hash-locked, so no threshold moves after a result.
tests/acceptance/LOCK.sha256
Merge rule
A branch merges only if it beats the current best at 2 min, is not worse at 1 min, breaks no locked test and every test passes.
D-MERGE-NIGHT-1

What a score means

Schematic, not data.

Metric	How it is read
Error at 60 / 120 / 300 s	Distance between estimate and truth 1, 2 and 5 min after the jam; the median over held-out flights.
Time to pass 5 m	Seconds from the jam until the error first exceeds 5 m.
Honest error circle	Share of samples where the truth lies inside the reported 95% radius; pass band 90-98% (AC-03).
Mission kept	Autopilot stays in AUTO and reaches every waypoint (AC-01).
Spoof caught	Fake GPS flagged within 5 s, no jump past the radius (AC-04).

Held-out flights and the comparison set

Held-out flights

Scored flights live in sim/scenarios/heldout/. Their seeds are never used for training, so a model is always judged on flights it has not seen.

Every flight is flown by every system

ArduPilot alone
Physics only
Physics + speed AI (each model version)
Physics + drift-correcting AI
Speed AI + drift AI
Drift AI, gated
Terrain matching

Defined once in sim/tools/naming.py (COMPARE_SET). A system whose setup preset is not on main yet is skipped with a note, never silently.

Held-out aircraft

Training flights are tagged by frame (quad, hexa, octa); sim/tools/fly_training.py:30 skips hexa unless --allow-hexa, so hexa scores a frame the model never saw. Pick it with lab.py --frame. In-flight learning (experimental-learn-air) is scored against the same system with it off.

The nine locked acceptance tests

Test	Proves	Pass mark
AC-01	Mission continues	AUTO kept, all waypoints, in 95% of held-out runs
AC-02	Less drift	median error 50% below ArduPilot alone, and below physics only
AC-03	Honest radius	truth inside r95 in 90-98% of samples
AC-04	Spoofing rejected	flagged within 5 s, autopilot switched, no jump past r95
AC-05	GPS return	re-trusted within 15 s
AC-06	Vehicle profile	frame, motors and IMUs detected on every frame
AC-07	Calibration helps	lower median error at 2 min with it on
AC-08	Cross-platform	core tests pass on Linux x86, ARM and Windows; builds for Android
AC-09	Deterministic	the same log replayed twice gives bit-identical output

Fast run (DNAV-TEST-1 "Acceptance tests in under 1 hour"): fast run 2 took 1,086 s, measured (sim). The files in tests/acceptance/ are never edited by a branch; the merge rule checks that too.

The Drift Benchmark

How drift grows with flight time, weather and turns: 135 scenarios, seeds 3000-3134, in sim/scenarios/benchmark/.

1-10 min flights
jam at 30 s
jam at 30 s, GPS back at 90 s
jam at 2 min
calm · wind 6 m/s · gusts
straight · zigzag · curve

Fly: python sim/tools/benchmark_run.py, then chart: python sim/tools/benchmark_chart.py. The old names ladder_*.py still work. Since 14 Oct each host flies two simulators side by side (--workers 2: one SITL instance per worker, own ports and lock, RAM guard before each flight); measured (sim): 0 IMU gaps over 50 ms with two running, about 6.5x faster than real time per simulator including boot and teardown.

Smoke, 6 of 6 flown (straight, calm, jam at 30 s, one seed), end error at 1 / 3 / 5 min: physics only 2.5 / 5.0 / 34.4 m; physics + speed AI 16.4 / 7.3 / 11.8 m. measured (sim)

Drift Benchmark smoke: error at the end of the flight against flight length, jam at 30 s — Smoke chart; the other 129 scenarios are not flown yet.

Distance series (D-DIST-1 "Distance suite")

How drift grows with distance flown after the jam: 60 scenarios, seeds 4000-4059, benchmark-dist-* in sim/scenarios/benchmark/. GPS is jammed at 30 s and never comes back.

0.5 · 1 · 2 · 5 · 10 km after the jam
5 · 10 · 15 m/s
straight · curve
calm · wind 6 m/s
every system in the comparison set

Scored as error in m and as % of the distance travelled since the jam (truth track), at the flight's target distance. Fly: python sim/tools/benchmark_run.py --series distance (narrow with --km, --speeds).

Smoke, 9 of 9 flown (straight, calm, 10 m/s, one seed), error at 0.5 / 1 / 2 km: ArduPilot alone 427 / 928 / 1,913 m (86-96%); physics only 0.7 / 1.5 / 17.9 m (0.1-0.9%); physics + speed AI 4.5 / 1.9 / 13.9 m (0.2-0.9%). measured (sim)

Distance series smoke: error as percent of distance since the jam against km, per system — Smoke charts; 5 and 10 km, 5 and 15 m/s, curve and wind are not flown yet.

Distance series smoke: error in metres against km since the jam, per system — Smoke charts; 5 and 10 km, 5 and 15 m/s, curve and wind are not flown yet.

Ablation and add-on checks (14 Oct)

An add-on is shipped behind an off switch and joins the headline only if it beats physics only when flown. Three levels of evidence, weakest first: unit test (synthetic input in cargo test), offline (replay) (100 flown logs replayed through the new code, sim/tools/wind_after_jam.py --variants), measured (sim) (flown closed loop).

Ablation benchmark (DNAV-ACC-10 "Ablation benchmark"): python sim/tools/benchmark_run.py --systems ablation flies physics only, each add-on alone and all add-ons fused (experimental-fused) on the priority set; python sim/tools/ablation_chart.py charts error at 1 / 2 / 5 min and % of distance. 324 flights on 4 simulators, flying.

What today's checks showed: last-layer tune won offline (306 to 145 m @120 s) and lost flown (2.45 vs 3.32 m, 1 of 7); wind after jam caught a gust in the unit test (4.09 to 0.41 m/s) but in replay 5 wind flights ran away; speed scale absorbed wind in replay (83.6 to 306 m). An offline win is not a flown win. Detail: 14 Oct day summary.

Results explorer: python sim/tools/export_explorer.py exports every flown result to docs/pitch/explorer-data.json; the deck (slide 26) and docs/pitch/explorer.html redraw from it, no hand numbers.

Repeats and labels

Repeats

Each scenario has one fixed seed, so a flight can be flown again exactly; a recorded log replays bit-identically (AC-09). Flying every system 5-10 times with new seeds is planned, not done: today most numbers rest on one seed per flight.

Every number carries a label

measured (sim) flown closed loop in the simulator.

offline estimate a recorded flight replayed through new code; the autopilot did not react to it.

estimate a calculation or projection, not flown.

What is not tested

No real hardware: no flight on a real drone, no real IMU vibration, no real weather.
No real flight logs yet; every number is from ArduPilot SITL.
AC-08 on Android: the core builds, it has not been run on a device.
Terrain matching wins offline but lost one closed-loop flight.

Sources: tests/acceptance/, docs/specs/2026-10-07-pathfinder-mvp-design.md, sim/tools/naming.py, sim/tools/benchmark_*.py, LAB.md, RESULTS.html.