ANCHOR 163 // home
// measurement

Whether the numbers hold up.

We build instruments that check whether an AI system's numbers hold up. The newest one, telltale, catches a bad node that a fleet's average score hides. It runs inside Flower and NVIDIA FLARE.

// on this page · 7 sections
In federated learning, many machines train one shared model without moving their data. When one machine's sensor slowly goes bad, it keeps sending normal-looking updates. The healthy majority outvotes it, so the fleet-wide score barely moves while the damage builds.
We measured that. In a ten-node simulation, the fleet accuracy an operator would watch hid all 90 faults we planted.
faults the fleet score hid90 of 90
slow drift caught4–7 rounds
sent per node~1 kB a round
false alarms0–1.5%
All four numbers are from image data. Results on underwater acoustic data differ, and both sets are on the drift-detection page.

Inside Flower and NVIDIA FLARE

How it works

Every node answers the same short quiz of probe items with its current model and sends back only its answers. No raw data leaves the node, and nothing about the training changes.
It runs as the aggregator, the step that combines every node's update, in either framework. The detector itself depends on neither.

What it catches

A node whose answers drift away from the group, or from its own past. Wrong labels are caught and the node is named within three to four rounds. Slow sensor drift is caught in four to seven.

What it does next

It can quarantine the flagged node, leaving its updates out of the shared model, or alert an operator with the evidence attached.

Where it fails

Gain drift on image data is often missed, and detection gets harder the more the nodes' data differs. We publish those limits with the results.
NVIDIA FLARE's research index lists the example as an externally maintained project. We also contributed a fix to FLARE's simulator, merged in October 2026.
The full results, the chart and the methods are in drift detection in federated learning. The code is open source on GitHub.

AI agents

The same method monitors AI agents: it catches the one agent whose behavior quietly changed and measures how often the alert is wrong. Details are on agent monitoring.

A check on training-data filters

AI labs filter their training data with a quality classifier. It is a model that scores every document and drops the low scorers. What the filter throws away is rarely measured.
We ran FineWeb-Edu's open classifier, which Hugging Face built to keep educational web pages, over 1,000 documents from each test set. It dropped 21.7% of college-level textbook sections and 1.9% of grade-school lessons.
The model card warns that it targets grade-school content, and this puts a number on that warning. It is one classifier at one cutoff, and the textbook sets are synthetic text, so treat it as a first measurement. The code is open source on GitHub.

Independent model audits

Leakage

Information that will not exist at decision time reaching a model during training, which inflates its reported accuracy.
The common forms are in the data leakage guide.

Calibration under drift

Whether a model's stated confidence still matches how often it is right as its inputs change.

Reproduction from source

Re-deriving a headline result from the rawest data by a different path. When the two disagree, that is the finding.
Scope and pricing are on the AI model audit page.

Where the discipline comes from

We run automated trading systems on our own capital, where a number that does not match the exchange's records costs real money. Every figure gets reconciled to source before it is believed. The reconciliation tool is open source as kalshi-truth.
The same habit drives this work. Thresholds are set before the data exists, and the failures are published next to the successes.

Working with us

SAM.govActive · All Awards
CAGE246L3
UEIN8TAP6NS5PY8
primary NAICS541511
Anchor 163 LLC is a small business in San Francisco. We work with ML teams, research groups and federal programs. Competencies and registration details are on the capability statement, and updates go out on LinkedIn.

Common questions

Yes. The detector is framework-agnostic. It needs a framework that sends a shared model to nodes, trains locally and returns updates. Flower and NVIDIA FLARE are the two integrations we have built and run.
No. It is our own code, built on each framework's public API. NVIDIA FLARE lists the example in its research index as externally maintained. We are not affiliated with Flower Labs or NVIDIA.
No. Each node returns only its answers to a small fixed quiz, about one kilobyte a round.
In a ten-node harness that reproduces exactly from its seeds, with faults planted at known times and severities, and false alarms counted on healthy runs the thresholds never saw. The plug-in was then run in a live five-node Flower federation, and in 72 NVIDIA FLARE simulator runs with ten sites.