// measurement
Measuring whether AI can be trusted.
We build instruments that check whether an AI system's numbers hold up. The newest one plugs into Flower, the open-source federated learning framework, and catches a node the fleet average hides.
// on this page · 5 sections
In federated learning, many machines train one shared model without moving their data. When one machine's sensor slowly goes bad, it keeps sending normal-looking updates. The healthy majority outvotes it, so the fleet-wide score barely moves while the damage builds.
We measured that. In a ten-node simulation, the fleet accuracy an operator would watch hid all 90 faults we planted.
faults the fleet score hid90 of 90
slow drift caught4–7 rounds
sent per node~1 kB a round
false alarms0–1.5%
A plug-in for Flower
How it works
Every node answers the same short quiz of probe items with its current model and sends back only its answers. No raw data leaves the node, and nothing about the training changes.
It works as a Flower aggregation strategy. The detector itself does not depend on Flower.
What it catches
A node whose answers drift away from the group, or from its own past. Wrong labels are caught and the node is named within three to four rounds. Slow sensor drift is caught in four to seven.
What it does next
It can quarantine the flagged node, leaving its updates out of the shared model, or alert an operator with the evidence attached.
Where it fails
Gain drift on image data is often missed, and detection gets harder the more the nodes' data differs. We publish those limits with the results.
The full results, the chart and the methods are in drift detection in federated learning.
Independent model audits
Leakage
Information that will not exist at decision time reaching a model during training, which inflates its reported accuracy.
The common forms are in the data leakage guide.
Calibration under drift
Whether a model's stated confidence still matches how often it is right as its inputs change.
Reproduction from source
Re-deriving a headline result from the rawest data by a different path. When the two disagree, that is the finding.
Scope and pricing are on the AI model audit page.
Where the discipline comes from
We run automated trading systems on our own capital, where a number that does not match the exchange's records costs real money. Every figure gets reconciled to source before it is believed.
The same habit drives this work. Thresholds are set before the data exists, and the failures are published next to the successes.
Working with us
SAM.govActive · All Awards
CAGE246L3
UEIN8TAP6NS5PY8
primary NAICS541511
Anchor 163 LLC is a small business in San Francisco. We work with ML teams, research groups and federal programs. Competencies and registration details are on the capability statement.
Common questions
The detector is framework-agnostic. It needs a framework that sends a shared model to nodes, trains locally and returns updates. Flower is the integration we have built and run.
No. It is our own plug-in, built on Flower's public API. We are not affiliated with Flower Labs.
No. Each node returns only its answers to a small fixed quiz, about one kilobyte a round.
In a ten-node harness that reproduces exactly from its seeds, with faults planted at known times and severities, and false alarms counted on healthy runs the thresholds never saw. The plug-in was then run in a live five-node Flower federation.
studio@anchor163.com
// copied