// measurement
Whether the number is real.
An independent check on the number your model was approved on. We re-derive it and tell you whether it holds up.
// on this page · 5 sections
A model gets approved on a measured figure, such as an accuracy or a false-alarm rate. Everything downstream assumes that figure is an honest estimate of field performance. Often it is not, and nothing in the reported metrics says so.
We look at the measurement more than the model. The question is whether the number behind the decision would survive being re-derived by someone who did not want it to be true.
What gets audited
Leakage
Information about the label, or information that will not exist at decision time, reaching the model through training or validation. It pushes the reported figure up and never announces itself.
The common classes are written up in the leakage guide.
Training-data filters
Labs and data vendors filter training data with a quality classifier. We measure what it drops, broken out by kind of document, against sets whose quality is known.
One public result: an open filter dropped 21.7% of college-level textbook sections. Details on measurement.
Calibration under drift
Calibration is whether a model's stated confidence matches how often it is actually right. A model can be calibrated at training time and drift out of it as your inputs change. We measure whether it still holds.
This is the failure that passes every accuracy check.
Distributed and federated monitoring
A shared model trains across many machines while one machine's data quietly degrades. The aggregate score barely moves as the damage builds.
Catching it takes a lightweight probe on each node. See the drift-detection guide.
Reproduction from source
We re-derive the headline result from the rawest data we can reach, by a different path than the one that produced it.
When the two paths disagree, the disagreement is the finding.
What an engagement delivers
starts at$5,000
scopeOne defined claim
thresholdsFixed before we look
you keepReport and harness
- We agree on the claim under test and write the thresholds down before anyone looks at the numbers.
- We audit the training and validation path for leakage, one class at a time.
- We measure calibration under the shift your deployment actually faces.
- You get a findings report on which claims survived re-derivation. Anything the data could not test is named as untested.
When we find a fault, a test harness that reproduces it comes with the report. Your team can re-run the check after the fix.
When you do not need this
If a wrong answer is cheap, the audit costs more than the failure. Internal drafts and rankings a person reviews anyway fall here. Ship them and watch them.
This work earns its price when the number is load-bearing. That means a model that gates a decision about money, safety or eligibility, or one being handed to an evaluator who will ask how you know. If that is not your situation, we will say so on the first call.
Where the discipline came from
We learned this running automated systems that trade company capital. There, a number that does not match the exchange's own records is a story, and trusting a dashboard over the actual trade records costs real money. The same practice gets pointed at your measurement. Thresholds are registered in advance, and a ledger records every conclusion that was later reversed.
The same method runs our research lab across three markets. Every hypothesis is written down with its thresholds and kill criteria before the data exists. It is then tested against replays of what the rule rejected as well as what it took.
Many of those hypotheses have died, and the ledger keeps each one with its lesson. When our own wallet audit produced profit figures that would not reconcile with the platform's per-trade records, we suspended it that day and marked it untrusted. That is the standard your number gets held to.
Engagements are confidential. Anchor 163 LLC is registered for federal work, with identifiers on the capability page.
Common questions
An independent check of whether a model's reported performance is an honest estimate of field performance. It examines the measurement procedure — leakage in the training and validation path, calibration under distribution shift, and whether the headline result reproduces from source data — rather than reviewing the architecture.
In-house validation is run by the team whose result it is, usually against thresholds that can move after the numbers arrive. An independent evaluation fixes thresholds before looking, re-derives the claim by a different path, and reports what did not survive.
Engagements start at $5,000 and are scoped to a defined claim rather than an open-ended review. The deliverable is a protocol, a findings report, and the harness that reproduces any fault found.
studio@anchor163.com
// copied