DOCS · REFERENCE
Evals
Quality is graded against independent oracles rather than asserted — what is measured, on what corpus, and what the numbers do not prove.
On this page
pnpm eval, with results committed under evals/results/. Extraction is scored against oracles that share no assumptions with the code they grade.
The scores
| What | Oracle | Corpus | Score |
|---|---|---|---|
| Symbol extraction (TypeScript) | The TypeScript compiler | this repo · 1,102 declarations | 98.7% precision · 100% recall |
| Symbol extraction (Go) | go/parser | Go stdlib · 7,654 files / 316k declarations | 99.5% F1 |
| Symbol extraction (Python) | The ast module | Python stdlib · 3,830 files / 94k declarations | 99.4% F1 |
| Retrieval | 15 labelled questions | this repo's knowledge base | 100% recall · 0.861 MRR |
The stdlib numbers are the ones worth quoting: large corpora nobody tuned against, graded by the language's own parser.
What the harness refuses to fake
- A missing toolchain is reported as skipped, naming which — not as a pass.
- A zero-file corpus reports "none here" rather than a meaningless 100%.
- The retrieval number is called what it is: fifteen labelled questions is a regression guard, not a benchmark, and
evals/README.mdsays so in those words.