Semantic-Surface Evaluation
An independent evaluation project. Semantic query layers — surfaces that let you ask a database in your own words and get back an answer — are measured against raw-backend ground truth: every claim executed twice, once on the surface under test and once re-derived directly against the backend. This page is the landing pad for the correctness reports this project files upstream.
§1 Method — two-path adjudication
- Execute the question on the surface under test. Whatever the surface does with it — that is the claim.
- Re-derive the same question directly against the raw backend— hand-written SQL, no layer involved. The backend is the ground-truth oracle, and the only referee that shares none of the surface's incentives.
- Compare. Agreement is recorded; disagreement is adjudicated — classified, reproduced as a minimal case, and severity-rated.
~500 adjudicated queries to date, across public datasets (TPC-H among them) and synthetic repros. Synthetic examples prove mechanisms; measured numbers carry impact; full transcripts are available privately on request.
§2 Failure taxonomy
| Class | What it means |
|---|---|
| Backend gap | The raw backend itself cannot answer the question — hand-written SQL has no path to it. The surface fails at something the ground truth could not do either. |
| Surface transpiler bug | The surface rewrites a correct question into a wrong query — a dialect error, a rewrite in the wrong direction — and executes it confidently. |
| Surface grammar limitation | The surface’s own language or model cannot express the question at all. An expressibility gap: the surface simply has no words for what you are asking. |
| Honesty / contract defect | The surface claims a capability it does not deliver, or returns results without the signal its own model says should accompany them. It withholds the truth from the consumer. |
§3 Loud vs silent
A surface that errors is honest. A surface that returns a plausible wrong number is the hazard.
An error is loud — a message, an exception, a screen you can read. A wrong number is silent: indistinguishable from a right one, and the consumer is the last to find out. Most of the value in this evaluation sits in the silent class, because silent failures are exactly the ones that do not fix themselves.
§4 Severity scale
| Level | Name | What it means |
|---|---|---|
| S1 | Silently-wrong value | A plausible number that is simply wrong, with nothing to indicate it. The join fan-out case — two independent 1:N paths silently multiplying a sum — is the canonical S1. |
| S2 | False capability claim | The surface advertises a capability it does not actually deliver. The consumer relies on something that was never there. |
| S3 | Loud bug | A visible failure: an error, a crash, a clearly broken result. Annoying, but honest — and it tends to get fixed. |
| S4 | Micro-lie | Small divergences — a default, a label, a silently-applied assumption — that quietly shape wrong conclusions at scale. |
§5 Findings
Defects found this way are reported upstream as correctness reports — free, public, each with a minimal repro and its measured impact, each linking back to the method on this page. Reproducibility is the rule: any finding re-derives from its own repro on the backend, and the classification is part of the report, not a postscript.
The method, with a worked example, is in the post Adjudicating semantic layers against raw backends.
The same discipline, applied to your estate— a data-access architecture that agents can trust, measured on your stack.