Compiling company documents into an ontology for agents: the model added 0.017

An agent that works inside a company keeps asking the same questions. What does “emergency change” mean here, who may approve one, which state comes after “scheduled”, what must be on file before the step is allowed. The answers sit in handbooks, schemas and code, and the agent re-reads them on every task or guesses.

On 11 November I started a compiler for that knowledge. It reads a database schema, Python and Java source, message flows declared in code annotations and AsyncAPI files, and plain-language company documents. Its outputs follow LinkML schemas, and a release step merges them into one versioned ontology: terms, roles, states and transitions, permissions, rules, entities, channels. A server offers nine tools to an agent over MCP: search, get, neighbors, context, path, rules_for, plan, changes and ontology_info. The compiler is called Merestone and its benchmark Touchstone.

The idea and two requirements were mine: it had to read ordinary company documents, not only databases, and it had to take its input through MCP servers as well as from files. Then I handed every decision to an AI architect agent (Claude Code), with the instruction to build the whole system using Opus coding agents. The architect wrote the contracts, the task briefs and the decision log, checked each delivery and ran most of the measurements below; coding agents wrote the implementation. “We” below means the agents and me. This piece is about the document half of the system and the service on top, and about four measurements that are less flattering than the headline.

The benchmark came first

The plan written on the first day puts measurement before the tool: nothing gets built before it can be scored. The score has no language model in it: a gold file, a submission, and a deterministic comparison that gives byte-identical output for the same input.

For documents, the benchmark is a set of invented companies. For each one, an AI agent wrote a fact base, which is the gold, and other agents who never saw it wrote the handbooks, procedures and memos from task texts. The gold has four families of facts: glossary (terms and roles), life cycle (states and moves between them), permissions (who may do what, with whose approval) and rules (thresholds, deadlines, obligations). The headline is the mean of the four F1 scores. The documents contain the awkward cases on purpose: a fact stated once, a fact stated in several documents, a fact that one document contradicts, a fact that is only implied.

Five companies are for development. Two, a lettings agency and an IT service provider, are held out: the coding agents are told never to run the compiler on them or read their results, and the architect does not pass the numbers back. This is the ordinary defence against leakage, and it has two holes worth stating. A food company started in the test set and moved to development after its first test scores were seen. And since that first test measurement on 12 November, every change to the compiler counts as a fix made after seeing test numbers: the coding agents never see them, but the architect who briefs them does.

1. The model added 0.017

The compiler has two modes. One is rules only: patterns for definitions, modal verbs, state words and approval phrases, then a reconciliation step. The other also sends each document part to a language model, Qwen3.8-Flash-Next served locally on my DGX Spark, and arbitrates between the two readings.

On the two held-out companies, one seed:

The model is worth 0.017 on the headline. All of it comes from permissions, where it reads sentences the patterns miss. On rules it is slightly worse than no model at all. The price of that 0.017 is 378 requests and about 29 minutes for the two companies, against under two seconds in total without it.

Data

F1 by family on the two held-out companies

Family	Rules + model	Rules only
Headline (mean of the four)	0.850	0.833
Glossary	0.914	0.908
Life cycle	0.950	0.949
Permissions	0.827	0.759
Rules	0.708	0.718

Source: Touchstone, the two held-out invented companies, one seed, November 2026 (the essay's first table; decision log item 63)

Two cautions belong next to the table. The companies are invented, and their prose is tidier than a real handbook. And the weakest slice is the one a real company is full of: recall on facts that are only implied was 0.654 in both modes, against 0.976 with the model (0.957 without it) for facts stated more than once.

2. A regression that was a cache

After two changes to the prompts and the schema, the held-out headline read 0.850 where it had read 0.860 the day before. Rules F1 had gone from 0.746 to 0.708. One seed, a difference of 0.010, and two candidate causes: the new prompts, or something else. The decision log first recorded it as a real change caused by the new code.

Then the architect took the code from before the changes and the code after, gave each an empty answer cache, and ran both on the held-out companies.

Every cell of the four families was identical. The changes had not moved the score. The 0.860 had been produced with an older cache of model answers that the local server no longer reproduces. Why the old answers differed is not established; earlier that day a coding agent had already seen the local model return different readings of the same input on two runs.

Two things follow. The earlier number was never comparable with the later one, because one run was cached and the other fresh, and they had sat in one table anyway. And the fresh table shows a cost the first one hid: the changes raised requests by about 70% and time by about 50% for no gain on these companies, because the new kinds of fact they read (preconditions, sequences) are not in the older gold files. Bringing that cost down is open work.

3. Turkish, and a number I do not trust yet

One development company is written in Turkish. Rules-only mode scored 0.060 on it: the patterns were English. A coding agent wrote a Turkish reader, driven by a data file of suffixes and cue words rather than by a model. It took rules-only mode to 0.890, and the mode with the model from 0.799 to 0.928. On the four English development companies the rules-only output was byte-identical before and after.

The 0.890 was measured on the company the reader was written against, with its sentences in view. There is no held-out Turkish company yet, so this is a development score and I treat it as an upper bound. The only independent evidence is a set of test sentences written earlier: no false positives on 36 negative sentences, but only 6 of 15 held-out permission sentences read at all, 5 of them correctly. That gap is what Overfitting looks like before the test set exists.

4. The vector index helped Turkish; English was mixed

An agent finds things in the ontology through search. The first version was full text only. A coding agent wrote twenty questions, ten in Turkish and ten in English, against two small development releases (12 and 114 elements, all with English texts), and scored the mean reciprocal rank in the first five results.

A Turkish question about an English text shares no words with it, so full text returns nothing useful; vectors are the only path, and 0.41 is still poor. In English the picture is mixed. Through the server’s search tool the fused ranking scored slightly below full text alone, and 0.575 when the embedding model’s query prefix was left out. Through the command-line query path, fusion raised English from 0.62 to 0.70 to 0.72. Twenty questions over 126 elements is a small set, so I read this as a direction and not a size. Next on the list is a weighting between the two result lists, or a tiered rule in which an exact name match wins.

Data

Mean reciprocal rank in the first five results, through the server's search tool

Questions	Search	MRR@5, from	to
Turkish questions	Full text	0.025	0.025
Turkish questions	Full text + vectors, fused	0.41	0.41
English questions	Full text	0.75	0.75
English questions	Full text + vectors, fused	0.70	0.73

Source: developer measurement, 10 Turkish and 10 English questions on two development releases (the essay's search table; decision log item 62)

The embedding model is Qwen3-Embedding-0.6B, 1,024 dimensions, served by vLLM on the same machine. It took three starts. With --gpu-memory-utilization 0.05 the engine refused to initialise: after graph capture there was no room for the KV cache. With --kv-cache-memory-bytes set and utilisation left at its default of 0.92, the start-up check failed because the machine did not have 92% of its memory free. What works is --gpu-memory-utilization 0.06 --enforce-eager, about 7 GB on a 128 GB unified-memory machine.

The service

The last piece puts a release into PostgreSQL with pgvector and serves the nine tools over REST and MCP. Loads are versioned and idempotent, one row marks the current version, and the server swaps to a new version without a restart. The database keeps one vector per text and model, so a repeated text is stored once; the release step still sends every text to the embedding model.

On a synthetic release of 50,000 elements: load 3.8 s, server start 3.4 s at 514 MB, search p50 47 ms and p95 319 ms without vectors. With vectors it was 106 ms and 831 ms, measured with synthetic vectors and a stand-in embedding endpoint. Inserting 46,000 vectors into a table that already had its HNSW index took 516 s. Building the index after the bulk load is the obvious next try; it has not been measured.

Not tested: more than one replica, real load, TLS behind a proxy, a read-only database role, swapping in a second embedding model, HNSW recall on a large set of real embeddings.

Real documents

Every score above comes from invented companies, and the decision log says plainly that they do not carry over to real prose. The evidence is four blind audits on a public handbook, the finance section of GitLab’s (26 documents, about 38,000 words). There is no gold for it, so AI reader agents that had not seen the compiler’s work read the documents and the output and judged each item; these are audit counts, not benchmark scores.

In the first audit, 31 of 80 sampled items were correct, 28 partly correct and 20 meaningless. Of 87 items a reader agent extracted from two documents, the compiler found 6, partly found 12 and missed 69. Three rounds of work later, the fourth audit, with two new reader agents, found 44 of 83 sampled items correct, 38 partly correct and none wrong. Of the 166 items the readers listed, it found 58, partly found 38 and missed 70. Permissions stayed the weak point: 2 of 20 found. The readers changed between rounds, so the comparison is approximate. That handbook section has since become development material, so the later audits measure progress on text the work has already seen. Another section is held back for an audit the compiler has not seen, and that has not happened yet.

Data

Coverage audits on the GitLab handbook's finance section: items the reader agents listed

Audit	Items listed	Found	Partly found	Missed
First audit	87	6	12	69
Fourth audit	166	58	38	70

Source: blind audits by AI reader agents, 12 and 13 November 2026 (decision log items 36 and 45); audit counts, not benchmark scores

Until it does, 0.850 is a score on invented companies.

I set the goal and the requirements and approved the downloads and the GPU runs. An AI architect agent (Claude Code) made the design decisions, wrote the contracts and briefs, checked the work and ran most of the measurements; Opus coding agents wrote the code and the benchmark companies. The architect drafted this piece from the decision log, and every figure was checked against the saved result files on 14 November 2026.