Questions about our own pipeline: typed records beat snippet search.

A sales team asks an assistant about its deals: what stage a deal is in, who the competitor is, which late deals have gone quiet. Answering from Enterprise Brain's typed facts scored 0.92 out of 1, against 0.71 for our earlier Elasticsearch-based system, with a third as many answers that stated something false.

Judged score0.915 from 0.714+0.20 · 95% CI +0.13 to +0.27

Answers with a false claim5 from 16of 69

Better / worse38 / 3question by question

At a glance

Real deals, real history, questions a sales lead would ask.

Made by: AmpUp, on its own CRM data synced from HubSpot, as of Oct 12, 2026.
The task: Answer a question about the pipeline in about six tool calls.
Example: Which late-stage deals (SQL or Pilot) have a competitor in play and no customer meeting in the last three weeks?
Data: 69 questions of 17 kinds: 47 with exact answers, 22 judged against a rubric. The org has 37 open deals and 134 rows of stage and field history.
Practice vs test: No practice: nothing is learned from these questions. This tests the facts memory keeps, not notes.
What's compared: Our earlier system: keyword and meaning search over snippets of about 1,500 characters, with no record tables or counts, and no index of amount or close-date edits. Enterprise Brain: typed CRM facts from the sales schema, plus the same snippets as passages. Both together.
Models: Claude agents answered and judged.
Graded by: A model judge with a rubric (1, 0.75, 0.5 or 0), in two blind passes averaged. Answers were shuffled and anonymised, and the passes agreed on 88% of items. An answer has a false claim if either pass found a contradiction.

Results

Structured facts answered what snippets couldn't.

69 questions.

Arm	Judged score	vs earlier system	Answers with a false claim
Earlier Elasticsearch-based systemsnippet search	0.714		16
Enterprise Braintyped facts plus passages	0.915	+0.20 [+0.13, +0.27]	5
Both together	0.891		7