Skip to main content
  1. Enterprise Brain
  2. Benchmarks
  3. Code review

Benchmark · built from AmpUp's own code and conversations

A code reviewer that remembers what the team decided.

AmpUp's code review bot reviewed 688 real regressions twice: once from the diff alone, and once with six searches over the team's past pull requests, Slack threads and meetings. With Enterprise Brain it caught more of them. A few catches needed a meeting, where the team had agreed on something no pull request wrote down.

Regressions caught20.3% from 17.1%+3.2 pts · interval +0.7 to +5.6
Same-size placebo15.3%Unrelated evidence lowers catches
Open pull requests, Oct 277 confirmed bugsflagged only with evidence

At a glance

Would the bot have flagged the bug before it shipped?

Every test case is a pair from AmpUp's own history: a pull request that introduced a bug, and the later pull request that fixed it. The bot reviews the buggy one as it stood before merge.

The task
Review a pull request: read the title, description and diff, then post review comments.
A catch
A review that flags the bug the later fix repaired.
Test cases
688 regression pairs from AmpUp's main repository, with the buggy pull request merged between March and September 2026. Sealed before any review ran.
What's compared
The same bot and prompt, given nothing extra, given Enterprise Brain's evidence, or given a placebo: another pull request's evidence of the same length. Two more arms tried learned notes and the bot's earlier advice.
Models
The reviewer is AmpUp's production review prompt on GPT-6.1 Sol. Three GPT-6 Sol passes judge each review.
The memory
7,357 merged pull requests, 3,668 Slack threads and 3,403 team meeting transcripts, from September 2025 to October 2026, with 86,803 facts extracted from them.
Grading
Each pull request reviewed twice per arm, 9,632 reviews in all. Three judge passes read each review beside the fix; two of three must agree it flagged the bug.

How a review uses Enterprise Brain

Six searches before the first comment.

The bot itself doesn't change. Enterprise Brain adds a short brief of what the team already knows about the code being touched.

  1. A pull request opens

    The review bot starts as it always does.

  2. Write six searches

    A model reads the title, description and changed files and writes six questions for memory, such as who owns this setting and what broke here before.

  3. Search memory

    Enterprise Brain returns the best facts and passages from past pull requests, Slack threads and meetings, each with its source. It has 30 seconds.

  4. Review with evidence

    Up to 40 facts and 8 passages go into the review prompt. If memory is slow or down, the review runs without them.

Building a fair test

688 regressions the memory was never built or tuned on.

Mined from the repository's history, May 2024 to Oct 20262,266
The fix changes code the buggy pull request wrote1,512
Test split, merged Mar to Sep 2026, inside the memory's months821
Not used in the first experiment that found the gain690
Every arm's input ready; list sealed688
How the pairs were found
How the buggy pull request was identifiedPairs kept
Git blame: the fix's rewritten lines trace back to the pull request that wrote most of them. A model checked each pair twice; a blind hand check found 28 of 30 real.750
Pairs from the earlier version of this benchmark454
A fix titled "Regression from #N"249
Other wording, such as "caused by #N"33
A revert made because of a defect20
A bug ticket naming the pull request that broke it6

  • No peeking at the future

    Memory is searched as of the moment the buggy pull request merged. Nothing newer, and nothing from the buggy pull request or its fix, can appear. A scan of all 32,387 evidence items found none that broke this.

  • A placebo arm

    One arm gets another pull request's evidence: about the same length, just as old, about unrelated files. If any extra text helped, this arm would gain too.

  • Sealed in advance

    The test list and the analysis plan were fixed and hashed before any review ran.

  • Judged without knowing the arm

    A separate judge rated every comment as valid, plausible or noise, without seeing which arm wrote it.

Results

What the evidence says matters, not how much there is.

Share of the 688 regressions each review flagged. Same bot, same prompt, same pull requests; only what sits beside the diff changes.

Enterprise Brain evidence, as it runs in production20.3%
Enterprise Brain evidence, original embedder19.8%
The bot's earlier advice17.4%
Nothing extra: the diff alone17.1%
Placebo note: another pull request's note16.6%
Notes learned from graded practice reviews16.1%
Placebo: another pull request's evidence, same size15.3%
Full table, with noise
What the bot saw beside the diffCaughtNoise comments per review
Nothing extra17.1%0.21
Enterprise Brain evidencefacts and passages from past pull requests, Slack and meetings19.8%0.27
Same, with the production embedderwhat reviews use today20.3%+3.2 [+0.7, +5.6]vs placebo +5.0 [+2.8, +7.3]0.28
Placeboanother pull request's evidence, same size15.3%0.20
Notes learned from graded practice reviews16.1%−1.0 [−2.3, +0.2]not rated
Placebo noteanother pull request's note16.6%not rated
The bot's earlier advice17.4%0.33

Intervals are paired bootstrap intervals over pull requests at 98.3%, wider than the usual 95%, because three memories were tested at once (Holm correction). 56 pull requests were caught only with evidence; 26 only without.

Where the extra catches came from

Mostly past pull requests. Sometimes a meeting.

The bot with evidence caught 46 regressions that the bot without it missed. For each one, we traced what the catching comment rested on.

  • 24Earlier pull requests
  • 2Meetings
  • 0Slack threads, so far
  • 20Only the diff: the evidence steered attention, or chance

Past pull requests carry most of the gain. Meetings occasionally hold a requirement that no pull request wrote down; the two cases below show it.

Two regressions only a meeting could catch

The decision was said out loud, never written in a pull request.

To check, we reviewed each pull request again with pull-request evidence only: the same searches, with Slack and meeting results removed and refilled from the next pull-request results.

A rep's messages landed in another rep's campaign.

A pull request captured replies that reps send by hand on LinkedIn and filed each one under the campaign its contact matched. It never checked which rep sent the message.

The bug

One rep's manual message could show up in another rep's campaign and pause that rep's sequence. A later pull request fixed it.

What the meeting said

In a lead-generation planning meeting on Jul 14, the team agreed that different reps may message the same contact in different campaigns. Enterprise Brain stored it as a decision.

With meeting evidence

"Different reps are explicitly allowed to contact the same person in different campaigns. […] A's messages are attached to B's contact and campaign instead."

Caught 2 of 2
Pull requests only

The same line was flagged as a newer sequence "stealing" replies. The cross-rep problem never came up.

Caught 0 of 2 · diff alone 0 of 2

Introduced in #11160, fixed in #12760.

The course report card became unreachable.

A pull request made each course row in the learner's history open the course certificate.

The bug

The completion report card, with its score, concept breakdown and retake, could no longer be reached from history.

What the meeting said

In a morning sync on Aug 13, the product lead asked to see the report cards for every completed course, which didn't exist yet.

With meeting evidence

"Replacing all course-row navigation with openCertificate removes the History route to the completion report."

Caught 1 of 2 (2 of 2 in the main test)
Pull requests only

Neither review commented on where the row navigates.

Caught 0 of 2 · diff alone 0 of 2

Introduced in #12346, fixed in #12822.

On open pull requests

Seven real bugs, flagged before merge.

We ran the production bot on AmpUp's 15 open pull requests, with and without evidence. Twelve were reviewed; three dependency bumps were skipped. Seven comments that appeared only with evidence rest on it directly. An engineer checked each one: all seven are real bugs.

  1. Reattach library files in roleplay chat#13480 · minor

    The new file picker accepts only document types, so people who build a roleplay persona from a ZIP of material can no longer attach it with the paperclip.

    Rested onA frontend team meeting where the product lead said people upload ZIPs of content through chat to build personas. The pull request never mentions ZIP files.

    Confirmed
  2. Show the overall metric score in the hero#13663 · major

    A score on a 10-point scale is merged with an org's own scale, such as 1 to 3, and shown out of 10, so those orgs see a misleading hero score.

    Rested onTwo earlier pull requests about org-specific score scales. This pull request never mentions scales.

    Confirmed
  3. Gate household pages for non-admin users#13541 · major

    Household memory uses the old access check, while account access moved to the new permission system. Someone who can open an account they created would be denied its memory page.

    Rested onThe earlier pull request that moved account access to the new permission system.

    Confirmed
  4. Advisor homepage: coaching this week#13548 · major

    Orgs with a saved homepage layout will never see the new section: the homepage renders only the sections in that saved list.

    Rested onThe earlier pull request that made the homepage render only saved sections.

    Confirmed
  5. Advisor homepage: coaching this week#13548 · major

    It asks for meeting analysis without the advisor's email, so uploaded meetings that only have per-person analysis show no coaching.

    Rested onTwo earlier pull requests: one that gave uploaded meetings per-person analysis, one on how the overview falls back.

    Confirmed
  6. Keep roleplay attachments out of the knowledge base#13524 · major

    The code reads a field that the data hook no longer requests, so files marked for the knowledge base would be dropped when the roleplay is saved.

    Rested onThe earlier pull request that trimmed the hook to request only the fields it reads.

    Confirmed
  7. Hide roleplay tags a guest has no content for#13695 · minor

    If a guest's active tag filter loses its last roleplay, the tag vanishes from the rail, leaving an empty grid and a filter they can't see or clear.

    Rested onAn earlier pull request that hit the same invisible-filter problem.

    Confirmed

Searching memory for all 12 pull requests cost $0.02.

What it costs

About a dollar a day for a busy repository.

$1–1.5a day: searches, longer prompts and keeping memory current
~6,100extra prompt tokens per review
≤ 30 sbefore the review starts
+0.07noise comments per review, at most