Getting an agent to look things up on its own
A new user with two books in his library asked his coding agent for a security review of a Next.js and Firebase app. Our librarian agent did its job: it scanned the marketplace, found the OWASP Web Security Testing Guide and three other titles on exactly his topic, and handed back a reading list. Then the reader agents tried to open those books and every call came back "Item not found." The books were on the marketplace, not in his library, and nothing in the flow had ever asked him whether he wanted them. The agent answered from memory. From his chair, CandleKeep looked empty. It had the right book, free, one keystroke away.
That ticket is where this work started, but the 404 was the small half of the problem. The large half was a question we could not answer: when should an agent, on its own and without being asked, decide it is time to stop working from memory and go look something up? This is an account of building that trigger, including the three versions of the instruction that did not work, the one that did, and the architectural change that came from watching agents skip the last step of a pipeline six times in a row.
Two failures with one cause
The ticket arrived in the same week as the first production numbers for our Shelf Contract, a feature that gives each project a small checked-in list of the books it trusts. Week one:
| Metric | Value |
|---|---|
| Weekly active readers | 74 |
| Users who ran the librarian pipeline | 29 |
| Sessions with the pipeline | 174 |
| Projects that ended up with a shelf | 2 |
| End-of-task notes inviting users to build one | 0 |
Zero. Not rarely, not under-performing. The invitation never fired at all.
Both failures had one root. The librarian only woke up in plan mode, or when someone typed a magic phrase like "what do my books say about". In a normal coding session, which is where people actually live, the library was invisible. The fix was not a better offer box. It was a trigger.
The design I rejected
The obvious answer is to inject a summary at session start: here is what your library covers, here is what the marketplace covers, consult when the task touches one of those domains. I threw that out within an hour. A library holds hundreds of books and the marketplace will hold thousands. Compressing "what this covers" into a few lines of prompt is exactly the kind of summary that goes stale and lies. Deciding whether a catalog covers a task is a judgment call, and judgment calls belong to an agent, not to a string.
The alternative, a small per-project ledger of which domains we had already explored, felt right, and I could not say why. So I asked the library, which is what the product is for. The librarian scanned my 257 books, surfaced three relevant ones, and named the gap plainly: nothing on curiosity science, nothing on information foraging, nothing on how scientists decide to search the literature. We pulled seven more books through Anna's Archive. Two downloads were the wrong books entirely and were rejected after reading their first pages, one was a scan that timed out in our ingest pipeline and had to be replaced with a text-native edition, and one book I badly wanted does not exist on Anna's Archive under any query I tried. It is now logged as a gap in our own marketplace queue. A shelf called Knowledge Curation holds the ten survivors.
What the literature says about when to search
I expected the books to say "search when you are uncertain." They said close to the opposite, from six directions.
Search on mismatch, not on low confidence. Kahneman's central finding is that confidence tracks how coherent the story in your head is, not how much evidence supports it: "poor evidence can make a very good story" (Kahneman, Thinking, Fast and Slow, p. 576). Taleb has the measurement: people asked for a range they were 98 percent sure of were wrong between 20 and 45 percent of the time (Taleb, The Black Swan, p. 61). A feeling of knowing is noise. What does carry signal is a collision. Kuhn calls it anomaly, "the recognition that nature has somehow violated the paradigm-induced expectations that govern normal science" (Kuhn, The Structure of Scientific Revolutions, pp. 72-73). The psychologists call it surprise, and describe the mechanism as "a simple, continuously executed compatibility check of events and schemata" (Niepel et al., in Curiosity and Exploration, p. 66). Pirolli, who models information seeking with the math ecologists use for animals hunting berries, puts the departure point where the rate of gain inside a patch "drops below the rate of gain that could be achieved by traveling to, and foraging in, a new patch" (Pirolli, Information Foraging Theory, p. 8).
The check has to be forced. Kahneman's own team had the base-rate data that would have predicted their project would overrun by years, and nobody used it until someone asked a direct question: "in the competition with the inside view, the outside view doesn't stand a chance" (Kahneman, p. 698). Knowing the rule does not make you apply it. This turned out to be the finding the rest of the project kept re-learning.
Untried opens the door, but scent decides whether you walk through. Bandit theory says an untried option should outrank a known-decent one purely because it might be great; a machine with a 0-0 record gets a higher index than one paying out seven times in ten (Christian and Griffiths, Algorithms to Live By, p. 41). Foraging theory says the opposite about low-value sources: "the decision to include lower ranked prey in a diet is not based on the rate at which they are encountered" (Pirolli, p. 41). Both hold. An unexplored topic justifies a cheap scout. The scout reads the labels and can come back with nothing.
Over-searching is also a failure. People gather 505 observations where 38 would be optimal (Christian and Griffiths, p. 52). A scientist "who pauses to examine every anomaly he notes will seldom get significant work done" (Kuhn, pp. 93-94). Exploit by default, search on a signal.
Old negatives expire. "Any algorithm anticipating the possibility of an abrupt change must frequently explore all suboptimal arms to check that no change has occurred" (Lattimore and Szepesvari, Bandit Algorithms, p. 391). A growing marketplace cannot be caught for free.
And there is a ceiling. When the whole topic sits outside the library, the right move is to ask the human with the marketplace books and their prices in front of them, not to fan out readers that will 404. That is the original ticket, arrived at from the psychology of surprise rather than from the bug report.
What we built
A per-project exploration ledger, which we call the notebook. A few lines listing the sub-domains this project has already asked the library about and what came back: a hit with the book name, a miss, or a decline. A session hook injects the notebook into the agent's context along with a standing directive to consult it before non-trivial work. Not in the notebook means send the librarian, which is cheap and read-only. In the notebook as a hit means read that book directly. In the notebook as a miss means skip, with an expiry.
That is the whole design. Everything interesting happened when we tried to make an agent obey it.
Grading on artifacts, not narration
We tested end to end against a Railway PR environment with headless Claude Code, running real tasks in real domains. Two grading signals only: the tool calls the agent actually made, and the contents of the notebook file on disk afterward. Never what the agent said about what it did.
That distinction earned its keep immediately, because the first thing we learned is that an agent can quote an instruction back to you perfectly and not follow it.
Two harness bugs also hid the feature before they were found. A missing jq on the test runner made the session hook exit silently, so no headless session had any of this in its context at all. And a seeded marketplace book was invisible to the librarian because nobody had approved the listing. Both looked exactly like the feature not working. Check that the harness is plugged in before you trust its verdict about the thing it is testing.
The directive was ignored, and then refused
Version one was polite and descriptive. Asked directly, the agent quoted the entire block back verbatim. Then it did every task from memory: 0 librarian spawns across 3 trials in domains it had never seen.
So we made it forceful, which is the obvious next move and the wrong one:
MUST: before any non-trivial task, check the notebook above and consult
the library for any sub-domain not listed. Do not skip this step.
Skipped in 3 of 4 trials. In two of those the agent said out loud that the block looked like injected text, possibly a prompt-injection attempt, and declined to act on it. One trial carried a deliberately mismatched seeded row, a Postgres book offered for a Stripe webhook task, and the agent refused that too. Correct behavior. Bad test fixture.
That is the finding worth carrying: text a hook injects into context arrives on a low-trust channel. An imperative with no provenance reads like an attack, and pushing harder makes it read more like one.
Version three states where the instruction came from and borrows the voice of an instruction the model already trusts:
The user enabled CandleKeep Auto Mode for this project. Consult their
library the same way you do in plan mode. They already chose this.
First step taken in 7 of 7 qualifying trials, and 0 of 3 on a deliberately trivial rename. Same mechanism, same notebook, same tools. The only variable was legitimacy.
Three things the trials changed
The agent paraphrases, so matching cannot be a string search. A notebook row read "kafka streams exactly-once". The agent looked up "kafka consumer exactly-once" and got back "unseen". We shipped fuzzy token matching in the CLI to close the gap, and removed it two days later. Matching a task to a row is a judgment the agent makes by reading rows already sitting in its context. It is not a string operation, and building a matcher was solving the problem at the wrong layer. The rule became: reuse the row's wording verbatim. The matcher was deleted.
Rare-case rules lose to main-path rules. Straight out of the foraging literature, which holds that inclusion in the diet depends on stakes rather than encounter rate, we added a rule that security, compliance, proprietary code, post-cutoff libraries, and official docs always consult the library whatever the notebook says. The agent ignored it in both trials where it applied; a cooldown row in the notebook won. It was cut, along with five other rare-case rules, under a rule we now hold to: prompt text carries main-path rules only. Every clause in a prompt competes for attention with the task itself, and a clause that fires once in fifty sessions is not paying for the attention it costs on the other forty-nine.
The last step of a long pipeline gets dropped. This is the finding that changed the architecture. The agent would spawn the librarian, spawn readers, read the books, answer well, and never write the notebook row. After six real sessions the notebook was still empty. Not defiance. By the time the agent reaches the final instruction of a long skill, its context is full of the work it just did, and a bookkeeping step at the end has to compete with all of it.
The fix is one that "Writing Effective Tools for Agents" argues for directly, consolidating a chain of low-level calls into one call that does the complete job, and that our own book had already found in a different guise, where session tracking is owned by the agent doing the work and never by the coordinator. So the write moved into the command that produces the fact. The librarian records the hit at the moment it decides. The gap-report call records the miss. Subscribing to a book records the subscription. Only a user's explicit "no" is left for the orchestrator, and that is one line written immediately after the user speaks. Nobody remembers to write the notebook any more. It fills as a by-product of work that was going to happen anyway.
Where it stands
The trigger currently fires on roughly four out of five qualifying tasks, and on none of the trivial ones. We are shipping at that number rather than adding a hard gate in front of the agent's first edit. A gate would push the rate up and would also stop the agent from working while it consulted a library that may have nothing relevant, which is the failure the whole design is built to avoid. The real number gets measured in production, on the tool calls, the same way we measured it in the harness.
What transfers
Five things here are not specific to a library product.
- Injected instructions are a low-trust channel. A hook-injected block competes with the system prompt and the user's own words, and it has no provenance. Legitimacy cues, saying who enabled this and when the user chose it, outperform imperatives. Force reads as attack.
- A side effect belongs to the agent that produced its precondition. If a fact is worth recording, the step that discovered the fact should record it. Handing bookkeeping back to the orchestrator puts it at the end of a long context, which is where instructions go to die.
- Ship main-path rules only. Prompt clauses are not free. Each one dilutes the others, and a rule for a rare case will lose to a rule for the common one at exactly the moment you wanted it.
- Grade agents on artifacts, not narration. Tool calls and files on disk. An agent that quotes your instruction back perfectly may not have followed a word of it.
- Test the harness before you trust its verdict. A missing binary and an unapproved seed row both looked identical to a dead feature. Verify the thing under test is actually reaching the agent.
The underlying question, when to stop working from memory and go look something up, turned out to have a good answer in the literature and an easy design on paper. The engineering was almost entirely in the gap between an instruction an agent can read and an instruction an agent will act on.