8 Ingesting knowledge at scale
Nobody has ever onboarded a colleague by pointing at four hundred wiki pages and walking away. Plenty of people have, and the colleague spent a month believing the architecture diagram from the year before the rewrite and calling the product by the name it had until the acquisition. The pages weren’t wrong when they were written. They were just handed over without the ten minutes at the whiteboard that tell a newcomer which of them to believe.
Those ten minutes are the subject of this chapter. Sooner or later the teaching of chapter 7, one fact at a time, meets the existing pile: the wiki, the shared drive, the decks. The agent can learn from it in an afternoon, and it wants to, like Johnny 5 flipping through pages of an encyclopedia: input, more input, need input. What it can’t do is bring the judgment a person brings to a wiki, the sense of what’s current, what was aspirational and which of two contradictory pages to trust. Chapter 1 said that judgment has to be handed over. This is how, and how you check what came back.
8.1 What a briefing contains
A briefing is the paragraph you’d say to a person before they opened the wiki, and the useful ones are concrete about the past. Names first: the product now called Settlement was called Ledger until the reorganization two years ago, so normalize to Settlement; the wiki says gateway where the code says edge, and edge is the word to keep. Then a rule for contradiction, since a wiki of any age contains them: trust the newer page, unless one of them is the runbook. Then what to skip: the archived spaces, the personal pages, anything under Drafts. Then the one everybody forgets because it’s obvious: the decks were written to persuade and the wiki is what we actually do, so where they disagree the page wins and the deck’s version goes in as an [idea] at most.
Add where things go: which domain, which folder layout, the tags in use. And the shape of the review you want: propose before writing, keep source links, ask about anything resembling a decision nobody wrote down. It fits in one message:
You: Read the engineering space of the wiki through the connector and
propose engrams for the engineering domain. Ledger is now called
Settlement - normalize. Where the wiki says gateway the code says
edge; use edge. Skip the Archive space and anything under Drafts.
On a contradiction trust the newer page unless one is a runbook.
The Q1 platform deck is aspirational; the wiki is what we do.
Keep a link back to every source page and its last-edited date.
Propose the list first and I will trim it.That message is the difference between an ingest that lands and one that produces a confident, well-formatted archive of things that stopped being true.
8.2 The sources, and who reads them
Crystalline ships no scraper, by design. The agent is the ingester, and it reads sources with whatever its harness gives it: the wiki over the MCP connector the harness provides, a folder of exported documents through its file tools, a repository through the code tools it already has. A repository, though, is a source the agent consults rather than one it ingests: the repository holds the code, and the engrams hold which repository holds the code for which purpose, how to get access and the architecture and concepts that make the right file findable in a minute. The division is precise: the harness supplies the reading, the agent supplies the distillation and Crystalline is the destination, where what was derived lands as engrams with provenance, and the review loop that checks it afterwards.
A wiki is the common source, read page by page through the connector. A folder of documents, PDFs and exports and the odd spreadsheet, is read the same way. Slide decks are the one source that usually comes in through Fluid instead: a person drops the deck onto the engram it belongs to, and the agent reads it back over MCP as an attachment, so the deck becomes teaching material rather than a file nobody opened (chapter 7). And a tree of markdown that already carries frontmatter, an old knowledge base that was nearly an archive already, doesn’t go through the agent at all: crystalline import <src> --domain <name> folds it into a domain as a pure file transformation, dry run first, normalizing legacy type values, backfilling status and dropping sentinel dates on the way. The pictures and documents its notes link to come along into the domain’s assets/ folder with the links rewritten, and the report names any it couldn’t take.
8.3 Distill, then propose
The cardinal sin of ingestion is mirroring: copying a source wholesale so that the archive looks complete, then filling the gaps with guesswork. A vendor recall notice is forty screens of marketing header, revision table, legalese and support matrix, with three paragraphs that matter. Distilled, it becomes two observations, a resource link back to the page and a source_date in the frontmatter: forty screens down to two facts and one link. That link points at the exact page or file the engram came from, pinned to the commit it was read at when the source is a repository. When the agent answers from the engram, it hands you the link with the answer, so you can check the source or read on. Every location the engram was distilled from also goes into its sources list, so when the wiki changes, the agent finds the engrams of the changed pages by that list. It rewrites the address of a page that moved and drops one that’s gone, and a page that’s gone is no reason on its own to retire the engram it fed.
Distilling is also where the briefing does its work. The agent reads a page about Ledger and writes Settlement; it finds a deck slide claiming a feature and files it as an [idea] because you told it decks persuade; it meets two pages that disagree and applies the rule rather than picking one silently. And one shape earns verbatim retention, only ever beside its distillation: a document whose exact words you may need to quote. The summary stays an ordinary engram where its topic lives; the full text goes in as a type: source engram under sources/, linked both ways with summarizes and summarized_by. It’s split into parts only when it outgrows its budget, which is four times the size an ordinary engram gets. On recall the summary is what gets read, and the full text is pulled in only when the summary falls short.
Then, before anything lands, comes the list:
Proposed for engineering (23 engrams):
runbooks/ Settlement deploy runbook
runbooks/ Retry queue drain procedure
Edge routing rules
Encom file format, v3
...
Write these? (y / n)You trim it. Proposing first is the rule for a whole space, not just a page, because the trimming is where your judgment enters the archive, and striking five lines off a list is cheaper than retiring five engrams next month.
A source too large for one session, a wiki space of a few thousand pages or an archive of documents, isn’t read front to back. The agent groups its parts by topic, hands each topic to a subagent of its own on a capable model (where the harness offers subagents) and folds what they bring back into the knowledge.
8.4 The review loop
What the agent writes is a proposal in a second sense: it carries provenance, and a person reviews it. Each engram names its source, a link, a date, an attachment, so the reviewer can open the page it came from and check the derivation. The review happens in Fluid, where an engram is a page and a fix is an edit, and a reviewer who knows the material corrects the one value the agent got wrong without going back through a session. A team domain sends the ingest out as a proposal (chapter 17).
Some duplicates never wait for the sweep. A write can come back naming the engrams it’s closest to, so a wiki page that repeats one the agent captured an hour earlier is merged on the receipt rather than an hour later. The agent says what it did with the list instead of moving on. Then run the sweep. The evolve tool is meant to be called after a large ingest, and its queue is built for this moment: links that don’t resolve because a page referred to a title nobody captured; near-duplicate clusters where two wiki pages became two engrams about one thing; a tag spelled two ways; an attachment nobody has analyzed; and the human captures that reviewers made in Fluid while correcting, each raised as a finding to verify, tag and wire into the graph. Work the mechanical findings in a pass, propose the judgment ones one at a time and sweep again until the number stops falling. That shrinking number is how you know the ingest settled.
8.5 Temporal hygiene
An ingest is where dates go wrong, because a page written two years ago lands in an archive that treats everything as true today. Three fields keep it honest. source_date is when the source was written, and the agent sets it from the page’s last-edited date, so an engram derived from an old page says so. valid_from is for a fact that started on a known date, a policy that came in with the reorganization, and nothing else; absent means it has always held, and an absent valid_to means it still does, so neither is set to mean “we do not know”. stale_after is the one that buys you the most: put it on anything with a shelf life, a vendor’s terms, a pricing table, a contact name, and the sweep raises the engram when the date passes and nobody has re-checked it. A re-check that finds nothing changed is one verified entry, or a later stale_after, and the engram goes quiet.
8.6 When not to bulk-ingest
Curation beats completeness, and the archive isn’t a mirror of the wiki. If a space is dead, leave it dead; if a page would not survive the “would I tell a new hire this” question from chapter 2, it does not become an engram because it existed. Four hundred pages distilled to nineteen engrams, worth more than the four hundred because every one is something the agent will act on.
The strongest case for not ingesting is the source tree. A person doesn’t remember the code by the letter. Well, Commander Data does, and yes, he’s a person, the court said so; but you didn’t hire Data. A person remembers where and how to find things, the map chapter 7 said to teach, and lets the territory stay where it is. Ingesting the source would give you a stale copy of something the agent can already read, and the map fits in a dozen engrams.
Nothing about an ingest is special on disk. Each derived engram is one markdown file with frontmatter, the same format as a fact taught across a desk, with resource, sources, source_date and generated as plain keys anyone can grep. A domain that lives in a repository shows the ingest as a diff: new files, nothing rewritten elsewhere, so a bad batch reverts with the version control the team already has. Attachments sit under the domain’s assets/ folder, referenced by relative path from the engrams that claim them. The index is derived from the files (crystalline reindex --full re-reads every file and keeps the embeddings), and crystalline verify in the repository’s CI refuses a malformed engram before the team pulls it.
Your reading log has four hundred and twelve entries, and a notes app holds a decade of half-sentences about them; you trust neither completely. You give your agent a briefing like the one above and let it read both over an afternoon. It comes back with a list of twenty-three engrams, each naming its source entry. You strike four: two are about a shop that closed, one is a wish list from 2019 dressed as taste, one is a note about a book you were given and never opened. The agent writes nineteen, every one with a link and a date, the shops among them with a stale_after for next year because opening hours change.
The sweep the next morning has eleven findings. Six are mechanical, links to titles the log spelled differently and the ingest renamed, and the agent fixes them in a pass. Three are judgment: a near-duplicate pair, two engrams about Dune from different years, which you have the agent fold into one; an orphan nothing links to yet, wired to the paperback rule; and a tag spelled two ways, which comes back as a crystalline tags merge command for you to run. The remaining two are your own: corrections you typed in Fluid while reviewing, which the agent verifies and wires into the graph. On Thursday you ask which Star Trek novels you already own, by series, and get the answer, the source entry for each and the date you wrote it, from a log you haven’t opened in a month.
Today the wiki is the source and the archive is derived from it. The last chapter turns that around: once the agent’s knowledge is what the team teaches and corrects, the wiki pages and the decks become presentations of it, kept in step with it, which is no revolution at all, since every diagram and slide that exists today was already a reflection of somebody’s knowledge. The ingest is the last time the flow runs this way round.
Bulk ingestion needs the briefing a new colleague would get before opening the wiki: what was renamed and what to normalize to, which page wins a contradiction, what to skip and which sources persuade rather than describe. The agent reads the sources through its harness, distils rather than mirrors, keeps a source link and date on every engram, proposes the list before writing and lands the result in Crystalline, where a person reviews it in Fluid and the next sweep surfaces the unresolved links, duplicates and unreviewed captures an ingest leaves behind. Set source_date, reserve valid_from for genuinely bounded facts and put stale_after on anything with a shelf life. And don’t ingest what the agent can read for itself: teach the map of the source tree, not the tree.