Company Memory and Retrieval

Every company wants the same thing eventually: a system that knows what the company knows. It is the most requested project and the most reliably disappointing one, and the reason is almost never the model.

Separate three things that all get called memory, because they have different solutions and people conflate them constantly.

• Memory within a run. What the system has in front of it right now. Finite, and it fills up.
• Memory across runs. What it recalls from last Tuesday. Usually a file or a database row, and usually simpler than people expect.
• Company knowledge. The twelve years of documents, tickets, contracts, and decisions that nobody has read in full. This is the hard one and the rest of this chapter is about it.

How retrieval actually works

You cannot put twelve years of documents in front of a model. So the pattern is fetch, then answer. Find the handful of passages likely to be relevant, put those in front of the model, and ask it to answer from them. Fetch, then answer.

The plumbing has four parts, and each one is a place things go wrong:

1. Chunking. Documents get cut into passages. Cut them badly and a table gets separated from its heading, or a clause from the contract it modifies.
2. Embedding. Each passage gets converted into a position in a mathematical space where similar meanings sit near each other. This is what lets a search for "termination terms" find a passage that says "either party may end this agreement."
3. Search. A question gets converted the same way, and the nearest passages come back. Meaning-based search alone is bad at exact strings like part numbers, so serious systems run keyword search alongside it and merge the results.
4. Reranking. A second, slower pass reorders the candidates by actual relevance. Skipping this is the most common cheap mistake.

This matters for how you respond when it goes wrong. The instinct is to change the model or rewrite the prompt. The useful move is to look at what was actually retrieved for the failing question. Nine times in ten, the right passage was not in the set, and no model can answer from material it was not shown.

The governance problem nobody mentions in the demo

Here is the part that should stop a project until it is answered, and it is specific to this pattern.

Your company's documents have permissions on them. Some folders are HR only, some are finance only, some are for one team. When you index everything into a retrieval system, you have built a copy of the company's knowledge with the permissions stripped off. Ask it a question and it will happily answer from a document the asker was never allowed to open. Nobody broke in. The system worked as designed.

There are two honest ways to handle this and one dishonest one.

• Index only what everyone may see. Simple, safe, and adequate for a surprising number of real use cases. Start here.
• Carry permissions through the whole path. Every chunk keeps the access rules of its source, the search filters by the asking user's rights before returning anything, and the rules stay in sync when they change upstream. This is real engineering work and it is the correct answer for anything company-wide.
• Instruct the model not to reveal restricted material. This is the dishonest one. It is a preference, not a control, and it fails the first time someone phrases the question sideways.

Documents, graphs, and the middle ground

Plain retrieval over passages is good at questions whose answer sits in one place. It is bad at questions that require connecting things: how a decision made in one project affected another, or every contract that shares a particular clause.

The alternative approaches extract entities and relationships into a structure first, so the system can traverse connections rather than only matching text. This works, and it costs more to build and considerably more to maintain, because the structure has to be rebuilt as the underlying material changes.

My honest read is that most companies asking for the sophisticated version have not exhausted the simple one. Good chunking, hybrid search, and reranking over a curated set of documents would get them eighty percent of the value. Start there, measure what fails, and let the failures justify the complexity. The failures will be specific and you can point at them, which is a much better basis for spending money than a diagram.

The unglamorous part that decides the outcome

The strongest predictor of whether one of these projects works is not the technique. It is whether the source material is any good.

If your documentation holds four versions of the same policy with no dates, a system built on it will cite the wrong one. It will do so in a tone that sounds authoritative. Retrieval has no opinion about which of your documents is current.

So before indexing anything, do the boring work. Pick the authoritative source for each topic and mark it. Date everything. Archive superseded versions out of the index. Give each area an owner. This is a documentation project wearing an AI project's clothes, and saying so early is more honest than discovering it in month three.

Where it should live

Three questions decide this, and only the third is technical:

• What tier is the material? Chapter six's classification decides most of this before you start.
• Who is asking, and from where? A system for twelve internal people has different requirements from one exposed to customers.
• What does the volume look like? Which leads directly into the arithmetic in the next chapter.

One durable principle regardless of where it lives: keep the index rebuildable from the sources. If the retrieval layer is the only place some piece of knowledge exists, you have turned a convenience into a system of record. People will treat it as one.