How Your Knowledge Base Dies, Mostly from Bookkeeping

Deep thoughts on AI and aspirations —— ByteDance Deep Thinking Circle

Most people’s knowledge bases follow a similar lifecycle. The first month is full of enthusiasm—creating directories, tagging, categorizing. By the third month, it’s just bookmarking with no organizing. After six months, you don’t even want to open it.

The cause of death is usually blamed on laziness. The real cause is more specific: updating cross-references, revising old viewpoints overturned by new material, keeping explanations consistent across dozens of pages. These tasks can collectively be called bookkeeping. It’s boring, time-consuming, and the burden grows heavier as material accumulates. People can’t stand it, and the knowledge base dies.

This diagnosis comes from a note by Andrej Karpathy, and the remedy he prescribes following this diagnosis is more interesting than the diagnosis itself.

Retrieval Can’t Save It Because It Starts from Zero Every Time

What’s the mainstream approach now? Upload files, retrieve relevant fragments when asking questions, and piece together an answer. NotebookLM follows this logic, various file upload and Q&A services follow this logic, and the vast majority of knowledge base products on the market still follow this logic.

This method works, but has a fundamental flaw: it starts from zero every time. You ask a question requiring synthesis from five documents, and the model searches, pieces together, and derives from scratch. Where did the previous derivation go? Lost in the chat history. Ask the same question twice and get two parallel derivations that don’t acknowledge each other. After a year of use, nothing has accumulated.

The difference between the two paradigms is clearer when placed side by side:

Change Direction: Compile on Write, Read Directly on Query

Karpathy’s approach is to switch paradigms. Don’t retrieve at query time; compile at write time. Each time you add new material, the model doesn’t store it waiting for your question but directly reads and digests it, integrating the information into a continuously maintained wiki: updating related pages, marking contradictions with old content, establishing cross-references, adding synthesized conclusions. When querying, you’re asking this already-digested base, answers come with references, and good answers can be archived back into the base.

The structure has three layers. The bottom layer is raw material—articles, papers, data—read-only and unmodified, the source of facts the model cannot touch. The middle layer is the wiki itself, entirely written and maintained by the LLM, with you only responsible for reading. The top layer is a configuration document telling the LLM what the structure of this base is, what the conventions are, and what rules to follow when encountering new material—in Claude Code it’s CLAUDE.Md, in Codex it’s AGENTS.Md.

He made an analogy: Obsidian is the IDE, the LLM is the programmer, the wiki is the codebase. A knowledge base should be managed like a code repository, with input, compilation, artifacts, and tests.

The intellectual origin of this goes back to 1945. The Memex envisioned by Vannevar Bush was a personal, curated knowledge system where the association paths between documents were as valuable as the documents themselves. The only problem he didn’t solve was: who maintains it. For eighty years the answer was no suitable person; now the answer has become LLM. It doesn’t tire, doesn’t forget to update a reference, maintenance costs are compressed to near zero, and for the first time a knowledge base is economically viable.

Don’t Celebrate Too Soon: The Compiler Will Silently Write Wrong Answers

This is what I consider the most underestimated risk in the entire solution. In a manually written base, errors are scattered and visible; in an LLM-compiled base, errors are systematic and beautifully formatted. When the model mixes in a hallucination while organizing material, it gets written into a concept page, tagged with proper cross-references, and becomes an “authoritative conclusion” in this base from then on. The tidier the base, the more confidently wrong it is.

So that periodic health check in the solution isn’t decoration. Having the model periodically scan the entire base, finding contradictions, orphan pages, and data gaps, is equivalent to running lint and tests on the knowledge base. Following the codebase approach further, there should also be version rollback: if overall quality drops after an ingestion, you can revert to the previous version. A compilation-based knowledge base without a checking mechanism is more dangerous than a manually written one because it wraps errors in a credible shell.

One more thing to think through in advance: compilation solves maintenance costs, not signal-to-noise ratio. If the material itself is garbage, the compilation output is structured garbage. The gateway of entry to the base is still guarded by humans—if you can’t guard it, everything downstream is wasted.

Copy-Pasting Will Fail: Three Places Revised by Practice

Karpathy himself admits that this setup currently resembles a pile of hacky scripts, and only a few people can get it running. Those who practice along these lines often make changes in three places, each one being a mechanism rather than a preference.

First, tools don’t matter. Some people find existing note-taking tools hard to use and directly have AI build an extremely simple tool only they understand, which can be up and running in a morning, with complete control over what to put in and how to design the index according to their own workflow. The mechanism is that tools should grow out of your own workflow—someone else’s solution transplanted over will inevitably fail, including Karpathy’s.

Second, the bookmarking step can be cut. Bookmarking satisfies the urge to collect; something you can’t quickly understand right now probably won’t be looked at again if bookmarked. A more ruthless approach is to only dialogue, not bookmark: when you see valuable content, open a window and chat through it immediately, useful conclusions go into the base, original links aren’t kept. Filter first then enter the base—much better signal-to-noise ratio than accumulating a pile of raw material then batch compiling.

Third, large base becomes project bases. Stuffing all topics into one base makes maintenance costs rise exponentially with scale; splitting by project means bounded scope, controllable costs, and acceptable redundancy. Knowledge belongs in specific projects, just as compiled applications ultimately run in specific containers.

Minimum Viable Loop, Can Be Set Up Today

Getting started requires no ceremony. One directory, two layers raw and wiki, one configuration document with clear structure and conventions. Each time new material is ingested, have the model update related pages and index; do a health check monthly, have it find contradictions and gaps itself. Judge what’s missing from the base and have it recommend the next direction to dig based on existing content.

The most imaginative scenario for this approach is actually in teams. Meeting notes, client calls, project documents—these things today are basically sunk costs, no organization will assign manpower to organize them, and historically it has never made sense. When maintenance costs drop to near zero, this calculation works out for the first time.

One final judgment for you: whether material should go into the base, the standard is actually simple—are you willing to put in the effort to understand it right now? If not, collecting it just adds to the graveyard.

Key points: Knowledge base death is caused by bookkeeping maintenance friction, not laziness; retrieval-based Q&A starts from zero each time, no accumulation; compilation-based puts digestion at write time, relies on LLM as maintainer; LLM compilation will silently write errors, health checks are a necessary testing mechanism; tools, granularity, and base segmentation have no standard answer, determined by your workflow; team sunk data is the biggest incremental scenario for this paradigm.