How Your Knowledge Base Dies—Most Likely from Bookkeeping

Deep thoughts on AI and aspirations —— ByteDance Deep Thinking Circle

Most people’s knowledge bases follow a similar lifecycle. The first month brings high enthusiasm—building directories, adding tags, categorizing archives. By the third month, only bookmarking remains, no more organizing. After six months, even you don’t want to open it.

The cause of death is usually attributed to laziness. The real cause is more specific: updating cross-references, revising old viewpoints overturned by new materials, maintaining consistency across dozens of pages. These tasks can collectively be called bookkeeping. It’s tedious, time-consuming, and the burden grows heavier with more materials. When people can’t stand it anymore, the knowledge base dies.

This diagnosis comes from a note by Andrej Karpathy, and the remedy he prescribes following this diagnosis is worth more scrutiny than the diagnosis itself.

Retrieval Won’t Save It Because Every Time Starts from Zero

What’s the mainstream approach now? Upload files, retrieve relevant fragments when asking questions, piece together an answer. NotebookLM follows this logic, various file upload and Q&A services follow this logic, and the vast majority of knowledge base products on the market still follow this logic.

This method works, but has a fundamental flaw: every time starts from zero. When you ask a question requiring synthesis of five documents, the model searches again, pieces together again, derives again. Where did the previous derivation go? Lost in the chat history. Ask the same question twice, get two parallel derivations that don’t recognize each other. After a year of use, nothing has accumulated.

The difference between two paradigms becomes clearer when compared side by side:

Change Direction: Compile at Write Time, Read Directly at Query Time

Karpathy’s approach is to change paradigms. Don’t retrieve at query time, compile at write time. Every time a new material is added, instead of storing it and waiting for your questions, the model directly reads, directly digests, and integrates information into a continuously maintained wiki: updating related pages, marking contradictions with old content, establishing cross-references, adding synthesis conclusions. When querying, you’re asking this already-digested base, answers come with references, and good answers can be archived back into the base.

The structure has three layers. The bottom layer is raw materials—articles, papers, data—read-only and unchangeable, this is the source of facts, the model can’t touch it. The middle layer is the wiki itself, entirely written and maintained by the LLM, you only read. The top layer is a configuration document telling the LLM what this base’s structure is, what conventions exist, what rules to follow when encountering new materials—in Claude Code it’s CLAUDE.Md, in Codex it’s AGENTS.Md.

He made an analogy: Obsidian is the IDE, LLM is the programmer, wiki is the codebase. A knowledge base should be managed like a code repository, with inputs, compilation, artifacts, and tests.

The ideological origin of this traces back to 1945. Vannevar Bush’s envisioned Memex was a personal, curated knowledge system where the associative paths between documents were as valuable as the documents themselves. The only problem he didn’t solve was: who will maintain it? For eighty years the answer was there’s no suitable person, now the answer has become LLM. It won’t tire, won’t forget to update a reference, maintenance cost compressed to near zero—for the first time a knowledge base is economically viable.

Don’t Celebrate Too Soon: The Compiler Will Silently Write Wrong Answers

This is what I consider the most underestimated risk in the entire solution. In manually written bases, errors are scattered and visible; in LLM-compiled bases, errors are systematic and beautifully formatted. When the model mixes in a hallucination while organizing materials, it gets written into a concept page, given proper cross-references, and from then on becomes an “authoritative conclusion” in this base. The tidier the base, the more confident the errors.

So the periodic health checks in the solution aren’t decorative. Having the model periodically scan the entire base, finding contradictions, orphan pages, data gaps—it’s equivalent to running lint and tests on the knowledge base. Following the codebase logic further, there should also be version rollback: if overall base quality drops after a certain ingestion, you can revert to the previous version. A compiled knowledge base without inspection is more dangerous than a manual one because it wraps errors in a credible shell.

One more thing to think through in advance: compilation solves maintenance cost, not signal-to-noise ratio. If the materials themselves are garbage, the compiled product is structured garbage. The gateway for entry into the base is still guarded by people—if you can’t guard it, everything afterward is wasted effort.

Copying Guarantees Failure—Three Places Revised by Practice

Karpathy himself admits this whole thing currently looks like a bunch of hacky scripts, and only a minority can get it running. People practicing along this line of thinking often make cuts in three places, each one touching mechanism rather than preference.

First, tools don’t matter. Some people find existing note-taking tools hard to use and directly use AI to build an extremely simple tool that only they understand, can get it running in one morning, what to put in and how to design indexing follows entirely their own workflow. The mechanism is that tools should grow from your own workflow—transplanting someone else’s solution is guaranteed to fail, including Karpathy’s.

Second, the bookmarking step can be cut. Bookmarking satisfies the desire to bookmark—something you can’t quickly understand right now, if bookmarked, probably won’t be looked at again. A more ruthless approach is to only converse, not bookmark: when you see valuable content, directly open a window and talk it through thoroughly, useful conclusions go into the base, don’t keep the original link. Filter first then enter the base—much higher signal-to-noise ratio than accumulating a pile of raw materials then batch compiling.

Third, large bases become project bases. Stuffing all topics into one base, maintenance cost rises exponentially with scale; dividing by project, scope is bounded, cost is controllable, redundancy is acceptable. Knowledge’s destination is specific projects, just like compiled applications ultimately run in specific containers.

Minimal Closed Loop You Can Build Today

Starting requires no ceremony. One directory, two layers for raw and wiki, one configuration document with structure and conventions written clearly. Each time new material is ingested, have the model update related pages and index; do a health check once a month, have it find contradictions and gaps itself. Judge what’s missing from the base, have it recommend directions to dig into next based on existing content.

The most imaginative scenario for this approach is actually in teams. Meeting notes, customer calls, project documents—these things today are basically sunk costs, no organization will allocate human resources to organize them, and this has never made sense historically. When maintenance cost drops to near zero, this calculation works for the first time.

One final judgment I’ll leave to you: for materials that should or shouldn’t enter the base, the standard is actually simple—the effort required to understand it, are you willing to give it now? If not, keeping it is just part of the graveyard.

Key points: Knowledge base death is caused by friction from bookkeeping-style maintenance, not laziness; retrieval Q&A starts from zero every time, no accumulation; compiled approach puts digestion at write time, relies on LLM as maintainer; LLM compilation silently writes errors, health checks are necessary testing stages; tools, granularity, base division have no standard answer, determined by your workflow; team sunk data is the biggest incremental scenario for this paradigm.