61. What’s It Like to Train Large Models at Harvey? — Joey Wang
Topics
AI Training Data and Evaluation Benchmarks
Startup Playbook for Vertical AI Applications
Autonomous Agents Reshaping Enterprise SaaS
Specialized SLMs vs General LLMs
Enterprise Private AI and Real Compute Demand
Enterprise Multi-Agent Orchestration Architecture
Pioneers
Joey Wang
Summary
- Harvey’s moat is closer to a legal evaluation capability that evolves with customer use cases. The company is valued at $15.5B, has 3,500 customers across 85 countries, and is used by about 80% of the top 100 U.S. law firms. Joey’s core question is when a model reaches the point where “a lawyer would actually use it”—a judgment that depends on accumulating customer feedback, legal expertise, and data taste.
- For now, engineering improvements are more likely than training a new model to improve the customer experience directly. Joey’s example: cut search time from 2 seconds to 200 milliseconds, increasing the number of files searchable in a minute from 30 to 300. Moving from one search at a time to 10 in parallel takes that to 3,000—a benefit customers feel immediately. Training Harvey’s own models is more of a long-term investment in finding the right balance between performance, cost, and future profit margins.
- As general-purpose models improve, vertical products are redesigning their workflows. Last summer, Harvey’s Workflow Builder spelled out each step—uploads, feedback, prompts, and branches—to keep agents from forgetting the task or stopping early. This year, the product has shifted to Agent Builder, where users describe the deliverable with a prompt and source materials. For example, they can specify Japanese and Singaporean legal materials and ask for a finished PowerPoint, leaving the workflow more open-ended.
- Joey favors strong agents coordinating multiple specialized legal models. The team benchmarks base models against specific use cases before post-training them. Joey has publicly mentioned Kimi, GLM, and other models; Raymond also mentioned Qwen. DeepSeek Flash might suit high-volume tasks. Raymond described Tanner as using Kimi K3 at the time, while Joey said training is not tied to a single base model and gave Kimi as a possible base for Tanner 1.0.
- Legal AI creates product value by reducing review costs and preserving the rationale for edits. NDA review requires understanding which clauses a client must hold firm on and where it can compromise, then presenting proposed changes as redlines. Raymond’s counterexample is instructive: rewriting the whole document turns review into a game of spot the difference. Traceable recommendations are what let the product fit into the lawyer’s negotiations with the other side.
- The returns on in-house model training have to be weighed against the possibility of failure. Raymond posed a scenario: OpenAI scores 80, Kimi K3 scores 78, and Harvey wants to train a model to 82—what if it falls short? Joey said performance could also go from 78 to 75. Behind public results are many failed attempts, and a trained model may never be deployed to customers; what the team learns for the next attempt matters just as much.
- For teams to share agents, the main hurdles are permissions and context management. Joey wants multiple people to contribute to the same context in Slack, see the same tool-call trace, and agree on the result. That requires enforcing rules for public and private information, individual permissions, and code review. Employees also have to believe that “even tomorrow, their boss won’t be able to access” their private chats.
- Harvey is currently prioritizing design partners equipped to manage AI outputs. Smaller firms may need things that do not fit Harvey’s existing product, while self-serve access could lead users to trust answers too much and assume Harvey is responsible for the outcome. Joey says both sides need the ability to trust and manage the relationship: “They trust Harvey, and Harvey can trust them” (“他们信任Harvey,Harvey也能信任他们”).
Deep dive
1. Harvey is now part of the workflow at most top U.S. law firms
- Raymond said Harvey is valued at $15.5B, with 3,500 customers across 85 countries. Joey added that about 80% of the top 100 U.S. law firms use the product. Harvey has around 1,500–1,600 employees, and its customers include law firms and corporate legal departments.
- Raymond guessed Joey joined in about 2023; Joey said it was roughly 3 years ago. Before Harvey, he was Figma’s first mobile app engineer and also worked at Uber. Early on at Harvey, he worked on History and Vault.
- Joey joined after a PhD he had followed, who was also a Twitter influencer, introduced him to the company. That person later became Harvey’s first recruiter. Joey took the PhD’s willingness to join as a sign of conviction in the field. After speaking with the founders, he thought Harvey could represent the next generation of software companies.
- He wanted to get back to the pace of building things: have an idea, know customers need it, ship it, and make them happy. At larger companies, approvals and interdepartmental friction had made that experience increasingly rare.
2. The product fits into how lawyers already work
- Joey starts with how clients work: reviewing contracts in Word, comparing materials in Excel, and coordinating emails in Outlook. Harvey connects to those systems, generates documents, and proposes edits while delivering work in formats lawyers already know.
- The challenge in NDA review is knowing which clauses must be changed, which cannot be changed, and where the client can compromise. Harvey applies what it has learned about the client’s past decisions to new tasks. Joey said this depends on clients trusting Harvey, keeping their data in their own cloud services, and allowing Harvey to access it.
3. Redlines make edits traceable recommendations
- Raymond’s observation was that rewriting a whole contract makes lawyers play “spot the difference.” Redlines show deletions, replacements, and explanations, so lawyers can review each change and decide whether to accept it. That is what actually reduces the workload.
- Joey added that the system also needs to distinguish whether an edit came from Harvey, the client, or opposing counsel, and explain why. It is like a PR and commit history in code: the point is to preserve an audit trail of how each edit was made.
- When exported, Harvey’s edits appear under the lawyer’s name. Lawyers may also keep the edits as suggestions for discussion rather than accept them all; the other side can still see the proposed changes before the final version is produced.
4. Legal agents could also contribute to decisions and creative work
- Joey wants Harvey to bring taste to legal decisions. One client told him that after Harvey had handled the tedious work, they kept discussing the project and Harvey raised several points the client “hadn’t thought of before.”
- Joey said Harvey has legal knowledge and extensive context, and could talk with lawyers like a partner. He also sees the generative model’s creativity as part of the vision. Some exceptional partners already use AI this way, but that does not mean everyone can immediately treat AI as a partner.
- Asked about the company’s name, Joey said he could neither confirm nor deny whether it came from Suits. He said the goal is to take part in legal decision-making, not just handle tedious work.
5. Better models are shifting products from step-by-step orchestration to goal-setting
- Early on, the founders ran legal workflows through GPT-3.5, compared the results with human work, and showed them to OpenAI’s general counsel. They then got early access to GPT-4. Joey said Harvey now also helps model providers evaluate and improve their legal capabilities; the relationship is “mutually reinforcing.”
- Last summer, Workflow Builder let teams specify file uploads, user feedback, prompts, and branches for reuse. At the time, agents might forget the task halfway through, stop early, or generate an answer without asking for the necessary materials.
- This year, Harvey shifted to Agent Builder: users can describe a goal with a prompt and source materials, such as generating a PowerPoint based on Japanese and Singaporean legal materials. Workflows that once required people to maintain each step can now be handed off more freely.
6. Commercializing in-house models builds on years of research and use-case experience
- Joey said Harvey’s early team was mostly made up of model researchers. When he joined, the real bottleneck was product: there were no PMs or designers yet, so he handled many product designs and specs himself.
- Joey clarified that members of the founding team came from Google DeepMind and Meta FAIR. Harvey worked closely with OpenAI early on, including on pre-training. It also worked with PwC on a text model and tried selling it to more customers.
- But as general-purpose models improve, purpose-built pre-training models may gradually fall behind them. Post-training on limited data may also struggle to keep pace with Anthropic, OpenAI, or Gemini.
- The change this year is that Harvey now has enough customer feedback to draw on. Feedback and use-case descriptions that do not involve customer data or private information can help the company understand the legal market, simulate dozens or hundreds of types of legal tasks, and generate synthetic data and benchmarks. Those can then support post-training and trade-offs among performance, price, and latency.
7. Data taste determines whether evaluations reflect customer needs
- Beyond data volume, Joey emphasized “taste in data”: the team has to judge what data is good, what is missing, and whether an evaluation actually correlates with customer outcomes. Training a model to summarize everything in one sentence may not suit lawyers, who may need a long argument with “every sentence indexed” so they can trace its source.
- Benchmarks also need to evolve with real usage and may need to cover scenarios customers could use in the future. For new evaluations the team is unsure about, it runs shadow tests and A/B tests against real tasks to see whether they reflect customer needs.
8. A strong agent can coordinate a group of specialized legal models
- Model selection starts with benchmarks for specific use cases, favoring a “small and specialized” approach. Joey has publicly mentioned Kimi, GLM, and other models; Raymond also mentioned Qwen. DeepSeek Flash might suit high-volume tasks that do not require especially strong reasoning.
- Raymond worried that using multiple specialized models would be too complex. Joey said a strong agent could handle delegation, like a capable leader assigning product, frontend and backend, infra, and security work to specialists. He also said this reflects his own view of how the architecture should work.
9. Tanner explores training methods and partnership combinations
- Joey described Tanner as a direction and one of the more promising, larger-scale training models Harvey is working on; it performs well on M&A and due diligence work. Raymond characterized its base model at the time as Kimi K3. Joey responded that training is not limited to a single base model and gave Kimi as a possible base for Tanner 1.0.
- Raymond raised Fireworks as an example of a matrix of base models and partners. Joey explained that internal bandwidth is limited, so Harvey cannot post-train or fine-tune every combination. Instead, it works with different labs in different ways and learns through those exchanges.
- Joey is willing to share the training methodology because he believes the real moat is knowing when a model reaches the threshold of “a model a lawyer would actually use.” Partner labs cannot make that call for Harvey.
10. Legal benchmarks build from repeatable judgments, then expand
- Raymond’s challenge is worth preserving: give the same case to 10 senior lawyers and they may start by arguing. Law is not 1+1=2, and it is hard to judge an answer as directly as whether code runs.
- Joey’s answer was to build in layers, starting with stable, measurable tasks. In-house lawyers continually review NDAs and contracts, accumulating yes-or-no judgments for different scenarios in playbooks. Those judgments can then guide improvements to post-training, fine-tuning, RAG, or other methods.
- More complex rubrics are developed with legal experts across different fields. They may work in legal, research, sales, or forward-deployed roles. Hiring also depends on whether customers are willing to talk with them and whether their understanding reflects the leading edge of legal practice.
- An important researcher skill is drawing out expert judgment and enabling experts to contribute. Some benchmarks and evaluation engines are even written by lawyers using coding agents, because lawyers are better positioned to know what is right and wrong.
11. Scaling annotation takes both people and customer feedback
- Internally, the team first defines the direction, sample queries, results, and evaluation criteria. Once the first 10 or 50 samples are good, taking the dataset from 50 to 500 and then 5,000 becomes a process problem. External vendors can help with annotation and scoring; Joey cited Mercor as an example.
- Language models can evaluate some things, but Joey does not advocate trusting them completely. Otherwise, the model itself becomes a bottleneck, and the training ceiling may be constrained by the evaluation ceiling.
- Asked for an example of an evaluation that looked good but produced a feature users did not like, Joey could not think of a specific one. He emphasized updating datasets and evaluations based on user feedback, so customers see “a Harvey that keeps getting better.”
12. Public model scores sit on top of years of failed attempts
- Joey said benchmarks can first help clarify model performance. If the best open-source model already beats OpenAI or Anthropic on legal evaluations, post-training it could produce a better model. He called the base model’s performance a potential floor, but did not say post-training is guaranteed to improve it.
- Raymond posed a hypothetical: OpenAI scores 80, Kimi K3 scores 78, and Harvey wants to train to 82—what if it cannot get there? Joey acknowledged performance could also go from 78 to 75, and a trained model may not be deployed to customers.
- Joey said Harvey has been experimenting with pre-training, post-training, and fine-tuning since its early days, with plenty of failures. What matters more is whether the team can apply what it learned next time, while weighing quality, latency, cost, reliability, explainability, and lawyer satisfaction. Customers do not want something cheap that does not work well.
13. Engineering improves the experience first; in-house models test the margin potential
- If there were only $100 left, should Harvey optimize the harness or try new post-training? Joey would prioritize benefits users can feel immediately. There are still plenty of low-hanging fruit in engineering, while models remain a long-term experiment.
- His example: cut document search from 2 seconds to 200 milliseconds, raising the number of files searchable in a minute from 30 to 300. Then go from one search at a time to 10 in parallel, reaching 3,000. Giving the agent more material could produce better results regardless of which model it uses.
- Investment in models also bears on how Harvey will make money in the future and whether profit margins can support a public listing. Joey thinks Harvey can consider that question at this stage, but for many early-stage startups it may not be the best use of capital.
14. Model choice reflects customer requirements and preferences
- Joey said OpenAI, as an investor, does not require Harvey to use only its models; it mainly provides a channel for discussion and feedback. Model choice depends on customer needs, local availability, and whether Harvey can ensure access, capacity, reliability, and performance.
- Some users want Claude as the default. Joey said that under Anthropic’s policy, data is retained by Anthropic for 30 days: “We can’t see it, but Anthropic can.” Law firms and corporate legal departments need to accept that condition.
- Raymond guessed that in-house models still account for a relatively small share of usage, with most usage on OpenAI and Anthropic models; Joey said “yes.” He also said he did not know the exact usage and could not disclose it even if he did. Anthropic models can sometimes be verbose in long explanations, but lawyers may want to know why a change was made.
15. Shared agents let teams work from the same context
- Joey said he first built a sandbox-agent prototype around February this year. It ran coding-agent harnesses such as Claude Code, Codex, Cursor, and Devin in the cloud, enabling more parallel workflows without being limited by the hardware on any one person’s computer.
- He does not want a manager to ask a question and 5 engineers to run agents locally, then paste back answers that are hard to verify. Instead, the team could enter the same agent runtime or conversation context, tell the agent their ideas, and let it coordinate and produce a result.
- Joey imagines a product where multiple people can tag the agent in Slack, see the full reasoning and tool-call trace, and agree on the result together. Choosing a harness employees already use also accounts for familiarity and trust.
16. Permission boundaries determine whether teams can trust shared agents
- The challenge is distinguishing each person’s permissions, the materials they can access, and what can be shared in public versus private channels. Joey needs to work with security and IT on the boundaries and ensure each agent follows the relevant access rules.
- For example, if internal policy requires 2 people to review code before it goes to production, the agent cannot bypass that rule. The team also needs to guard against edge cases such as one person getting another person’s agent to change code and submit it directly.
- Joey’s trust test is specific: if an employee privately chats with an agent today, they need to believe that “even tomorrow, their boss won’t be able to access it” (“明天老板也拿不到”). If people worry someone else might see the content, they will stop sharing information.
17. Harvey is prioritizing design partners who can use the product well
- Raymond suggested selling to large firms through SLG and using a Figma-like PLG model for smaller firms. Joey replied that smaller firms may want something Harvey does not offer, or may not care about features it already has, making the product’s value harder to convey.
- More importantly, self-serve access could lead users to overtrust the results and assume Harvey is responsible for them. Joey said legal consequences are more serious than those in code, so Harvey should currently focus on design partners with rules in place and the ability to manage outcomes: “They trust Harvey, and Harvey can trust them” (“他们信任Harvey,Harvey也能信任他们”).
Verification Notes
- The transcript’s opening calls Harvey’s internal cloud agent platform “Spire.” Later, Raymond calls the shared-agent tool “Spectre.” The transcript does not say whether these are the same product.
- Kimi K3 is Raymond’s characterization of Tanner’s base model at the time. Joey did not identify a single fixed base model for Tanner and said Harvey may train different models with different vendors.