Artificial Analysis: The Independent LLM Analysis House — with George Cameron and Micah Hill-Smith
Summary
Artificial Analysis has turned independence itself into the product: no one pays for public placement, while enterprises buy decision reports and AI builders buy private benchmarking. The company now has just over 20 people and two customer groups, pairing free public data with subscriptions and bespoke work. swyx’s shorthand is the “presumptive new Gartner of AI,” but the commercial firewall is the thesis: “There’s no use doing what we do unless it’s independent AI benchmarking.”
Reliable evals cost far more than firing off a question set because prompts, output parsing, answer ordering, sampling variance, endpoint manipulation, and repeated runs can all move rankings. Artificial Analysis targets roughly ±1 point at 95% confidence for its Intelligence Index, making its real costs higher than the one-repeat figure shown publicly. Private lab endpoints are checked through a “mystery shopper policy” using unidentified accounts, because the served model might differ from the endpoint supplied for testing.
The Intelligence Index has to keep changing because yesterday’s frontier tests are now saturated and successful benchmarks rapidly become optimization targets. V1 tasks such as HumanEval-style Python functions would be close to trivial for many current models; V3 blends 10 datasets spanning Q&A, agents, long-context reasoning, and use cases. The core warning is that “things that get measured become things that get targeted,” so improving competition-math scores need not equal broader intelligence.
The frontier has shifted from an apparently unassailable OpenAI lead to a market that Micah-Hill Smith described as “strictly more competitive every quarter.” At recording, Gemini 3 Pro High led, followed by Claude Opus 4.5, GPT-5.1 High, and Kimi K2 Thinking. DeepSeek’s decisive signal arrived with the open-weight DeepSeek V3, described as a 61.1B MoE, on Boxing Day the previous year—before R1 made the broader world pay attention.
Artificial Analysis is expanding “intelligence” beyond percentage-correct scores by explicitly penalizing hallucination and rewarding “I don’t know.” Its Omniscience metric runs from -100 to +100, subtracting a point for a wrong factual answer; Claude models showed the lowest hallucination rates, while general intelligence had little correlation with knowing when to abstain. The trade-off is contextual: on Critical Point’s research-level physics problems, where the top score was only 9%, researchers deliberately raise temperature because exploratory hallucination can be useful.
Agentic performance depends as much on the harness as the underlying model, creating a new competitive layer above model APIs. GDPval-AA turns 44 white-collar tasks with spreadsheets, PDFs, presentations, audio, and video into a model-agnostic benchmark; every tested model performed better in Artificial Analysis’s harness than in its corresponding consumer chatbot. That minimalist harness—web tools, filesystem, code execution, context management, and image viewing—was released as Stirrup, with the advice to “let the models work as long as they want.”
The infrastructure paradox is that GPT-4-level intelligence is at least 100× cheaper, yet total inference spending can still rise by orders of magnitude. Larger sparse frontier models, reasoning tokens, long agentic workflows, and repeated turns overwhelm declining unit costs; one startup was reportedly spending $5,000 per employee on coding agents alone. George Cameron expects both trends to continue: another order-of-magnitude reduction in comparable intelligence cost and another order-of-magnitude expansion in feasible consumption.
The next efficiency battleground is not “reasoning versus non-reasoning” but whether models spend the right tokens and turns on each problem. Earlier in the year, reasoning models averaged 10× the tokens per Intelligence Index query; now model-to-model token efficiency itself varies by more than an order of magnitude. In τ²-bench Telecom, GPT-5 could be cheaper overall than smaller open models despite pricier tokens because it resolved the customer’s problem in fewer turns.
Deep dive
1. A benchmarking side project became an independent data business
Smith’s origin story begins with a 2023 legal-research assistant: as he optimized each stage for accuracy, performance, and cost, “the more each bit of what you’re doing ends up being a benchmarking problem.” Nobody was independently measuring those trade-offs across models and providers, so he and Cameron built what they themselves needed.
The first version was genuinely a side project: a purchased domain pointing at a Vercel deployment, publicized with a tweet, with only hundreds of dollars of personally funded evaluation spend. It became more useful as model releases accelerated and developers needed to compare not only quality, but output speed, provider choice, and price.
Two years in, Artificial Analysis has just over 20 people and two commercial tracks. Enterprises subscribe to standardized Benchmarking Insights reports—one compares serverless inference, managed deployments, and leasing chips for self-hosting—while enterprises and AI vendors can commission custom private benchmarking.
Joining AI Grant batch four supplied mentors and connected the company with startups pushing the limits of AI. swyx questioned whether frontier startups represented the real customer, but Cameron and Smith argued those companies are archetypal power users: they combine many models, switch frequently, and optimize different application stages for accuracy, speed, and cost.
2. Benchmark rankings rest on an unglamorous stack of controls
The founding methodological choice was to rerun evaluations rather than simply take lab claims. Labs prompted nominally identical benchmarks differently; in the extreme, “you can put the answer into the model.” For Gemini 1.0 Ultra, Google reportedly—and Micah-Hill Smith said “I think”—constructed 32 never-published chain-of-thought examples for every MMLU topic while seeking a number above GPT-4.
Output parsing creates its own judgment call. A model may reason correctly but violate the requested format: if the benchmark isolates reasoning, an LLM answer extractor can recover the result; if instruction-following is part of the capability, the malformed answer may deserve zero. Modern models usually follow demonstrated formats well enough for a regex.
Multiple-choice order bias, temperature, dataset size, and stochastic reasoning can produce large variance. Artificial Analysis repeatedly runs candidate evals while developing them, choosing repeat counts intended to keep the composite Intelligence Index within roughly ±1 point at 95% confidence. Its published cost assumes one repeat, so actual internal evaluation spending is materially higher.
swyx’s database-industry concern was whether lab-supplied endpoints could be tuned or differ from public service. Artificial Analysis answers with a “mystery shopper policy”: unidentified accounts run both intelligence and performance tests without the labs being able to identify them. Competitive self-interest helps—the labs want assurance that rivals cannot manipulate the benchmark either.
3. Benchmarks must evolve because measurement changes model behavior
Cameron’s largest concern is conceptual rather than fraudulent: once an eval drives launches and headlines, researchers rationally optimize against it. Models are now exceptional at competition mathematics, yet that capability maps only imperfectly onto economically useful coding agents or general work. A rising score can therefore outrun generalized improvement.
The current Intelligence Index combines 10 datasets into what the guests call the “best single number” for model intelligence, while retaining underlying charts for narrower decisions. It spans important Q&A benchmarks, a couple of agentic datasets, Artificial Analysis’s long-context reasoning work, and other use-case-focused evaluations.
V1 would now be saturated by many models, including relatively small ones; writing HumanEval-style Python functions is no longer frontier behavior. V2 and V3 became harder, covered more use cases, and moved closer to developer needs. The next priorities are agentic work, economically valuable tasks, and persistent weaknesses such as reasoning across long contexts.
4. The model market became decisively multi-polar
The recording’s leaderboard was already destined to date quickly: Gemini 3 Pro High ranked first, followed by Claude Opus 4.5, GPT-5.1 High, and Kimi K2 Thinking. swyx joked that a future listener would say “how cute”; that rapid expiry is itself evidence of the release cadence Artificial Analysis now tracks.
Zooming out produces the more durable finding. Before o1 and Claude 3.5 Sonnet, reasoning models and coding agents were not established categories; earlier still, OpenAI looked “untouchable for well over a year,” raising the possibility that everyone else would merely consume a few frontier labs’ APIs. Smith described the resulting market as “strictly more competitive every quarter.”
Cameron remembers evaluating DeepSeek V3 result by result on Boxing Day in New Zealand. The described 61.1B MoE architecture became Artificial Analysis’s best open-weights model at that point and showed DeepSeek was no longer one player among many. R1’s reinforcement-learning follow-up generated the wider shock, but “the groundwork” was the exceptionally strong, completely open-weights base model.
5. Omniscience makes abstention an evaluated capability
Most AI benchmarks reward percentage correct, implicitly telling models to attempt every question. Omniscience changes that incentive for factual knowledge: its score ranges from -100 to +100, subtracting one point for each incorrect answer. In this context, Cameron and Smith argue it is “strictly more helpful to say I don’t know” than to fabricate a fact.
The first results separate factual knowledge from abstention behavior. swyx noted that Haiku hallucinated less than Sonnet, which hallucinated less than Opus, and suggested this might quantify some previously “vibes”-based user preferences. Across families, smarter models were not strongly more likely to recognize what they did not know.
Gemini 3 Pro made a large jump in factual accuracy over Gemini 2.5 Flash and 2.5 Pro without a comparable hallucination-rate improvement. Omniscience accuracy also tracks total parameter count more closely than any other Artificial Analysis metric, while the hallucination differences appear more connected to post-training choices.
swyx resisted treating any proprietary dataset as humanity’s canonical hallucination number. The guests agreed: this is specifically the “AA Omniscience hallucination rate,” with 10% of the factual questions public and the rest held out against contamination. Their aim is a useful dimension, not “humanity’s last hallucination” benchmark.
6. Hard science shows that hallucination is sometimes the feature
The guests explicitly reject a universal penalty on speculation. In coding, ideation, and open research, users may want the model to “have a go.” Their best counterexample is Critical Point, an academic dataset of research-level physics questions resembling FrontierMath, where the leading model scored only 9%.
Critical Point’s creators reportedly turn temperature as high as practical when using models as physics thought partners, deliberately seeking broader exploration. That makes hallucination useful rather than defective—the correct evaluation policy depends on whether the task needs reliable factual recall or novel candidate ideas.
This also illustrates Artificial Analysis’s broader sourcing model. It will build evals internally, partner with academics and AI companies, and independently run strong datasets originally released by labs. Cameron and Smith are not “obsessed” with inventing everything internally; with appropriate disclosure, they are comfortable partnering while preserving the independence of the benchmarking.
7. GDPval-AA treats office work as a long-running agent task
OpenAI’s GDPval data set contains 44 broad white-collar tasks with detailed instructions and artifacts such as spreadsheets, PDFs, presentations, audio, and video. Smith’s useful framing is “quite hard take-home exam tasks”: they capture meaningful work products, though not the full, messy context of an actual job.
Artificial Analysis turned that dataset into a benchmark runnable on any model. Its reference harness lets agents use web search, browsing, code execution, and the filesystem; Gemini 3 Pro Preview grades extracted visual and textual outputs against explicit criteria. The team says it tested the evaluator comprehensively against human preferences; notably, Gemini 3 Pro itself did not perform especially well as a GDPval-AA test-taker.
Because a marketing video or slide deck has no simple correct answer, the benchmark uses relative Elo rather than percentage correct. swyx wanted a contracted human in the pool so 50% could retain OpenAI’s “normal human” interpretation; the guests preferred Elo for scaling across many models and noted that humans and agents solve these tasks differently.
Every tested model performed better in Artificial Analysis’s harness than through its corresponding consumer chatbot; Claude Opus 4.5 was the cited example. The reference agent can run for up to 100 turns, while consumer products face different use-case and cost constraints. “We let the models work as long as they want, basically.”
8. Stirrup makes the agent harness a first-class layer
The GDPval result suggested that Artificial Analysis had accidentally built a strong generalist agent, not merely an evaluation wrapper. Stirrup, released on GitHub the day before recording, is deliberately minimalist: context management, web search and browsing, code execution, filesystem access, plus an image-viewing tool where visual artifacts must enter context.
The intended workflow is pragmatic: clone the small codebase, then have a coding agent adapt it for the target task. Cameron and Smith see increasingly capable models doing better with a compact toolset and control over their own workflow than with elaborate frameworks that dictate each step.
Consumer agents remain meaningfully distinct products because their tools and connected data sources have diverged. Smith can already have models query Supabase read-only through MCP, run SQL analysis, make charts, and consult Gmail or Notion—but his caveat survives intact: the stuff “barely works,” and he has still never sent an email drafted by a chatbot.
9. Openness is more than permission to download weights
The Openness Index adds a missing dimension to license tracking: “how much is disclosed about how it was made.” It scores access to pre-training and post-training data, permission to reuse that data, methodological transparency, and training-code availability alongside the rights attached to model weights.
The current maximum is 18 points, with AI2’s OLMo 3 32B Think leading at recording. The most valuable chart plots openness against intelligence, making the desired hill-climb explicit while exposing an uncomfortable slope: the most reproducible models are not necessarily the smartest.
swyx challenged the point system because a small data disclosure and a large open-weights contribution may receive categories that do not represent their different material impact. The guests conceded that impact weighting is hard; they optimize for rules objective enough that observers cannot reasonably dispute which bucket a release occupies.
For commercial users, the simplest high score comes from a standard OSI license such as MIT or Apache 2: “the box is just checked.” Lower categories capture attribution requirements or restrictions on commercial use. NVIDIA’s Nemotron work received special credit because its models and synthetic-data contributions support substantial downstream activity.
10. Cheaper intelligence can produce vastly higher infrastructure spend
Artificial Analysis tracks cost by intelligence tier rather than asking whether one flagship API cut its price. Its conservative figure is that GPT-4-level intelligence now costs over 100× less than GPT-4 did at launch; swyx argued Amazon Nova could make the decline closer to 1,000×.
The apparent contradiction is that users can simultaneously spend much more. Small models inherit yesterday’s capability, but frontier demand moves to larger models; reasoning adds output tokens, and agentic systems multiply both input and output across long workflows. A cited startup was spending $5,000 per employee on coding agents alone—high, but not inconceivable if utilization is productive.
Hardware economics depend on workload, target user speed, and throughput per GPU: serving faster costs more. For large sparse models, Cameron believes Blackwell delivers “a lot better than two or three x” over Hopper, with future NVIDIA generations lowering cost per token while enabling larger models and many more generated tokens.
swyx proposed that active parameters might have fallen from roughly 25% toward 15% and asked whether sparsity was nearing a floor. Cameron said he was not sure that a 5% floor was obvious; the GPT-OSS models were cited at about 5% active, and he thought 2% or 3% might be possible. The guests also noted that benchmark performance often correlates more with total than active parameters. They expect another order of magnitude on both cheaper intelligence and feasible consumption.
11. Token and turn efficiency replace the reasoning/non-reasoning binary
Earlier in the year, the average reasoning model used 10× the tokens per Intelligence Index query as the average non-reasoning model. That clean distinction has broken down: reasoning strength is adjustable, and token efficiency now differs by more than an order of magnitude across models.
The desired behavior is conditional computation—few tokens for easy questions, many for hard ones. Artificial Analysis has internally tracked the correlation between question difficulty and token use and found models improved during the year. In long agents, that adaptation compounds across every step.
Fewer turns can dominate cheaper tokens. On τ²-bench Telecom, GPT-5 could cost less end to end than smaller open models despite charging more per token, because it resolved the customer’s query sooner. “Number of turns” is therefore likely to become a headline efficiency metric alongside price and output-token count.
Multi-turn benchmarks are operationally difficult, not conceptually optional. GDPval-AA already allows up to 100 turns, but infrastructure must execute that workload quickly across hundreds of models whenever a launch arrives. Cameron and Smith’s stance is simple: “You can do it. We’re putting in the work.”
12. V4 will fold reliability and agency into intelligence
The planned Intelligence Index V4 will incorporate GDPval-AA for general agentic performance, Critical Point for unsolved research physics, and AA-Omniscience, including hallucination rate. Weighting incompatible score types is the hard part; each major version is a deliberate reset, followed by no score drift within that version.
The guests said a high number of τ-bench tasks have potentially been impossible, while they were reasonably confident that the earlier τ-bench Telecom version was sound. Its remaining issue may simply be that models have become very good at it.
Beyond text, Artificial Analysis already benchmarks speech, image, video, and hardware. Its video arena pre-generates outputs, then gathers preferences; users can propose neglected categories and prompts. swyx’s request was workhorse infographics rather than more art—the broader lesson being that measurement can redirect effort toward underserved industrial uses.
Asked what changes next year, Cameron gave the intentionally “boring answer”: most chart lines will continue in their established directions until the products feel qualitatively different. Separately, swyx argued that demand for greater AI intelligence will be “insatiable,” asking whether any manager would refuse a button that made a colleague—or themselves—smarter and better at the job.