The Future of AI Avatars and DeepFake Technology w/ Joshua Xu & Steve Brown | EP #161
Summary
- HeyGen’s commercial bet is that video production becomes a software-scale function, not a camera-bound craft. Joshua Xu says fewer than 1% of people can create great video today and predicts businesses will make “100X more videos,” turning emails and text into video as AI removes camera friction and is expected to solve editing soon. HeyGen’s original 2020 mission said it plainly: “replacing the camera.”
- The larger opportunity is to make video the human interface for AI labor. Xu’s progression runs from typed chat to voice and then video meetings, where agents can operate across languages and hundreds or thousands of simultaneous sessions. When Diamandis asked when his envisioned 20-agent virtual marketing workforce would be available, Steve Brown answered: “Now.”
- The production proposition combines speed, localization, and lower-cost iteration. A HeyGen promotional clip claims avatar videos can be made “10 times faster than traditional production,” in 4K and across 175 languages; Xu says roughly two minutes of training footage personalizes facial movement, expression, and body language. Diamandis illustrated that pricing, naming, and other product elements could then change without another shoot.
- An avatar is the presentation layer; an agent adds knowledge, judgment, and action. Brown’s shorthand is “Avatar is AV, audio visual. Agent is agency,” with books, interviews, documents, values, life stories, and anecdotes supplying the brain. Diamandis also imagined independently named agents appearing in Slack, email, or video calls.
- Consent and moderation are strategic requirements, not peripheral features, once realistic identity becomes programmable. Xu says every HeyGen avatar requires first-party video consent and that the company has “never compromised on that consent”; AI and human moderators also restrict hate speech, fraud, and political campaigns. Diamandis’s practical countermeasure for cloned emergency calls is a private family password.
- Brown’s medical-agent demonstration shows the upside of cross-silo pattern recognition, while remaining a single personal case rather than clinical validation. An agent reviewing his earlier records connected low immunoglobulins, anemia, elevated ferritin, and weight loss to a possible plasma-cell disorder before specialists had done so; Brown had been diagnosed with multiple myeloma. His broader thesis: differentiated agents can search the same fragmented data from many perspectives, and Brown says they “might also be saving your life.”
Deep dive
1. HeyGen began as a bet to replace the camera
Xu dates HeyGen to 2020, when, he says, the term “generative AI” was not yet in use. While working on AI-enhanced camera experiences at Snap, he saw the possibility of generating something nonexistent that still felt highly realistic.
The founding mission was “replacing the camera.” Mobile cameras expanded creation over the previous 15 years, but Xu saw camera shyness and production expense as persistent barriers to business storytelling; later, Diamandis illustrated the related burdens of lighting and reshoots.
His projected endpoint is “visual storytelling for all”: fewer than 1% of people can currently make great videos, while every business could eventually produce “100X more videos” and turn routine emails, images, and text into engaging video.
2. Video is becoming the interface for parallel AI labor
Xu’s digital twin demonstrated prerecorded delivery without a camera or crew, speech in over 170 languages and dialects, and real-time conversations embedded in a website, product, or platform.
Xu’s interface roadmap moves from typing to voice and then to video meetings with agents. The vision is not merely watching generated clips, but interacting face-to-face with an AI.
Diamandis imagined an identity-rich agent that knows his work well enough to return the same answers he would, then sending 1,000 copies to conferences. Xu extended the idea: those agents could cross language barriers and run “hundreds, thousands of different session[s]” simultaneously.
When Diamandis asked when a 20-agent virtual marketing workforce would be available, Brown answered: “Now.”
Diamandis illustrated an immediate use case: rapid iteration. Xu has created more than 300 avatars over four years, yet says filming still demands preparation, good lighting, and sometimes one or two reshoots; an avatar can revise a pitch whenever pricing, naming, or positioning changes.
3. The avatar supplies presence; the knowledge layer supplies agency
Xu distinguishes an AI agent as an assistant that can perform not only an initial action but also a sequence of actions, while a digital twin enables access to the person it represents. Brown’s shorthand is compact: “Avatar is AV, audio visual. Agent is agency.”
A convincing face creates emotional connection, but granting agency requires a brain that knows the user and has earned trust. HeyGen describes its system as a “human-centric video model,” learning not only a face and voice but how an individual’s gestures, expressions, muscles, and body language correspond. Roughly two minutes of training footage provides that personalization.
Documents, keynotes, and interviews become an LLM knowledge layer behind the twin. For people without a library of published work, Brown is building interview agents that elicit values, purpose, life stories, and personal anecdotes—potentially the “hundreds of pages” contained in a biography.
The product clip advertised 4K output, 175 languages, and creation “10 times faster than traditional production.” At the event, one attendee said capture took “literally…four minutes”; his avatar was processed overnight and spoke another language in his own voice.
4. Consent is the boundary between an avatar and a deepfake
Diamandis warned that a realistic FaceTime caller may impersonate a relative and request bail money or a wire transfer. His practical advice is to establish a family password, and to preserve living parents’ or grandparents’ video, voice, and stories while possible.
Xu says every avatar requires first-party video consent and that HeyGen has never waived it, regardless of the requester. A combined AI-and-human moderation system also prohibits hate speech, fraudulent uses, and political campaigns on the platform.
5. A medical agent exposed the value of searching across silos
Brown’s case began after the Los Angeles fires destroyed his dream home and displaced him to Palm Springs. An emergency-room visit for stomach pain led to eight days of testing and a multiple-myeloma diagnosis, after earlier specialists had found nothing and suggested stress, overwork, or depression.
Asked to inspect only the earlier records, Brown’s “Hippocrates” agent cited low IgA and IgM, low gamma globulins, mild anemia, elevated ferritin, and unexplained weight loss. It suggested a possible plasma-cell disorder and recommended discussing bone-marrow evaluation with a physician. Brown said the agent found the pattern even earlier than the eventual diagnosis.
In a follow-up, the agent named a serum free light-chain assay measuring kappa and lambda chains, a definitive bone-marrow biopsy, and 24-hour urine protein electrophoresis for Bence-Jones proteins. Brown said his light chains fell by more than 50% after two weeks of chemotherapy and daratumumab immunotherapy; the agent called that encouraging while advising continued oncologist monitoring.
Brown’s conclusion is that expanding medical knowledge pushes specialists into narrower silos, while “AI is not siloed.” Beyond this single “god agent,” he is working on 100 differently oriented medical agents whose distinct viewpoints become alternative search strategies through fragmented health data. This remains Brown’s personal demonstration, not clinical validation.
Verification Notes
- In the Hippocrates exchange, the first response lists low gamma globulins, while the follow-up calls the pattern “hypergammaglobulinemia”; the transcript does not resolve this inconsistency.