FFmpeg: The Incredible Technology Behind Video on the Internet | Lex Fridman Podcast #496

Summary

• FFmpeg and VLC are civilization-scale infrastructure built with remarkably little conventional capital: Fridman cites FFmpeg in more than 90% of video workflows and at least 6.5 billion VLC downloads. The same stack serves home movies, YouTube, Netflix, browsers, televisions, broadcasters, and spacecraft, creating a rare open-source flywheel in which every additional codec makes the whole platform more valuable. Kunhya’s formulation is the strategic one: VLC and FFmpeg are a “binary star system,” not competitors.

• Video’s core economic problem is exchanging compute for bandwidth while preserving what human eyes and ears perceive, not reproducing the source like a ZIP file. Audio may compress roughly 10x and video 100x–200x, with each codec generation delivering about 30% more compression at comparable quality while potentially requiring one or two orders of magnitude more encoding compute. That asymmetry works because content is often “encoded once” but distributed to millions of viewers.

• Handwritten assembly remains an enormous source of operating leverage where code runs billions of times, despite the software industry’s faith in compilers and auto-vectorization. Kunhya says dav1d contains about 30,000 lines of C and 240,000 lines of assembly—versus roughly 100,000 assembly lines across FFmpeg’s codecs—and cites individual functions running 62x faster than C. With an estimated 3 billion devices decoding AV1, including roughly 30% of Netflix and 50% of YouTube video as stated, “every cycle matters.”

• The projects’ durable moat is their social contract: permissive access, demanding review, broad hardware support, and a refusal to monetize by betraying users. Kempf turned down offers worth tens of millions because they involved toolbars, spyware, search hijacking, or ads; his test was whether he could “go to bed at night and be happy” with the decision. LGPL licensing still permits commercial products around FFmpeg and libVLC, but modifications to those components must flow back.

• Large technology companies frequently treat volunteer projects as vendors without paying vendor economics, creating a growing infrastructure and security liability. The episode’s sharpest example is AI-generated security reporting against obscure codecs: discovery scales cheaply, but volunteers absorb triage, remediation, publicity, and fixed 90-day deadlines. Public pressure produced patches, repair bounties, and more donations, yet Kunhya says FFmpeg’s donations still did not fund even one full-time developer.

• Open source here is simultaneously a global talent market and an elite engineering school whose only admission credential is excellent code. Teenagers have written thousands of lines of production assembly, contributors have worked from war zones, and maintainers explicitly reject corporate pedigree: “Maybe you’re a dog. I don’t care.” The bar stays high because only roughly 10–15 core FFmpeg developers and six to eight VLC contributors may inherit code submitted by thousands of short-term participants.

• Trust comes from architecture and verifiability rather than promises: VLC has no telemetry, rejects government backdoors, and builds releases on offline systems with layered signing. The remaining risk is distribution and dependency code—fake VLC installers have delayed malware activation, while a real media process invokes hundreds of plugins, FFmpeg, and GPU drivers. VLC’s response is process-level sandboxing so a decoder or demuxer can fail without gaining access to the user’s documents.

• The next investable edge moves from passive playback toward synchronized, ultra-low-latency machine control and richer sensory media. Kempf’s Kyber sends video, audio, commands, and synchronized sensor data over one QUIC connection; Kunhya reports seven-millisecond Windows-to-Windows or Windows-to-Mac glass-to-glass latency against a four-millisecond goal. Beyond AV2, the same architecture can extend to point clouds, XR, haptics, depth, odors, and eventually neural streams: multimedia is “several streams for the human senses.”

Deep dive

1. VLC’s absurd compatibility begins with treating every input as suspect

• Kempf’s opening proof is not a fashionable streaming format but VHS: connect a capture card through SCART or RCA, and VLC can play or record the signal while some modules directly control VCR camcorders. The project also spent a summer implementing DVD-Audio, including its custom encryption, after commercial support had largely disappeared.

• FFmpeg’s long tail reaches a ten-second opening from an old Star Wars game: someone implemented its peculiar codec and ensured that one sequence on one disc decoded bit-exactly. The archival ethic is implicit—obscurity is not a reason to abandon data when a contributor cares enough to recover it.

• A VideoLAN contest deliberately created hostile media: an MKV changed resolution, aspect ratio, and rotation every frame; another “video” consisted of black frames beneath animated SSA subtitles; another file was described as simultaneously a valid ZIP and MP3. VLC played the stupid files because pathological specimens exercise the same resilience needed for damaged real-world media.

• The traffic cone became a distribution asset precisely because it is “ridiculous and absurd.” Kempf says 25% of VideoLAN website traffic came from searches for “cone player,” while an April Fools’ replacement generated about 10,000 protest emails. A distinctive meme became culture—and a globally recognized identity a tiny nonprofit never had to buy.

2. Pressing play activates a stack of lifetime-sized engineering problems

• Kempf maps the first stage from an address—HTTP URL, local path, DVD, or device—to a byte stream supplied through the operating system. A demultiplexer then identifies container structure and separates compressed video, audio, and subtitle tracks before any pixels or sound samples exist.

• Kunhya says the player must probe each video stream before choosing a decoder because as many as 45% of files may not be GPU-decodable. Codec variants and GPU vendors expose different capabilities; compatible material goes into a hardware “black box,” while everything else falls back to software.

• Software decoding first removes entropy coding such as Huffman or arithmetic coding, reads syntax for intra prediction, reconstructs the prediction residual from quantized frequency-domain coefficients, performs the inverse transform, and returns that residual to the spatial domain. Only then does compressed syntax become raw image data.

• The final stage hands images to the graphics system and raw audio to the sound device, which produces the analog signal for speakers. Kunhya’s warning prevents the neat diagram from looking easy: “Every sentence is someone’s lifetime’s work,” with books, academic specialties, and thousands of practical edge cases behind each arrow.

3. Compression succeeds by degrading signals in the way humans notice least

• Kempf’s distinction is load-bearing: media compression is not ZIP. A lossless archive tries to reproduce identical data, whereas common audio and video codecs intentionally discard information—roughly 10x compression for MP3-like audio and commonly 100x or 200x for video—while optimizing the degradation for human perception.

• Video processing therefore moves from RGB into YUV-like representations that separate luminance from color information. Because vision is more sensitive to brightness detail, chroma resolution can often be reduced enough to halve uncompressed size before later compression, yet most viewers do not consciously notice what disappeared.

• Codecs divide images into blocks, predict their contents, transform residual error with discrete-cosine-transform-like mathematics, quantize coefficients, and encode the result compactly. The familiar blocks in a badly encoded or damaged image are not arbitrary artifacts; they expose the grid on which much of the mathematical machinery operates.

• Each generation may provide approximately 30% lower bitrate for equivalent quality, but Kunhya adds the hidden cost: achieving that gain can demand one or even two orders of magnitude more encoding CPU. Better compression is thus an infrastructure trade—greater up-front compute against storage, network delivery, battery consumption, and repeated decoding.

4. A codec monetizes redundancy across space, time, and audiences

• Kunhya defines the codec’s job as removing spatial and temporal redundancy, often with far more compute devoted to compression than decompression. That asymmetry is economically rational when one expensive encode can serve many viewers, but it also constrains live encoding, interactive systems, and devices with small power budgets.

• Kempf’s concrete image is a camera panning past a cloud or an unchanged background: encode the object once, then tell later frames to reuse it. Searching farther back—perhaps across 200 frames of 4K video—can find better matches, but memory and compute turn theoretically unlimited reuse into a practical optimization problem.

• Modern labels such as AV1, AV2, and VVC describe collections of coding tools rather than one uniform technique. A Zoom-like stream may alternate between a PowerPoint screen and live video, requiring different prediction and compression tools; the encoder’s burden is selecting the best subset for each region and moment.

5. Containers organize tracks, but filenames routinely lie

• A container—also called a muxer—is the structure holding multiple tracks; demuxing separates them. AVI came from Microsoft’s earlier ecosystem, MOV from Apple and evolved toward MP4, while Matroska was designed in the open-source community as a more extensible, future-oriented container.

• Confusion between MP4 and H.264 is partly the industry’s fault. H.264 is also AVC and MPEG-4 Part 10, while MPEG-4 itself spans containers, audio, and multiple video specifications; in ordinary practice, Kunhya says an MP4 is H.264 plus AAC perhaps 99% of the time, making colloquial shorthand understandable.

• VLC and FFmpeg treat .mp4 as a hint, not ground truth. The extension raises the MP4 demuxer’s probing priority, but the software inspects the bytes and tries alternatives because files are frequently mislabeled, malformed, or constructed by tools that do not follow specifications cleanly.

• VLC’s refusal to trust input was inherited from late-1990s UDP streaming, where packet loss and damage were expected. The same philosophy made it popular during partial downloads: AVI metadata might sit at the file’s end, yet VLC would say, in Kempf’s paraphrase, “This file is broken, but I’m still going to try.”

6. FFmpeg turned professional media processing into a programmable commodity

• Kempf describes FFmpeg as low-level libraries for codecs, muxers, demuxers, and filters, plus command-line tools that compose them into pipelines. Its core libraries include libavcodec, libavformat, and libavfilter; it also integrates external work such as x264, libvpx, and other specialized components.

• The command-line interface is effectively “an actual language.” Simple jobs take one input, one output, and a format, but production commands can span thousands of characters or be generated programmatically; users now ask AI systems to generate them because the available filter graph and parameter space exceed what most people remember.

• Fridman’s own specimen joins intro, program, and outro clips; dips video to black; crossfades audio; burns styled captions into the picture; and accepts disparate input formats. Kempf compares the result to doing Adobe After Effects work from a shell—an unusually complete automation surface for moving images.

• Fridman argues that FFmpeg helped cause the podcast, streaming, and YouTube revolutions. Compression equipment that cost hundreds of thousands of dollars and occupied car-sized systems in the 1990s became software available to an individual on nearly the same technical footing as a large company.

7. Open-source licensing is the constitution of the community

• Kempf’s cheesecake analogy makes source availability concrete: open source gives away the finished cake, its recipe, instructions for constructing the oven, and legal permission to modify and resell the recipe. The license matters because software is not a few kitchen steps but billions of tiny instructions accumulated across many authors.

• He estimates that 2,000–3,000 people have contributed to FFmpeg over its life. They coordinate across borders, religions, languages, and employers around a narrow shared objective—making multimedia work—while disagreeing about almost everything except the license that permits them to collaborate, fork, and later recombine.

• Permissive licenses such as MIT and BSD allow broad reuse, sometimes with attribution; copyleft licenses require modifications to return under progressively stronger conditions, from MPL-like approaches through LGPL, GPL, and AGPL. Kempf calls that choice a “social contract” defining how commercial adopters and community contributors relate.

• FFmpeg and libVLC use LGPL for important library components: proprietary applications may link them without opening the entire product, but changes to the component must be returned. GPL remains appropriate for other VLC releases, while the iOS and Apple TV applications use MPL because Apple’s App Store terms complicate GPL distribution.

8. Relicensing exposed the human ownership hidden inside a codebase

• Moving libVLC from GPL to LGPL required consent from more than 350 copyright holders because VideoLAN did not demand copyright assignment. Kempf tracked old email addresses, visited workplaces, and even met a factory worker whose deceased son had contributed several useful lines; the conversation became about a life and a photograph, not administrative cleanup. “It’s important to do it right.”

• The change enabled libVLC inside proprietary games and mobile applications and supported a consulting business around integration. Yet it also demonstrated why FFmpeg cannot simply change its license to block cloud providers: thousands of contributors, including people who have died, retain rights in work whose influence may survive even after their original lines are replaced.

9. The projects judge identities by code because maintainers inherit the consequences

• Kempf’s meritocratic rule is categorical: “Maybe you’re a dog. I don’t care.” Nationality, employer, age, and credentials do not establish technical quality; a patch from an engineer at a famous company can still be rejected because the standard protecting the community is the code itself.

• The harsh bar reflects maintenance arithmetic. Kempf counts roughly five core VLC developers and 10–15 in FFmpeg against thousands of historical contributors; with perhaps a 1% chance that a contributor stays, the core team must maintain code after jobs, families, accidents, and changing interests take its author away.

• Linus Torvalds represents an extreme tone, though Kempf notes his strongest criticism usually targets subsystem maintainers who already know him. The desired distinction is between attacking a person and declaring that code is “crap”; difficult as the culture can feel, maintainability and correctness cannot be compromised in infrastructure used globally.

• Terseness also comes from volunteers reviewing patches after a full workday and from multilingual communication centered in Europe. English carries tonal subtleties many contributors never learned, so a sentence that sounds hostile to an American reader may be fatigue or literal translation rather than personal contempt.

10. VLC grew from students routing satellite television across a campus

• École Centrale Paris’s campus was managed extensively by students because an alumni nonprofit owned the site: students handled rooms, utilities, shops, radio, television, and networking. That unusual responsibility gave young engineers both the authority and necessity to build infrastructure rather than wait for an institution.

• Its 1980s token-ring network became intolerable when Doom and Duke Nukem made latency visible. Around 1994–1995, students seeking a faster network approached Bouygues; its CIO proposed one large satellite dish and decoder feeding roughly 1,500 students over the network instead of separate receivers.

• “Network 2000” streamed MPEG-2 transport, video, and audio across a 155-megabit-per-second ATM network. Its hacked prototype crashed after 45 seconds, but the demo lasted 40; memory leaked, so the team installed 64 MB rather than the normal 8 or 16. A supposedly disposable showcase became a foundation.

• Christophe Massiot and another student later created VideoLAN for network video and spent about three years persuading the school to open-source it, finally succeeding in 2001. Kempf joined in 2003, formed the independent VideoLAN nonprofit, and kept VLC alive in 2005 when only two active developers remained.

11. Refusing predatory monetization preserved VLC’s trust and survival

• Kempf confirms that he rejected tens of millions of dollars, more than once, but rejects the mythology that he opposes money. The offers involved bundled toolbars, spyware-like browser changes, search hijacking, or advertising—businesses that would damage users, provoke a fork, and plausibly leave the project dead three years later.

• His honest counterfactual is instructive: a legitimate service such as Netflix proposing a useful integration might have produced a different answer. The line was not commerce versus purity; it was earning money ethically versus exploiting distribution trust through mechanisms users did not knowingly request.

• The most difficult bidder argued that its “obscene” payment could finance another open-source project. Kempf recognized the moral laundering: “This doesn’t work like that.” Selling out would also appropriate work contributed by many others, while personally he needed to remain proud of the choice when he went to sleep.

12. AI security discovery became a denial-of-service attack on volunteer attention

• Kunhya’s criticism begins with a concession: Google is one of open source’s largest supporters and employs capable security researchers. The conflict arose when it used AI to generate FFmpeg vulnerability reports, gave volunteers a standard 90-day deadline, offered limited funding, and publicized the AI’s success before fixes were ready.

• The reported target could be an obscure 1990s game codec, yet the security process spoke as if every lock protected Fort Knox. Kunhya’s analogy is a hobbyist’s household padlock being picked with industrial-scale machinery, followed by an urgent demand that the hobbyist redesign it without the discoverer supplying money or construction labor.

• Automated reports were long, numerous, and uniformly alarming—“high, high, high, vulnerable, scary, scary, scary.” Another researcher rated an integer overflow that could make one pixel the wrong color at 7.5 severity; Kunhya argues this repeated escalation trains maintainers and users to ignore alerts, the security equivalent of crying wolf.

• Alex Strange’s quoted incentive critique lands because discoverers can receive bounties, conference prestige, names, logos, and publicity, while “Nobody is going to do any of this for you when you fix it.” After the dispute, Google began sending patches and created rewards for fixing issues, partially correcting the discovery-to-remediation imbalance.

13. Corporations often demand service-level urgency without buying service

• The Microsoft Teams example followed the XZ debate: a manager marked an FFmpeg issue high priority because it affected a visible Microsoft product. When volunteers suggested a long-term maintenance contract, Microsoft offered a one-time payment of only a few thousand dollars, despite its substantial corporate scale.

• Kempf’s boundary is simple: “We are not your supplier.” A public bug tracker is not a contracted vendor’s Jira, and there is no SLA unless a company buys one; the open-source program offices meant to teach that distinction often fail to connect product teams with the communities beneath their stack.

• Public pressure nevertheless worked. Donations rose substantially but, Kunhya says, still could not support one full-time FFmpeg developer; spicy posts also forced responses when VLC spent more than a year unable to update on Google Play and when VideoLAN lacked ordinary points of contact despite being a top-10 Windows application.

14. The FFmpeg account turns conflict into recruiting, education, and leverage

• Kunhya calls the online feuds a “rap battle” or WWE: attacks can be theatrical without becoming permanent personal hostility. The account’s purpose is not dunking alone but showing that FFmpeg is not Kubernetes with hundreds or thousands of paid developers—it is largely people working in basements after their jobs.

• The most productive provocation celebrates contributors whom the industry overlooks. “Teenagers have written more assembly in FFmpeg than Google engineers” points to people such as Daniel Kang and 16-year-old Ruikai Peng, who found and fixed issues without converting a three-day development bug into a branded CVE spectacle.

• Kempf says the tone occasionally went too far, including during the dispute involving Theo; he called him, calmed participants, and drew a line. Yet awareness of community-built open source rose dramatically, while prominent supporters including Tim Sweeney, John Carmack, Andrej Karpathy, and Fridman amplified the work.

15. Contributors arrive for video and stay for mastery, impact, and joy

• Kempf’s advice to prospective contributors is “work on something you love.” Movies, anime, subtitles, and the satisfaction of making an image appear attract people; a 16-millisecond frame budget then forces them to learn C, assembly, SIMD, CPU pipelines, memory hierarchies, I/O, and the discipline of real-time systems.

• FFmpeg becomes “the best school ever” because world-class reviewers expose every flaw in code that has nowhere to hide. The reward is unusually legible impact: a programmer can tell a grandmother, “I do this so that you can play video,” rather than describe an internal invoice portal nobody outside an employer will ever see.

16. VLC and FFmpeg compound each other like a binary star system

• Kunhya’s compact analogy is “VLC is to FFmpeg as Android is to Linux.” VLC depends on FFmpeg for decoding breadth; FFmpeg gains exposure to damaged and unusual files through VLC; developers overlap, donations have crossed project lines, and failures found in one environment strengthen the other.

• x264 makes the coupling commercial as well as technical. Kunhya estimates that more than 80% of many FFmpeg pipelines depend on the VideoLAN encoder, while he also says companies often adopted FFmpeg specifically to reach x264—the open-source implementation that made H.264 encoding exceptionally good.

• The dependency graph dwarfs each brand’s repository. Kempf says compiling VLC for Windows involves about 16 million lines of code, only roughly 1 million from VLC itself and perhaps 2 million from FFmpeg; FFmpeg in turn incorporates x264, Opus, and many other independent libraries.

• VideoLAN’s broader portfolio includes libdvdcss, libdvdnav, libdvdpsi, libbluray, libVLC, x264, dav1d, spatial-audio work, and checkasm. The ecosystem therefore has no single center: “A quick reminder that it’s FFmpeg inside” understates the reciprocal architecture through which projects supply capabilities, testing, reach, and contributors.

17. FFmpeg’s history is a succession of people solving impossible compatibility eras

• Fabrice Bellard supplied the founding concept. Kunhya characterizes the 2000s as Michael Niedermayer’s era: exhaustive support for DivX, Xvid, and innumerable MPEG-4 Part 2 variants at a time when Windows Media Player, RealPlayer, and separate codec packs divided the playback world.

• Native decoding replaced bloated proprietary players and suspicious DLL bundles, but the real work was Sisyphean. A Chinese CCTV system might implement one eccentric MPEG-4 ASP variation; supporting it without breaking every other variation multiplied that problem across millions of edge cases.

• Reverse engineering began with proprietary Windows Media and RealMedia work involving contributors such as Benjamin Larsson and Kostya Shishkov. Around 2008, H.264 maturity and high-definition video began another era, with specialists such as Paul Mahol and Kostya tackling codecs whose vendors had never supplied usable open implementations.

• VLC 1.0 arrived around 2009–2010 as this compatibility stack matured. Its promise—no codec pack, no spyware, just playback—was not merely interface polish; it was the consumer surface of years spent reconstructing incompatible proprietary formats and keeping every earlier decoder working.

18. Reverse engineers treat executable binaries as specifications

• Kostya Shishkov’s signature phrase was “binary specification”: documentation was optional because the executable itself disclosed what the codec did. Kempf describes him as borderline genius, capable of reconstructing 20–30 MB binary blobs when Kunhya estimates even one megabyte can represent roughly a month of work.

• GoToMeeting became a leading VLC request because old recordings required a proprietary decoder and otherwise could not be played by VLC. Kempf posted a bounty; Kostya returned after roughly two months with elegant code, jokes involving Kempf’s name, and support that could preserve those meetings on future ARM, RISC-V, or mobile systems.

• Kunhya outlines the archaeology: locate the decompression module, hook it, dump reference YUV output, open a disassembler, identify entropy decoding and transforms by pattern, then step machine instruction by instruction. For long periods nothing recognizable appears; one mistaken coefficient buffer can send the reasoning down the wrong branch.

• His CineForm work started with a lucky animated sample containing flat blocks and few complex tools. Additional samples revealed branches bit by bit until the implementation matched; a corporate archive of 100,000 files could become a “goldmine” for validation. Modern codecs’ bit-exact definitions require independent decoders to emit identical output for every sample.

19. FATE protects portability across a combinatorial hardware matrix

• FFmpeg’s Automated Testing Environment continuously exercises codecs, filters, compilers, operating systems, and instruction sets on volunteer-hosted machines. Its dashboard spans Apple Clang, GCC, Visual Studio, Intel compilers, macOS, iOS, tvOS, ARM, x86, PowerPC, RISC-V, and formerly rarities such as DEC Alpha.

• The matrix catches more than programmer regressions. Compilers sometimes miscompile valid C, and a one-bit or one-pixel decoder error can cascade across dependent frames into large visual corruption; a change that works locally may still fail with GCC 11 or on one old Apple target.

• Kempf’s portability burden extends from Windows XP through Windows 11, macOS 10.7 through macOS 26 as spoken, iOS 9 through iOS 26, Linux, BSD, Solaris, and OS/2. Supporting iOS 9 requires a “Frankenstein” mixture of old Xcode and SDK components to retain ARM32 output after Apple dropped it.

• That backward compatibility has a material user consequence: someone can keep an iPad 2 as a functioning movie player instead of buying new hardware. Kempf connects assembly and legacy support to a broader lost art—optimizing software sufficiently that working devices do not become obsolete merely because upstream toolchains moved on.

20. Assembly converts architectural knowledge directly into throughput

• Assembly expresses the processor’s own instructions rather than asking a compiler to translate higher-level intent. FFmpeg’s critical flavor is SIMD—single instruction, multiple data—where one operation can add a value across perhaps 16 elements at once, naturally matching blocks and vectors of video pixels.

• Kunhya reports common function-level improvements of 10x–50x, explicitly not 10%–50%; one function is cited as 62x faster than C. His company, Open Broadcast Systems, applies the FFmpeg ethos to live sports systems, including a handwritten assembly matrix converting every packed 10-bit video format into every other format across CPU generations.

• Kempf ties the renewed value of low-level work to hardware limits: demand for AI, video, memory, and networking is rising faster than single-core performance. More cores help only so far, whereas optimizing the small kernels executed billions of times produces “more power per dollar invested”—the same constraint-driven instinct visible in FP8 and FP4 quantization and the Microsoft Phi-1.5 example discussed.

• Expert programmers also “abuse the machine,” using cryptographic instructions for unrelated video operations or bypassing operating-system calling conventions inside a controlled library. These gains require knowledge of registers, cache traffic, instruction latency, ALUs, pipelines, and data layout that a compiler cannot infer from ordinary C abstractions.

21. dav1d is the episode’s proof that handwritten optimization still matters

• When AV1 arrived, even Alliance for Open Media participants argued that its complexity required hardware decoding. Kunhya and collaborators insisted software was essential during the years before hardware deployment and built dav1d around the motto “every cycle matters.”

• Kunhya’s scale comparison is stark: roughly 30,000 lines of C and 240,000 lines of handwritten assembly in dav1d, versus about 100,000 assembly lines across all FFmpeg codecs. A cited code breakdown—79.9% assembly, 19.6% C, and 0.5% other—became the provocative image of “what peak video codecs should look like.”

• The optimization targets perhaps 3 billion devices; Kunhya says about 30% of Netflix video and 50% of YouTube video is now AV1. dav1d could decode 720p with one or two CPU cores, converting a specification thought to demand dedicated silicon into software deployable across existing machines.

• The implementation detects capabilities at runtime and selects separate kernels for x86 generations, AVX, AVX-512, ARM64, Neon, SVE, SME, and RISC-V. Despite years of examples, critics still answer that compiler auto-vectorization or intrinsics should suffice; the rebuttal remains empirical: “It’s not even close.”

22. Assembly must be taught through real problems before its craft disappears

• Kunhya rejected instruction-by-instruction textbooks because they teach programming like grammar rather than conversation. His asm-lessons start from high-school mathematics, pointers, registers, and practical vector problems; contributors have studied them and submitted working FFmpeg patches, showing the craft need not remain blacksmith-like person to person.

• His aesthetic analogy is flying a Spitfire: “It’s you and the processor,” with the joystick physically connected to the control surfaces and the pilot pushing the aircraft beyond its designer’s expectation. LLMs understand more assembly than he expected, but scarce training data leaves them repeatedly producing non-bit-exact modifications even after being asked to fix them.

23. Rust improves new systems but does not erase mature C or unsafe assembly

• Kunhya values memory safety but sees “a very big Esperanto vibe” in claims that Rust alone will replace working infrastructure. A rewrite delivering 85% or 90% of coreutils is not equivalent because “the last 1% takes 99% of the time”; production adoption requires equal features, stable ABI, flawless testing, and broad compiler support.

• Kempf has written Rust modules for VLC and is building Kyber mainly in Rust. His positive case is greenfield parsing and networking: ownership and bounds checking make Rust “a better C++” for memory-sensitive new systems, particularly when the entire project can share one language and security model.

• His default rule remains “don’t rewrite.” Writing is an order of magnitude easier than reading; newcomers cannot see undocumented business logic or historical edge cases, rapidly recreate 80%–90%, then discover that the remaining behavior contains most of the accumulated wisdom.

• Rewriting dav1d’s C in Rust would not secure the handwritten assembly, which can still access arbitrary memory. Kempf’s preferred research direction is “secure assembly”: compile-time instrumentation, related to checkasm, that checks optimized kernels stay within allowed regions instead of destroying Rust’s guarantees at the performance boundary.

24. Forks can strengthen projects, but maintenance pressure can break people

• Fridman relays that FFmpeg split from Libav in 2011 amid governance, leadership, and development-process conflict rather than a fundamental codec disagreement. Some Linux distributions apparently switched sides for a period; over time FFmpeg incorporated much of Libav’s work, contributors returned, review practices stabilized, and the surviving project became a broader superset.

• Kempf treats forks as open source’s lawful method for changing a blocked status quo, citing GCC/EGCS and KHTML/WebKit/Blink. Fridman’s pushback is human rather than technical: even productive institutional conflict can exhaust the few individuals whose passion originally sustained the work.

• The larger burnout threat now includes “AI slop”—bad reports and patches that transfer review costs to maintainers. Kempf connects it to XZ, where attackers pressured a lone maintainer relentlessly, including at odd hours, until he surrendered commit access to someone positioned to compromise the project.

• Abuse can become physical. After VLC dropped new PowerPC support around 2009–2010, Kempf received a death threat containing powder and “PowerPC forever”; it was flour rather than anthrax, but police and his frightened family were real. He says the experience forged a thick skin no volunteer should have needed.

25. x264 beat sacred metrics by optimizing what viewers actually saw

• Earlier codec research privileged PSNR, derived from mean-squared error, even when its incentives spread small errors across an image and visibly blurred detail. x264’s developers instead judged output with their eyes on ordinary laptops, asking whether a person’s video looked good rather than whether a laboratory metric increased.

• Two innovations carried the shift: psychovisual rate distortion accounted for human perception and block energy, while adaptive quantization redistributed bits according to visual complexity. The ParkJoy sequence—runners, water, grass, foliage, reflections, and motion—showed that a lower-PSNR encode could look decisively better when those tools were enabled.

• Kempf offers a harsher specimen from Planet Earth: the frame fills with birds until millions of moving shapes overwhelm ordinary encoders. Anime added gradients, banding, unusual digital textures, layered Japanese readings, and elaborate subtitles; fan-sub communities lacking official distribution built Aegisub-like tools and supplied the feedback loops that improved open codecs.

• Professional users eventually took the same wager. Kunhya cites Warner Brothers engineer Chris Henderson using x264 for the Fringe box set and an engineer comparing it on Cinema Paradiso; he still prefers physical discs for highly cinematic films because carefully authored Blu-rays can look better without requiring an extravagant television.

26. New codec generations save bandwidth by exploding the search space

• From MPEG-2 through DivX-era MPEG-4, H.264, HEVC or VP9, and AV1, the basic ingredients remain transforms, intra prediction, motion compensation, and entropy coding. The rough rule offered is 25%–50% improvement per generation—summarized as 30%—but AV1 encoding may require roughly two orders of magnitude more CPU than H.264.

• The extra compute searches larger blocks, more partitions, more prediction directions, and more content-specific tools. YouTube therefore encodes broadly in H.264 but re-encodes popular material in AV1: expensive work becomes rational only when halving delivery size across hundreds of thousands or millions of viewers repays the server cost.

• I-frames are complete pictures; P-frames depend on earlier pictures; B-frames may depend on both past and future, so decode order differs from display order. A GOP can span roughly 250 frames, while intra-refresh can avoid later full I-frames by rebuilding clean regions gradually—compression that feels miraculous because viewers never see its dependency graph.

27. Adaptive streaming is simple locally and formidable end to end

• A streaming player downloads encoded segments, measures whether they arrive fast enough, and switches among perhaps seven prepared resolutions. Kempf says dropping quality can be as basic as reacting when a segment consumes more than half its playback duration; deciding when to climb again without oscillation is the subtler control problem.

• Kempf argues audio transitions can be more jarring than video changes: moving from full AAC to a Spectral Band Replication profile makes sound suddenly tinny, while visual quality shifts can pass unnoticed. Viewers tolerate surprisingly poor sports video at 30 FPS, but an audio glitch produces immediate recognition that something changed.

• Complexity returns at system scale: a live event must be captured, routed, encoded in several resolutions without QA time, protected with DRM, distributed through CDNs, and decoded across browsers and heterogeneous devices. Accepting 10–20 seconds of latency simplifies the network problem; interactive control removes that luxury.

28. VLC’s security model assumes both hostile media and hostile distribution

• WikiLeaks’ Vault 7 documents described a modified VLC carrying an extra DLL that collected, encrypted, and transmitted documents while the victim watched a movie. The attack did not exploit official VLC; it exploited trust in a counterfeit build, using expected fan noise and CPU load during two hours of playback as cover.

• Similar abuse by Chinese attackers targeting Indian users contributed to VLC being blocked in India until VideoLAN fought the ban. A fake German distribution site has reportedly persisted for 12 years: its installer waits about three weeks before downloading spyware or adware, evading simple analysis while search ranking places it before the legitimate site.

• VideoLAN’s official response is unusually strict: no telemetry, no knowledge of what users watch, offline build machines that first compile their own compiler, and double signing. Two intelligence agencies asked for backdoors; Kempf’s answer was “No,” adding that if compromise became mandatory, “we would shut it down.”

• Sandboxing must still contain honest bugs across VLC’s roughly 500 plugins, FFmpeg, shaders, and vendor GPU drivers. The team is splitting demuxing, decoding, and filtering into separate restricted processes, Chrome-style, but multimedia sandboxes must move hundreds of megabits or gigabits per second without copying away their performance advantage.

29. Kyber applies codec discipline to the control loop of machines

• Kempf’s new company targets teleoperation where video is feedback rather than entertainment: robots, drones, remote vehicles, submarines, cloud gaming, surgery, and eventually teleobservation of autonomous machines. Its promise is to “make distance disappear” through either projection of skill or projection of power.

• Kyber multiplexes video, audio, mouse, keyboard, gamepad commands, and sensor streams over one encrypted QUIC connection. It also corrects clock drift so several cameras, GPS, control events, and training records remain synchronized; otherwise a robotics model could learn from observations that do not correspond to the action timestamps they supposedly represent.

• Reliability comes partly from forward error correction: transmit a few percent of redundant data, reconstruct missing packets at the receiver, and avoid waiting for acknowledgments that add latency. A CES demonstration had a small 3D-printed rover near Las Vegas controlled from France through a webcam and compact board.

• The target is four milliseconds glass-to-glass, equivalent to roughly 240 Hz. Kunhya reports seven milliseconds Windows-to-Windows or Windows-to-Mac; he attributes about 3.5 milliseconds to NVIDIA’s hardware encoder and two to Intel’s decoder, leaving codec latency itself as most of the remaining budget.

30. AV2’s battle is as much about patent economics as compression

• AV2 is Alliance for Open Media’s successor to the 2018 AV1 specification, with a stated target of roughly 30% lower bandwidth at the same quality. Kempf expects an AVM encoder, likely other implementations such as SVT-AV2, and a VideoLAN decoder called dav2d—spelled D-A-V-2-D and described as “dav1d 2”—extending dav1d’s recursive naming joke.

• The parallel standards line is H.264 or AVC, H.265 or HEVC, and H.266 or VVC, where ITU and ISO naming reflects joint institutional history. The episode deliberately uses 30% per generation as a practical simplification, while acknowledging that screen content or another well-matched case can gain 70%–80%.

• Patent economics split the families. H.264 licensing was comparatively manageable, but HEVC accumulated MPEG LA, HEVC Advance, independent owners, uncapped claims, and tools included largely to establish patent positions; Netflix, Google, and Meta could contemplate hundreds of millions of dollars annually and instead fund a royalty-free alternative.

• Alliance for Open Media considers patents during standardization, forcing “double creativity”: improve compression while avoiding protected tools, sometimes inventing something that is effectively “I can’t believe it’s not a B-frame.” France’s rejection of software and mathematical patents also lets VLC implement formats whose combined fees, Kempf estimates, would exceed €200 per user.

31. Archivists treat FFmpeg as a Rosetta Stone for future civilizations

• Kunhya highlights an archival community associated with Dave Rice and institutions such as the British Film Institute. Its members value open source not only because budgets are limited but because playback software must survive institutions, vendors, and hardware—unlike Britain’s New Domesday project, which became difficult to read within decades.

• FFmpeg’s C implementation is central to that wager. Kunhya predicts C will remain decipherable in 1,000 years “like Latin”: perhaps no longer the everyday language, but close enough to logic and mathematics that future engineers can reconstruct a compiler and understand how historical bitstreams become images.

• Archivists funded FFV1, a lossless codec designed for fast software encoding, resilience, and bounded damage when storage bits flip; they also supported GPU acceleration. Their requirements reject losing an entire GOP and demand mathematical proof that subtle historical evidence was not compressed away.

• Their domain knowledge flows back into software: 1950s tape colorimetry, film scanners, degrading reels, obsolete tape heads, and storage recovery. With more material than institutions can save, choosing which films and tapes survive becomes what Kunhya called “digital stewardship,” while noting that he had coined the phrase; open workflows also let volunteers teach FFmpeg-based preservation in countries without wealthy archives.

32. Multimedia will expand from pictures and sound into every synchronized sense

• Kempf defines multimedia broadly as digital representations of multiple streams for human senses. FFmpeg and VLC already face point clouds, volumetric video, RGBD depth, XR, spatial audio, and 3D assets; VLC has even supported synchronized haptic control for hydraulic “4D cinema” systems.

• The logical extension is a container with tracks for touch, movement, odors, or neural interfaces—Fridman jokes about stereo smell and ffmpeg -i for the human brain. Kempf answers without irony: if a timed stream represents something a person senses, the architecture should eventually accept a module for it.

• The maintainers’ role is therefore not to implement every future device themselves but to keep the architecture extensible. Kunhya cites roughly 150 annual VLC contributors and perhaps 300 for FFmpeg: core teams create the first module and interfaces, then specialists can add the second and third implementations without redesigning the framework.

• Existing deployments already stretch the imagination: FFmpeg compressed imagery for the Mars 2020 rover, VLC monitored CERN’s 27-kilometer accelerator ring, and Formula 1, SpaceX, and the European Space Agency have used the tools. Kunhya’s favorite summary is “FFmpeg, nothing is beyond our reach”; his closing discipline is equally durable: “Learn from your mistakes, but don’t regret.”