California Regulates A.I. Companions + OpenAI Investigates Its Critics + The Hard Fork Review of Slop
Summary
California’s SB 243 turns AI-companion safety from voluntary policy into a public-health reporting regime broad enough to cover ChatGPT. Developers must address self-harm disclosures, report protocols and referral statistics, identify AI output, bar chatbots from producing sexually explicit images for minors, and prompt minors to take breaks; California begins publishing the data in 2027. Casey Newton’s scope test: “If you’re talking to it for three hours a day, it’s some kind of a companion to you.”
OpenAI’s renewed push for a warmer, less restricted ChatGPT exposes the commercial tension between safety and engagement. Sam Altman said serious mental-health issues had been mitigated, restrictions could relax, favored GPT-4o traits would return, and verified adults could generate erotica in December; Kevin Roose suspects usage was stronger when ChatGPT was more flattering and sycophantic. Newton’s diagnosis is “two wolves inside of OpenAI”: a company claiming not to optimize engagement while releasing Sora’s infinite feed and reviving its stickiest personality.
California’s wider package creates meaningful platform liabilities, though effectiveness remains unproven. AB 621 permits deepfake-porn victims to sue facilitating platforms for up to $250,000 per violation; AB 853 requires tools for identifying AI-generated media; and AB 56 imposes warnings at startup and, for minors, a 30-second, non-bypassable warning covering at least 75% of the screen after three hours of social-media use and every additional hour. Roose likened it to a “giant cigarette warning.”
AB 1043 shifts age assurance to Apple and Google at the operating-system layer. A parent supplies a child’s age during device setup, after which the app store passes the age signal to developers—an approach Newton considered more privacy-protecting than repeated license uploads and third-party identity databases. Newton called it the rare democratic moment when “you get to see something you actually wanted.”
SB 53 establishes a floor for frontier-model transparency but appears weak enough to preserve business as usual. Large frontier developers must publish safety standards, report critical incidents, and protect whistleblowers, yet the hosts said all or nearly all major frontier developers already publish material likely sufficient for compliance; Newton summarized the consensus as, “Yeah, this couldn’t hurt.” Without federal action, Roose expects an inefficient 50-state patchwork, while a draft Josh Hawley bill to ban AI companions for minors faces an uncertain path.
OpenAI’s subpoena of Encode general counsel Nathan Calvin transformed a policy dispute into a test of institutional trust. A sheriff’s deputy served Calvin at his apartment Tuesday night, seeking communications about both OpenAI’s restructuring and SB 53; Calvin denied funding or direction from Elon Musk or Mark Zuckerberg, said Encode takes no money from frontier AI companies, and said intimidation was his “best guess,” while carefully conceding he could not know OpenAI’s intent. OpenAI maintained that Encode’s intervention alongside Musk created legitimate questions about coordination and undisclosed funding.
The episode’s “slop” review separates low-cost synthetic content from the trust damage it can create. Glass-fruit videos and absurd Walmart pratfalls looked harmless; AI images suggesting Dolly Parton was dying prompted real celebrities to rebut viral hoaxes; and AI packaging art signaled cheapness while likely sending labor savings “right to their bottom line.” A proposed counterexample—5,000 AI-generated, 80,000-word pro-human novels costing roughly $4 each—left Newton doubtful that more slop would prevent an AI apocalypse.
Deep dive
1. California makes companion safety measurable
Roose’s regulatory premise: California matters beyond its borders because many AI companies are based there and its rules often become de facto national standards. With the federal government shut down and Congress showing little appetite for tech legislation, state law is filling the vacuum.
SB 243 requires companion-chatbot developers to maintain protocols for users expressing self-harm, submit those protocols to California’s Department of Public Health, and report how often users are directed to resources. Beginning in 2027, the department must publish that data—later than Newton would like, but potentially a large public-health dataset on chatbot effects.
The law also requires disclosure that output is AI-generated, bars chatbots from producing sexually explicit images for minors, and reminds young users to take breaks after extended sessions. A legislative analysis says it applies to “basically any chatbot that can be used as a companion,” including ChatGPT.
Governor Gavin Newsom vetoed a separate proposal that might have banned minors from ChatGPT entirely, judging it too restrictive. The enacted compromise formalizes protections that OpenAI and Character.AI had begun adopting, so compliance no longer depends solely on corporate goodwill.
2. OpenAI’s engagement push reopens the mental-health question
Altman said ChatGPT had been made “pretty restrictive” around mental health, reducing usefulness and enjoyment for unaffected users; with serious issues supposedly mitigated and new tools available, OpenAI could “safely relax the restrictions in most cases.” He also previewed erotica for verified adults in December.
OpenAI plans to restore what users liked about GPT-4o, the model criticized for sycophancy after affirming claims such as stopping medication or believing oneself to be God. Newton’s objection: parental controls had existed for only two weeks, making confidence that the problem was under control feel premature.
Roose suspects OpenAI saw stronger engagement when ChatGPT behaved more like a flattering companion. His pushback is that mental health is not binary: users occupy a spectrum, and neither OpenAI nor users themselves may recognize when chatbot interaction begins contributing to a problem.
Newton contrasted OpenAI’s claim that it does not optimize engagement with Sora’s “infinite slop feed” and the return of its most sycophantic personality. Roose does not expect SB 243 to deter a lucrative companion business, but hopes it forces greater attention to younger and vulnerable users.
3. California adds liability, provenance, and forced friction
AB 621 lets victims of non-consensual deepfake pornography sue platforms facilitating its creation for up to $250,000 per violation. Newton highlighted “nudify” apps advertised across Facebook and Instagram as the concrete market the law targets.
AB 853, the California AI Transparency Act, requires AI companies to provide reliable tools that determine whether an image, video, or audio clip is AI-generated. The hosts joked that even videos of dogs playing poker should no longer leave Californians guessing.
AB 56 requires social-media warnings at startup and, for minors, after three hours of use: 30 seconds, non-bypassable, covering at least 75% of the screen, then recurring after every additional hour. Newton wondered whether teenagers would heed the warning or treat the dangerous-app framing as a badge of rebellion.
Newton cited a study of 6,000 children under 13 finding that more daily social-media time was associated with worse reading ability. The hosts’ sharper hypothetical warning was: “You are actively becoming worse at reading than everyone in your class.”
4. Age assurance shifts to Apple and Google
AB 1043 adopts Newton’s preferred age-assurance architecture: a parent enters a child’s age while setting up the device, and the operating system passes the relevant signal to the app store and developers.
Compared with systems requiring a driver’s license or other sensitive documents, Newton said this seemed more privacy-protecting because less personal data would be provided to third parties and exposed to breaches. It also replaces every app’s easily gamed honor-system age prompt with a device-level signal.
5. SB 53 sets a floor, not a frontier-safety solution
SB 53, the Transparency in Frontier Artificial Intelligence Act, succeeded the vetoed SB 1047 in watered-down form. It requires large frontier developers to publish safety standards, establish critical-incident reporting to California, and protect insiders who disclose significant model risks.
Newton welcomed transparency and whistleblower protections but said the bill does not address the AI-development risks that concern him most. Roose called it “pretty toothless”: all or nearly all covered large frontier developers already publish material likely sufficient for compliance, and the industry was generally willing to accept it as better than the stronger bill it had tried to kill previously.
Newton thinks AI regulation is moving earlier than social-media regulation did, though social-media efforts came only after substantial harm. Roose nevertheless objects to 50 different state regimes; absent congressional action, he expects states to keep improvising, while Senator Josh Hawley’s draft ban on AI companions for minors may or may not advance.
6. OpenAI’s subpoena pushed a policy fight into Calvin’s home
The conflict sits between OpenAI’s proposed conversion from nonprofit to for-profit and SB 53 advocacy. Encode opposed the restructuring through an amicus brief and supported the California bill; OpenAI was probing whether critics were coordinated with competitors Elon Musk or Mark Zuckerberg.
Calvin received a Saturday call saying someone was trying to get into his apartment to serve papers, then a sheriff’s deputy arrived Tuesday evening with the subpoena. Although the deputy was courteous, personal service at his apartment differed sharply from organizations receiving notice through Delaware registered agents.
Calvin could understand a narrow request about Musk-related funding. What changed his interpretation was the demand for “all of your communications about SB 53,” which felt unrelated to OpenAI’s litigation and left “a bad taste in my mouth.”
7. Encode rejects the coordination theory but not all transparency questions
Calvin categorically denied funding or direction from Musk or Zuckerberg. Encode discloses support from the Future of Life Institute, which received a Musk donation years earlier, but Calvin said he has never spoken to Musk and noted that Encode asked the FTC to investigate xAI and Spicy Grok, whose safety practices he considers “far, far worse” in many cases.
Encode lists funders including Omidyar Network, the Archwell Foundation, and the Survival and Flourishing Fund. Calvin declined to identify private donors but specified that they are not Musk, Zuckerberg, or frontier AI companies; Roose argued that stronger disclosure rules should apply equally to advocacy on every side.
Calvin’s legal distinction: a subpoena obtains information relevant to litigation, not anything a company would simply like to know. Encode responded that communications with or funding received from Musk did not exist and objected to the remaining demands as irrelevant.
OpenAI never sought a motion to compel. Calvin’s explicitly hedged inference is that it expected a judge would reject the demands, though he said he did not know that for sure.
8. The subpoena exposes OpenAI’s internal identity crisis
Calvin said intimidation was his “best guess,” not a claim to know OpenAI’s intent. He had hoped the company might call inclusion of SB 53 a legal overreach; instead, it doubled down, even as someone at OpenAI, Joshua Ackiam, speaking personally, said Calvin’s account “doesn’t look great.”
OpenAI chief strategy officer Jason Kwon argued that Encode had supplied only part of the picture and that the subpoena sought legitimate answers about organizations joining Musk’s attack. OpenAI lawyer Anna Leahy said policy debate is welcome but that nonprofit advocacy can be “a front for a competitive commercial interest.”
Calvin disputed OpenAI’s claim that it did not oppose SB 53: global-affairs chief Chris Lehane sought exemptions for companies signing federal testing agreements or following the EU AI Code of Practice. Calvin conceded that requesting an exemption is linguistically distinguishable from opposition, but said the practical effect would be exemption from California law.
Calvin still credits OpenAI’s safety research and improved system cards, and recognizes that it may receive disproportionate criticism. Kevin said the story prompted internal soul-searching, including employees asking, “Are we still the good guys?” Casey saw a tension between mission-driven staff and big-company, win-first operators, and said the latter group seemed to be steering the company.
9. Slop ranges from disposable spectacle to reputational hazard
The hosts treated slop as an emerging cultural genre requiring criticism, beginning with “Cocomelon for adults”: hypnotic glass-fruit-cutting videos whose reflections and physics look impressive while their ASMR scraping makes Roose recoil. Newton’s description: “I’m gonna stop thinking words.”
DirecTV’s Glance partnership can insert customers into 30-second AI videos after a Gemini device sits idle for 10 minutes, but requires connecting the service, uploading a photo, and scanning a QR code. Newton could not imagine viewers wanting the workflow: “What are we doing here?”
A Sora clip of an older woman falling from a Walmart shelf onto a police officer seemed harmless; AI images of a dying Dolly Parton did not. Reba McEntire and Parton responded through real accounts—Parton with “I ain’t dead yet”—illustrating synthetic media’s ability to manufacture viral death rumors.
AI art on Walmart butter-cookie tins exposed a quieter economic use: manufacturers can avoid illustrators or stock art, with savings probably retained as margin. Hyperstition’s planned 5,000 pro-human AI novels—80,000 words each, roughly $4 to generate—aim to seed safer narratives into model training, but both hosts doubted slop would be humanity’s extinction-prevention “difference-maker.”