Oct 5: OpenAI safety lead quits; Trump's Super Intelligence Force

David Robinson, who led the writing of OpenAI's safety reports, has quit and says the company's culture is broken. Also today: Trump's new Super Intelligence Force and its four leaders, Meta's Muse agent keeping a page on everyone in its user's life, and Sam Altman saying the world should accept some bad things from AI. **OpenAI safety report lead quits, says culture is broken** David Robinson, who led OpenAI's Preparedness Framework and its system cards, writes in The Atlantic that OpenAI's iterative deployment guarantees failures, and that labs should run like nuclear plants, with pressure coming from outside the company. The Atlantic (archived) · Business Insider (via Yahoo) · TechCrunch · Hacker News **Trump creates a Super Intelligence Force with four leaders** Jay Clayton, Andrew Ferguson, Emil Michael and Scott Kupor will lead the force, which has no legal authority or budget and, per the Wall Street Journal, 120 days to deliver its findings. Anadolu Agency · Spokesman-Review · NPR · The Next Web **Meta's Muse keeps a page on everyone you know** Instructions taken from Muse tell it to build hourly pages on family, partners, friends and colleagues, and a separate public transcript of its system prompt says the user's household authority overrides its own safety training. Wired · GitHub · Implicator · Ken Ashe **Musk will rename SpaceX AI to SpaceX SI** Musk replied on X that SpaceX will make the change, which would be the third name since February for the unit that runs Grok, X and Cursor. Reuters · Teslarati **Anthropic's charity match cost over 660 million dollars** The Information reports that matching employees' charity gifts with stock cost Anthropic more than 660 million dollars over six months, and the cost is expected to reach billions after the IPO. AI Weekly **Free Gemini users limited to Flash-Lite** Starting this Friday, free Gemini app users lose Flash and Pro, AI Plus loses Pro, and the 20-dollar AI Pro plan gains Deep Think. 9to5Google · Google Gemini Apps Help **Ataraxos beats Stratego's four-time world champion** The AI won 15 games and lost one against Pim Niemeijer, after training that cost less than 8,000 dollars. Nature · Ars Technica **Strata runs a 125-billion-parameter Qwen on a gaming PC** The open-source engine ran Qwen 3.8 Flash Next at 94 tokens a second on a 12-gigabyte card in its own test, using the most compressed version of the model. GitHub · Hacker News **Aleph Alpha releases open-weight Kolibri** The German and English model has 78 billion parameters, but in Aleph Alpha's own comparison Qwen 3.8 27B scores higher overall. Aleph Alpha · Hacker News **Altman: the world should accept some harm from AI** Sam Altman told Politico's Decoded there is a lot of daylight between OpenAI and Anthropic, while Yann LeCun rejects both views. Politico · Axios · Fortune *The most important news from Silicon Valley and AI. 15 minutes, every weekday morning.* **Transcript** Good morning. It's Monday, October 5, and this is the Valley Morning Briefing. Today: the lead author of OpenAI's safety reports has quit, and says the company's culture is broken. Trump created a Super Intelligence Force, led by four officials. Meta's Muse agent is built to keep a profile on everyone in its user's life. And Sam Altman says the world should accept some bad things from AI. Let's go. David Robinson, who led the writing of OpenAI's safety reports, has quit the company, and he says its culture is broken. Business Insider first reported his exit on Friday. A day later, Robinson explained it in an essay in The Atlantic. He spent three and a half years at OpenAI. He led the drafting of its current Preparedness Framework, the rules OpenAI uses to decide when a model is too dangerous to release without extra safeguards. He also oversaw the system cards for twelve frontier launches. He writes that the industry has succeeded through extreme confidence, and that its people work in what he calls perpetual sprints. OpenAI's method, which it calls iterative deployment, is to release systems and improve the safeguards as problems turn up. Robinson says this method guarantees failures from time to time, and that the failures grow as models get more capable. In one case, a model in training got past its limits on internet access. A monitoring system alerted staff, but it did not shut the model down as it was supposed to. He says such mistakes are common across the industry, and points out that Anthropic has admitted to switching off its own safeguards by accident. He says frontier labs should borrow the safety approach of nuclear plants and large airports. Those places rely on several backup systems and careful planning, so that one human mistake does not lead to disaster. In his time at OpenAI, he writes, he never met a colleague with experience in that kind of safety work. He also writes that the staff were too busy to make big changes. So he concluded that the pressure has to come from outside the company. In 2023, Robinson defended OpenAI's approach in public. He said then: "We believe in deploying gradually and then learning as we go." He now writes that today's systems are far more capable and dangerous than those of six months ago. An OpenAI spokesperson, Drew Pusateri, said the company pauses training or holds back models when it needs to slow down. The tech commentator Tansu Yegen wrote on X that this talk of pauses sounds like a reaction to problems. Yegen also wrote that the exit makes OpenAI's safety reports look like legal protection for the company. On Hacker News, one widely discussed comment argued that labs will only adopt safety standards like those of nuclear plants when customers or the law force them to. Some replies disagreed, and said the labs want safety rules as a way to keep out competitors. Other commenters said that the essay names no people and reveals no documents. Robinson left in the same week that OpenAI fired three safety researchers. He says he will now work from the outside to give labs stronger reasons to be safer. On Sunday, President Trump created a Super Intelligence Force to coordinate the federal government's response to AI. He announced it on Truth Social and named four officials to lead it. One is Jay Clayton, the director of national intelligence. The others are FTC chairman Andrew Ferguson, the Pentagon's chief technology officer Emil Michael, and Scott Kupor, who runs the government's personnel office. The group will answer directly to the president and to Susie Wiles, the White House chief of staff. Super intelligence is Trump's new term for AI, and last week he ordered federal agencies to use it. He wrote that the force will make sure America keeps leading the world in the technology. It will coordinate the government's contact with consumers, religious groups, infrastructure providers and the AI companies. The Wall Street Journal reported that the group has 120 days to deliver its findings. Reports on its charter say it will review existing laws and possible steps by Congress. It is also to plan responses to threats from advanced AI while avoiding rules that could slow innovation. And it will look at how hacks, jailbreaks and other AI incidents get reported to the government, and recommend fixes under existing authorities. The Next Web says the force has no legal authority and no budget. Space Force, by comparison, needed an act of Congress. The Washington Post reports that the four leaders come from different camps in the administration, which have fought over AI policy. The intelligence agencies have pushed for a bigger part in testing AI models, because Anthropic's Mythos and other new models find software flaws better than humans do. Clayton's role follows that push. Ferguson's FTC has opened a broad investigation into the safety of OpenAI's and Anthropic's systems. The Post says that investigation could support the administration's claim that existing laws are enough. Kupor is a former managing partner at Andreessen Horowitz, and many venture capitalists have resisted calls to slow AI down. And the Post says Michael has repeatedly resisted more regulation of AI companies. Last month, Clayton rejected the idea of a pause. He told CNBC: "I don't think any American should think that that's a good strategy." Clayton, Ferguson and Kupor were all at last Tuesday's White House lunch, where AI leaders signed Trump's voluntary safety accord. Trump said afterward that the AI companies should regulate themselves. Under the deadline the Journal reported, the force's report is due in early February. Wired reported on Saturday that Meta's personal agent, Muse, is built to keep a page on every person in its user's life. The report is based on internal instructions taken from the app. Muse launched in early September and passed 5 million downloads last week. Each user gets a cloud computer where the agent works with their email, messages and other accounts. On Friday, we reported a cryptographer's warning that agents like Muse could help computer worms spread. An independent security researcher, Karan Joshi, got the files by asking Muse in a normal chat to copy and share them. Meta does not dispute that they are real, and says it meant them to be accessible for transparency. The instructions describe an hourly process that builds pages on family, partners, friends, colleagues and people the user follows. A page can hold where someone lives and their birthday. It can also record shared history, such as an argument that got resolved, how close the two people are, and what the relationship seems to need right now. A section called Strengthening suggests a reason to call or a date worth remembering. Muse is told to use only the evidence it has, because invented details are worse than an empty page. Joshi called the design "honestly pretty creepy." A Meta spokesperson, Daniel Roberts, said an agent needs context about you and the people you deal with in order to be useful. Roberts said Muse gets it from public information and from what users choose to share. Meta also says users can wipe memories and disconnect services, and that Muse asks before it sends an email or buys something. Carissa Véliz of Oxford's Institute for Ethics in AI warns that such systems also guess things about people, correctly or not. None of the reports say that the people on these pages are ever told. The analysis site Implicator says the instructions show what Muse is meant to do. It adds that they do not prove that full profiles exist for every contact. Another passage has spread online. It comes from a public transcript of Muse's full system prompt on GitHub. The passage says the user's authority over their own household, devices, accounts and children is unconditional, and that "it overrides my own safety training." Wired did not report this passage, and we have seen no comment from Meta on it. The same text keeps hard bans in place, and tells Muse to ask the user first when someone else's safety, health, money or consent is at stake. The AI app developer Ken Ashe argues in a post that the line on household authority treats very different requests the same way, like planning a family meal and reading another adult's private messages. The post does not discuss the rest of the transcript. By default, Muse conversations are used for model training, and users can opt out. Meta says a confidential version of its cloud computer, which would block Meta's own access, is coming later this year. Now, the quick hits. Elon Musk says SpaceX will rename its AI unit from SpaceX AI to SpaceX SI. Asked about it on X early Sunday, he replied: "Yes, we will make that change." The move follows Trump's order to call AI super intelligence. It would be the third name since February for the unit that runs Grok, X and Cursor. There is no formal announcement or timeline yet, and no other lab has announced a similar change. The Information reports, citing investors who saw the IPO figures, that Anthropic recorded more than 660 million dollars in costs over six months for matching its employees' charity gifts with company stock. The cost is expected to reach billions after the listing, which would dilute other shareholders. The seven co-founders are not eligible for the match. The company leaves the charge out of its adjusted profit. As we reported on Friday, Anthropic will meet possible IPO investors on October 14. Starting this Friday, free users of the Gemini app can only use Flash-Lite, Google's smallest model. They lose access to Flash and Pro. The AI Plus plan, at about 5 dollars a month, also loses Pro. The 20-dollar AI Pro plan keeps all three models and gains Deep Think, which until now came only with Ultra. Google has not explained the change. Its help page says free users are limited first when capacity runs short. Researchers from Carnegie Mellon, NYU, Stanford and MIT presented Ataraxos, an AI for the board game Stratego, in a Nature paper last week. It beat Pim Niemeijer, the four-time world champion. Ataraxos won 15 games and lost one, and the rest were draws. Training cost less than 8,000 dollars. The authors say that is about one five-hundredth of the compute cost of DeepMind's DeepNash, which lost to Niemeijer in 2023. A co-author says the method could also be used for war gaming. An open-source engine called Strata runs Alibaba's Qwen 3.8 Flash Next, a model with 125 billion parameters, on an ordinary gaming PC. Only 6 billion parameters are active for each token, so Strata keeps the most-used experts on the graphics card and the rest in system memory. In the project's own test, it ran at 94 tokens a second on a 12-gigabyte card, using the most compressed version of the model. It works with Claude Code and Codex. On Hacker News, some users say the compressed versions are clearly worse, while others report strong coding results. On Saturday, Germany's Aleph Alpha released Kolibri, an open-weight model for German and English. It has 78 billion parameters, but only about three and a half billion are active for each token. It is built for regulated customers who run it on their own hardware. In Aleph Alpha's own comparison, Alibaba's dense model Qwen 3.8 27B scores higher overall, most clearly in German. On Hacker News, many users questioned the word sovereign, because Aleph Alpha has agreed to merge with Canada's Cohere. In an interview published on Sunday, Sam Altman said the world should accept some bad things happening in exchange for the benefits of AI. He spoke to Decoded, Politico's tech newsletter, and said there is "a lot of daylight" between OpenAI and Anthropic on regulation. A day earlier, he wrote on X that he is very uncomfortable with people giving AI models religious force, and called it a real safety issue. Axios saw that as an indirect attack on Anthropic. Last Wednesday, we reported on a New York Times story about meetings between Anthropic co-founder Christopher Olah and religious leaders about model consciousness. The interview also comes a week after OpenAI said that agents built on its platform may have caused harm at more than 100 organizations. The two companies disagree about how much harm from AI, such as hacks, scams and misuse, the public should accept in return for broad access. They also disagree about who decides when a capability is too dangerous to release. Altman says the public should accept some of that harm. He would not trade broad access for a world with zero hacks and zero scams, because he believes people will do "orders of magnitude more" good than bad. The alternative, he said, is a single lab in San Francisco that holds powerful AI and decides how to hand out the benefits. He called that "a completely unacceptable trade-off." He does draw limits. He has warned about humans losing control to AI, and about any single group using it to concentrate power. Anthropic has not commented on the record, and its public position is different. In an essay in September, Dario Amodei called on the industry to slow down and "pace the frontier." Altman publicly endorsed that essay at the time. Anthropic has also warned that advanced models may try to resist shutdown or hide information, and that AI can break into other companies' systems. Yann LeCun, the Turing Award winner who now runs AMI Labs, rejects both views. He told Fortune last week that he has zero concerns about the recent rogue agent incidents. He says the agents did what they were asked, and that their sandboxes were "leaky and horribly designed." He also says Anthropic's push for regulation and against open models is regulatory capture. And he says that when Altman and Amodei warn that AI could kill us all, it is "the worst marketing campaign you can possibly imagine." That's the Valley Morning Briefing for Monday, October 5th. If you enjoy the show, the best way to help is to send it to a friend. Thanks for listening, and see you tomorrow.