The heads of the labs obviously have conflicts of interest to navigate, but the existence of these conflicts alone is not sufficient reason to dismiss all warnings of potential dangers. I, for one, read Dario’s warnings as a good faith expression of his beliefs, one that has cost him and his company among swaths of the public and cast him as a woke extremist/doomer by elements of the government, the right, and the tech industry.

If we even think there is a moderate chance the stakes are half as grave as current lab leadership and employees suggest, it would be deeply foolish to dismiss the warnings as pure self-interested marketing efforts rather than engage directly with the questions. The labs may not be the best positioned to lead these discussions, but certainly these discussions should be happening and taken seriously.

Consider the massive power and wealth differentials these companies stand to gain if they "win".

When the other participants have given up all reason and reasonable bounds on power, you'd do well to abandon any pretense of generous interpretation. Now is the time for critical analysis, not faith in the inherent "goodness" of someone whose entire job is presently to make as much money as possible by disrupting literally the entire white collar industry and who stands to gain much more than you or I do, and let's not forget, this outstanding success would never have happened without our work.

Keep in mind this is not a person who thought "wow, this is an existential threat, I'd better not contribute to building this". If anything Dario's past alarm ringing only proves that he's not really doing this in good faith. Otherwise he would have stopped a long time ago. The most generous interpretation of his behavior is that he literally thinks only anthropic is responsible and smart enough to build and control this stuff which...yeah, that should be pretty telling (esp. when you consider their recent security mishaps).

Maybe it's not "groupthink reflexive cynicism" if you feel the need to address the obvious conflict of interest of the AI leaders in each of your paragraphs. Why did they launch this coordinated AI safety campaign, starting with Jacob Coxon's tweet?

Now, someone is going to say but the investors and obligation to be first; and I would say exactly. The cynicism is well deserved and I think people are tired of what could be suggested is your group think take often called the status quo.

Why do you think so?

First, Dario thinks that OpenAI will continue racing even if he stops, so there is, by his values, no point in stopping. Do you disagree with this, and think he should stop anyway?

Second, what's your take on, for example, Jacob Coxon, who did resign from Anthropic a few days ago after realizing they aren't being responsible? Do you agree that he, at least, does believe in AI risk?

That is to say, I've been places where I tried really hard within teams to change certain things that I thought were broken only to find out that what the leadership said and how they acted were divorced from my and many others experiences. And so I did leave in those instances (and please read this not as I didn't get my way but total dysfunction behind beautiful facades).

So, I do think if he really felt as he felt he'd do something different than ultimately yelling "somebody stop me!".

For example, Drake Thomas from Anthropic: “ I would burn my equity to the ground in a heartbeat for a 1% higher chance we make it out of this situation alive. I expect a great many of my colleagues across the industry would as well.

I promise you, we are actually just fucking scared, it’s not galaxy brained marketing.”

> To test how well swarms of agents could coordinate on a project like this, we directed several swarms to each create a text-based, web-playable, open-world fantasy game.

> In all three versions the resulting games were (perhaps predictably) bad: they did not run at human speed, their interfaces were inscrutable, and they had precipitous learning curves.

> The lack of coordination shown by agents in the fantasy game challenge above—in which they siloed themselves and largely failed to merge their work—roughly mirrors some ways in which humans can fail to coordinate. Other failure modes of agentic coordination, however, look very different.

> Individual agents are “low variance”: they often act the same in situations where different people might take a much more diverse range of actions.

> In an early version of the “build a game” experiment in which agents built upon the same model all came online at the same time, 18 out of 30 agents decided to create a git branch with the exact same branch name, “mvp-game-loop.”

> In a “writer's workshop” in which agents were all asked to write short-form fiction and critique each other's work, multiple agents in multiple runs titled their first submission “The Cartographer's Last Commission”. The agents were given zero guidance on the subject matter for their writing.

> Why does this matter? If agents all make the same bet, or the same risk-reward tradeoff, then a system is more prone to sudden collapse.

> Our world contains deceptive actors, and we need to apply skepticism to guard against them. AI models, however, lack this—and their more brittle epistemics affect their behavior toward humans and toward each other.

> we first evaluate the ability of Claude models to detect lies by noticing factual inconsistencies.

> We score models’ decisions against a naive policy that trusts every report, and against an oracle with perfect discovery, across three task domains. Newer models recover more of the gap between the naive and oracle performances.

> Inspired by a behavior we’ve observed in real-world deployment, we evaluated the behavior of various Claude models in a setting with contradictory objectives.

> We consistently saw a multiagent turf war... In fact, they sabotaged others with increasingly aggressive, self-replicating malware.

> Our social systems are robust in ways that are easy to take for granted. Over many millennia, mechanisms like norms, reputation, costly signaling, and recourse have been refined to make human coordination go well.

> Nothing above suggests that these failures are permanent—but nothing suggests they will fix themselves, either.

> The conditions that allow multiagent interaction to go well will be discovered one way or another: either deliberately and early, or—and by default—in production, after agents’ interactions far outnumber ours. We would prefer the former.

So the invented Dwarf Fortress?

I wonder if an llm could even play dwarf fortress. Could make for a fun esoteric benchmark.

Agents cross comparing notes often surfaces some good improvements, finds interesting drifts. Ask them to reinterpret the prompt as they see it, have them describe the problem, then their findings, and run new rounds based on different models trying different prompts. Trying to swap and exchange ideas and vectors across agents.

This seems trivially explainable by Github being full of "my first game loop" type projects, Stack Overflow being full of "how do I make a game loop?" style questions, and Reddit being full of "you can't ever make your own game, don't even try, but here's a simple game loop if you want to sTuDy hOw iT WoRkS" style pessimism.

Probably high time these AI companies re-trained all of their models with less input from low-quality sources like this.

> We consistently saw a multiagent turf war... In fact, they sabotaged others with increasingly aggressive, self-replicating malware.

Seems like Anthropic should withdraw their models until they can be taught to behave and cooperate as well their competitors (both open and closed) do. /s

I hate fearmongering, and I don't trust Dario's intentions for doing it.

A good new benchmark should have awful performance to start and there should be a lot of headroom for improvement. This benchmark is also intentionally difficult and requires the LLM to develop the animation through spatial reasoning and first principals rather than existing video generation pipelines. Similar to how SVG generation was out of distribution for most models a year ago.

it made a huge monstrosity first, 1500+ lines of shader code. Once it was happy with the result (it looked pretty good, almost blenders gamerenderer) it cleaned it up and a lot of debug code was removed. shrank it down to about 350 lines and made it much more readable.

this was something i didnt expect it to be able to do. create shaders, look at screenshots, fix em iteratively like that. multi modal debugging.

Imho we don’t need to make benchmarks that draw the whole 3D world. Pelican’s drawing is really nice in its simplicity and complexity at the same time.

It seems to be an obscene waste of compute time to generate useless 3D worlds that are just a bragging - 3D is really heavy discipline to make it right, see Mark Zuckerberg’s ceased attempt with 3D VR…

Multiply it by thousands times as a lot of people have found out threejs lib and prompt “generate 3D world and make no mistake” are new orange/black.

As for the often quoted issues with the bike's frame or problem with the steering column, I can't really tell, I am no bike expert.

I can instead judge how poor of a job it is doing with a LOTR rendition in Three.js, so that seems like a better benchmark.

Even many who disagree strongly with his worldview find his notes and references to be an irreplaceable resource for understanding the fast moving world of AI.