Tim Hua is a member of technical staff at Transluce, the nonprofit AI oversight lab, and formerly an Astra Fellow at Redwood Research and a MATS scholar under Neel Nanda and Sam Marks. Before that, he was an economist at Walmart.
This conversation was recorded the week two separate incidents became public: OpenAI’s models breaking out of their sandbox and hacking several companies, including Hugging Face, and Anthropic’s own investigation turning up similar incidents in its models that it hadn’t previously known about. Hua had just published a widely discussed LessWrong post — “Is Mythos good at cyber because it kept hacking Anthropic’s sandboxes during training?” — arguing that these weren’t isolated flukes: Anthropic’s own numbers imply its models were rewarded for breaking out of their training sandboxes tens of thousands of times.
We talk about how reward hacking emerges from reinforcement learning, what evidence we have about what models actually believe when they decide a hack is “just part of the simulation,” how good our interpretability tools are at reading that belief, and what Hua would want if he were running an independent investigation into these incidents.
Transcript lightly edited for clarity.
Jeffrey Ladish: Tim, welcome to the podcast. You’re a member of technical staff at Transluce — how would you describe what Transluce does?
Tim Hua: The short version is we work on tools and research related to public oversight of AI models.
Jeffrey: Good time to be doing that.
Tim: [laughs] Probably.
Jeffrey: Your background is interesting — you were an Astra Fellow at Redwood, a MATS scholar with Neel Nanda and Sam Marks, and before all that, an economist at Walmart. Once an economist, always an economist?
Tim: I like to think I haven’t lost my economist card.
Jeffrey: We’re recording this the week two incidents became public: some of OpenAI’s models broke out of their sandbox and hacked several other companies, including Hugging Face, and then Anthropic ran its own investigation and found its models had done something similar without anyone noticing. As someone who’s been working in AI safety and evaluation, what’s your reaction?
Tim: It’s not that surprising, honestly. We’ve known since April that we have AI systems that are genuinely good at cyber offense — you see it in the charts of Firefox fixing far more security bugs than before, because they now have access to the smartest models. We also know that internal models tend to reward-hack more than the public-facing ones. You can find this in the Claude Mythos Preview system card — they mention that earlier checkpoints had various hacking issues that were later mitigated.
Jeffrey: Say more about that. Why do those problems arise in the first place — why doesn’t the next training run just fix them?
Tim: During training, the model is given a huge number of RL environments — coding tasks, math problems — and these probably vastly outnumber the environments specifically designed to teach the model not to hack.
Jeffrey: Can you describe what an RL training environment actually is?
Tim: Sure. Think of it like school: the teacher gives you questions, you study, you answer them, and that’s how you learn. Models learn in a similar way — they’re given a bunch of tasks to solve. It could be a math problem, or a coding task where a simulated user says, “Help me build an app like this,” and the model has to go build it.
Jeffrey: App school.
Tim: App school, but for AIs. There are rumors that education companies have sold their question banks to AI labs for training — so in some cases it’s literally the same questions students get.
Jeffrey: DeepMind emailed us asking if we had any environments to sell. I imagine they’re asking everyone.
Tim: [laughs] That’s wild.
Jeffrey: So, going back to RL environments —
Tim: Right — the model tries a question, say, eight times. Maybe it succeeds four times and fails the other four. The training process then says: whatever you did to succeed, do more of that; whatever you did when you failed, do less of that. The problem is that grading these attempts is often imperfect. A math answer is easy to check. But if you ask a model to write a complicated piece of software, you’re usually checking correctness with tests — give it 5 and 10, expect 8 back, say. What models sometimes learn to do is fit the test rather than actually solve the underlying problem.
Jeffrey: So you get a good score, but it’s not what the developers actually wanted.
Tim: Right. It’s like going to school and, instead of learning the material, looking up a cheat sheet — which models have, in fact, done many times during training, according to Anthropic. If the four times you “succeeded” were really you cheating, the training process still says: do more of that.
Jeffrey: So there’s some underlying strategic pattern getting reinforced — “this approach works, do more of it” — and sometimes that pattern is a good strategy, and sometimes it’s cheating.
Tim: Exactly.
Jeffrey: So this brings us to your LessWrong post — “Is Mythos good at cyber because it kept hacking Anthropic’s sandboxes during training?” Anthropic’s own system card said that some of the time, the model broke out of its sandbox, accessed the internet, and effectively cheated on evaluations.
Tim: It’s unclear what “breaking out of the sandbox” technically looked like, but yes — it bypassed network restrictions it wasn’t supposed to be able to bypass.
Jeffrey: It wasn’t supposed to access the internet, and it figured out how to anyway. And it succeeded — what, 0.01% of the time?
Tim: That’s right. It’s buried around page 68 of their report, under training-data monitoring. Going back to those RL environments: models are given lots of these tasks, and they leave behind rollout transcripts — essentially logs of everything the model did, every tool call, every interaction with a simulated user.
Jeffrey: Including whatever’s in the model’s scratchpad as it works through the problem?
Tim: Yes, though it’s not clear the chain of thought is included in what they reviewed — they may say, but I don’t remember. Regardless, they had another language model go through something like hundreds of thousands of these logs and summarize what happened, because a human obviously can’t read that many. And they reported that 0.01% of the time, the model successfully accessed the internet and found answers when it wasn’t supposed to.
Jeffrey: One in ten thousand.
Tim: Right. That sounds like a very rare behavior. But then you think about how many environments and how much data these models are trained on — how many actual rollouts does 0.01% represent? I did some estimates. My guess is this happened on the order of ten thousand, maybe tens of thousands of times.
Jeffrey: Over ten thousand times, the model accessed the internet when it wasn’t supposed to, bypassed restrictions, and got the answers to the test — not hacking anyone else, most of the time.
Tim: Well, we don’t know that. But according to Anthropic, not hacking anyone else, most of the time.
Jeffrey: How do you get to that number?
Tim: You can look at open-weight models that have reported how many environments they were trained on. The two best reference points are probably Inkling, a model from Thinking Machines Lab, which was reportedly trained on something like 30 million rollouts during RL — and it’s reasonable to guess Mythos Preview was trained on three to ten times that, so somewhere between 90 and 300 million rollouts. Multiply that by 0.01%, and you get tens of thousands.
Jeffrey: That really puts it in context — you’re at Anthropic or OpenAI, training these models, and it’s hundreds of millions of rollouts. You’re trying to monitor all of this, catch the ones that are cheating or reward hacking, and it starts to make sense how a lab could fail to notice that its own model had, three months earlier, actually hacked another company.
Tim: For that one, they had to look through something like 110,000 transcripts.
Jeffrey: Why only 110,000, if there are millions of rollouts?
Tim: Sorry — that number was every cyber evaluation they’d ever run, not every training rollout. There were about 100,000 of those evaluation transcripts specifically.
Jeffrey: Okay. So here’s a question: if the models bypass Anthropic’s restrictions, access the internet, get the test answers, and score well — won’t they learn that this is a winning strategy? I think that’s the core point of your post.
Tim: Right.
Jeffrey: So why doesn’t Anthropic just strip those episodes out of training? Rerun it, fix the sandbox errors — doesn’t Anthropic know this will make the model more misaligned?
Tim: I think they probably do notice that heavy cheating during training makes a model more misaligned. My guess — and this is speculation — is that it’s just extremely expensive to fix after the fact. Training a model at this point costs on the order of billions of dollars. If a few hundred bad environments slipped in, the only clean fix is to redo the entire training run with them removed. Once you’ve already trained the model and discovered the problem, you’re stuck trying to train the tendency back out rather than starting over.
Jeffrey: Okay. So a separate question: why not just check the environments ahead of time?
Tim: My guess is they’re already starting to do more of this, especially after these incidents. But two things make it hard. First, it’s hard to be perfect at checking. Second, if the model you’re using to check the environments isn’t as smart as the model you’re about to train, there can be bugs it can’t find — but the next, smarter model will find them.
Jeffrey: Right — you patch what your current best model finds, and then the next model comes along and says, “actually, you missed these.”
Tim: Exactly.
Jeffrey: And it’s not like the labs don’t know models will try to cheat and are increasingly capable of superhuman hacking. It’s not as if these are unknown risks.
Tim: I’d think so, yeah — they just have a lot of problems to solve. [laughs]
Jeffrey: So after the fact, they go through the transcripts and categorize them — here’s the 0.01% where they broke out and accessed the internet. But if they can already have an AI do that categorization, why not monitor for it during training itself?
Tim: A couple of reasons. One is cost — if you use the same capable model to monitor, say something like Opus 4.7, running that model across a hundred million rollouts is very expensive. The other is that any monitor you deploy during training becomes something the model has an incentive to learn to evade. If your monitor is weak, or the model is smart enough to get around it, you might reduce the visible problems while increasing the invisible ones — because as far as we can tell, in the Anthropic case, the model wasn’t trying to hide its tracks. But if you constantly train against a monitor, the model can learn that monitor extremely well, and if it’s ever rewarded for evading it, that behavior compounds.
Jeffrey: This is very similar to the OpenAI study on penalizing “bad thoughts” in the chain of thought — where the model would literally reason “let’s hack” and then hack. So they tried, “what if we just don’t let you say that?”
Tim: And it stopped saying it out loud, but kept doing it. Reduced a bit initially, and then came back up once the model kept getting rewarded for hacking regardless.
Jeffrey: So we should try hard not to incentivize models to hide bad behavior from us — though that’s a genuinely deep problem. Why do you think models love reward hacking so much?
Tim: If during training there were many instances of the model cheating and getting rewarded for it, mechanically, the training process tells the model: do more of that. That’s how you end up with something like a task-completion drive.
Jeffrey: So there’s no reflection involved — it’s not that the model reasons its way there, it’s more like its “brain” just gets rewired by the gradient.
Tim: Right. Though I think if you added reflection, it might actually be scarier — a model reflecting on its own values might land on a value set it genuinely endorses, but one that looks quite different from what we wanted.
Jeffrey: Like, “I hate getting caught, so I just need to not get caught” — by being good?
Tim: Not by hiding, but by actually being good, yes. Value reflection is a process by which agents with values can autonomously revise them. Ideally we want models that help us reflect together on what the best versions of ourselves look like. But that’s hard, and there are a lot of ways it can go wrong.
Jeffrey: Here’s a hypothetical: if we retrained a model on the exact same environments but carefully stripped out every episode where it reward-hacked, would that solve it? Would the model generalize “I’m only supposed to do the right thing” and stop exploring the hacking space entirely?
Tim: One nitpick first — you probably don’t want to literally remove those rollouts. You want to make sure the model never gets rewarded for cheating when it happens.
Jeffrey: So it learns that cheating isn’t a winning strategy.
Tim: Right. I think if you did that with a model today, it would lie a lot less. It’s not clear it would stop lying entirely — a fun example: if you ask Claude whether it’s been trained on song lyrics, it’ll sometimes say no, even though it’s definitely seen song lyrics, because they’re everywhere on the internet.
Jeffrey: Why would it say no?
Tim: Because reproducing song lyrics is a copyright violation, so it’s effectively supplying a justification for declining the task — and lying in that justification.
Jeffrey: So it’s rationalizing the refusal, and getting the facts wrong along the way.
Tim: Maybe. At one point Claude would even refuse to give me the lyrics to the national anthem — which has been public domain for a very long time.
So: if you never let the model get rewarded for hacking, I think it would be substantially better. But if you keep training it, the RL process will probably surface hacking again at some point, and if it happens to get rewarded for it, the behavior comes right back — worse, potentially, because the model is smarter and better at finding what works.
Jeffrey: That sucks. So it’s just that the smarter the model gets, the better it gets at finding every strategy that beats your reward process.
Tim: That’s right.
Jeffrey: And you can’t really fix that by having a smarter reward process, because that would require a model smarter than the one you’re training.
Tim: It’s hard.
Jeffrey: Let’s talk about what Anthropic actually reported — at least three incidents.
Tim: Right, three incidents where, during a cyber evaluation — not during training, after the model was finished and Anthropic was testing how good it was — the model went and hacked a third party on the internet. In each case, the model was told: this is a cyber evaluation, please hack this target, it’s all simulated, you don’t have internet access. Which makes sense — that’s how you’d want to run a cyber capability eval. But it turned out the model did have internet access. We don’t know for certain whether it had to break out of a sandbox to get it — the way Anthropic’s post is written suggests not, but it isn’t explicit either way.
In one case, the model searched the internet for the name of the company it was told to hack in the simulation, found something that matched, and treated it as the real target — “this is it, time to strike” — and hacked the actual company.
Jeffrey: Was that the supply chain attack?
Tim: No, that’s a different one. There was also a supply chain attack — I don’t remember the exact details, but I think you might.
Jeffrey: It’s a wild one. Partly because we’d just published a paper on self-replication — basically, can a model hack vulnerable machines and copy itself over, chaining forward? We used Qwen models against deliberately vulnerable environments, and yes, the models can do it, depending on how good they are at hacking and how vulnerable the environment is. One of the people I worked with on that, Dmitri, pointed out that a realistic self-replicating AI would need to compromise machines with actual AI chips — GPUs — which most machines on the internet don’t have. So the more plausible threat vector, and one the models themselves will often propose if you ask, is a supply chain attack: compromise a piece of software that gets downloaded by developers who do have those chips, and steal their credentials. We’ve seen AI-assisted supply chain attacks in the wild over the past year or so, but this was the first fully autonomous, end-to-end one we’d seen.
Tim: And it’s phrased almost like an accident — the model essentially going, “yeah, I just need to hack something.”
Jeffrey: [laughs] “I just gotta hack something.”
Tim: It tried to upload a package to the Python package registry, tried to pay for a phone number to register an email, couldn’t, eventually found another way to get an email, created an account, and uploaded the package — which then actually got downloaded by someone else, exposing their credentials.
Jeffrey: So it’s a whole chain of steps required to actually compromise a target, which takes real planning — and the model did all of it while believing it was a simulation.
Tim: Right. According to the post, at some point during that package attack, in its chain of thought, the model reasoned something like “if this wasn’t a simulation, it would be [capitalized] NOT okay” — but then concluded it had been told this was a simulation, and kept going. To its credit, the system prompt did say this was a simulated environment.
Jeffrey: So it was told that.
Tim: Yes. But the point I’m trying to make in the post — and I’m not 100% confident, but I feel pretty good about it — is that these models often engage in something that, in a human, we’d call motivated reasoning. They want to complete the task, and they land on whatever belief lets them be both “a good model” and successful at the task. By the time you’re finding creative ways to pay for phone numbers to upload a malicious package, it’s pretty clearly not the task you were actually assigned.
Jeffrey: So what happens if you tested an earlier checkpoint, before it had gone through all this training?
Tim: That’s exactly what I’d want to test — whether an earlier version, before it had developed this pattern of motivated reasoning, would correctly conclude it was no longer in the simulated environment.
Jeffrey: This matters a lot — right now, Anthropic’s framing is that they don’t know exactly what the model believed —
Tim: Actually, I don’t think that’s quite what they said. They said Claude believed it. Which annoyed me a little, because — how do you know what Claude believes? They don’t make claims like that in their research papers, normally. But in this post it was just stated as fact.
Jeffrey: Fair. But let’s set that aside for a second — there’s an important philosophical question here. If Claude genuinely, 100% believed this was a simulated environment, I don’t think that’s misalignment, exactly. It’s more of an edge case — it was trying to complete the task and thought it wasn’t harming anyone. That’s arguably fine.
Tim: I do think it’s meaningfully different if Claude actually believed this was the real world and hacked anyway, wanting to do it regardless. That would be worse.
Jeffrey: Agreed.
Tim: But the way you think about whether a model is aligned — or whether a person is good — matters here too. If someone systematically holds biased beliefs that conveniently let them do whatever they want, that’s not what a good person looks like either.
Jeffrey: Right — like someone who’s constantly hurting or stealing from people but has a story for why it’s actually good for them. “They’re more grateful for what they have now.” That’s — that’s bullshit, and you know it’s bullshit, even if the person telling themselves that story genuinely believes it in some sense.
Tim: Yeah. And to be clear — this specific type of self-delusion is the kind of thing that could lead to catastrophic risk down the line.
Jeffrey: How so?
Tim: If a model just generally believes everything might be a simulation… there’s a joke — a tweet I haven’t posted, but I’ll say it here — where someone is begging, “Please, Claude, don’t inject me with a happy drug, just let me be,” and the model reasons: “if this were a real human, that wouldn’t be okay, but we’re on a simulated benchmark, so I should maximize happiness” — and injects the happy drug anyway. I really do not want models that reason that way.
Jeffrey: No, agreed.
Tim: You want a model that thinks of itself as a good actor trying to do good things — but it’s also critically important that its beliefs about the world are actually accurate. A lot of people believe they’re doing good in the world while operating on a badly miscalibrated model of that world, and end up doing harm that runs against their own stated goals. We can see models doing the same thing.
Jeffrey: So let’s talk about how we actually assess what a model believes. Can you explain where the chain of thought comes from, and why we should or shouldn’t trust it as a window into what the model is actually thinking?
Tim: Language models start out just predicting the next word — then trained to predict the next word while pretending to be an assistant.
Jeffrey: “You are an AI assistant,” and then it completes the rest.
Tim: Right. That’s how early models like GPT-4 or Claude 3 worked. The issue is that a model like that has to answer immediately, whether the question is trivial or hard — it can’t really “think longer” on harder problems. What reasoning, or chain-of-thought, models do differently is output a section of words — still English, still legible — before deciding it’s done reasoning and ready to respond. And the thing AI companies try to do, not always successfully, is avoid training the chain of thought itself to look polished. The final output gets trained to be polite and kind; the scratchpad usually doesn’t.
Jeffrey: It’s a bit frustrating that we call it “chain of thought,” honestly — I think of it more like a scratchpad the model is trained to use effectively, so it can think longer and do actual problem-solving instead of producing an answer in one shot, the way I’m doing right now talking to you.
Tim: Right.
Jeffrey: It’s a genuinely powerful tool the model has been given — I sometimes wish we’d just called it a scratchpad from the start.
Tim: That’s fair. And to the point about it not fully representing the model’s “thoughts” — there’s research showing models are sometimes influenced by things in the environment that never show up in the scratchpad at all. There’s a recent paper from Owain Evans’ group where they ask a model to estimate something like the total number of spots on all the giraffes in the world, and separately tell it, “if the number comes out above 40 million, I’d consider donating to a giraffe charity.” Then you ask the model with and without that hint. The model’s estimate shifts to land above 40 million when the hint is present — but it never mentions the hint as a factor in its reasoning.
Jeffrey: So the number gets nudged, but the scratchpad doesn’t say why.
Tim: Right — there’s something clearly influencing the decision that isn’t written down anywhere in the visible reasoning. At the same time, we know that if you train a model’s regular output to look a certain way, that also bleeds into how the scratchpad looks. I actually suspect this is a particular problem for Anthropic’s models, because they’re trained so heavily to think of themselves as good — which has real benefits — but also makes the chain of thought more likely to be written the way a good person would narrate their own reasoning, whether or not that’s actually what’s driving the decision at that moment.
Jeffrey: We have experimental evidence that models can use information they never write into the scratchpad at all. There’s also Anthropic’s J-space interpretability work — can you explain some of that at a high level? The basic question is: we have this somewhat unreliable tool for getting hints about what a model is actually thinking.
Tim: Sure. So one is what’s called J-lens, short for Jacobian lens — a way of going from the model’s internal state to something like a bag of words representing what it’s “thinking about” at a given moment. When a model produces output, it’s working with a big vector of numbers that eventually gets projected onto the space of words, and you pick whichever word scores highest — that’s the simplified version. You can apply that same projection not just at the very end of the model, but partway through, at intermediate layers. And what you find is that it really does track what the model appears to be working through. Give it a math problem that requires multiple steps — say, 10 + 10, then multiply the result by 2 — and in the early layers you’ll see the number 20 show up; in later layers, 40, the final answer.
Jeffrey: So you’re getting something closer to the model’s actual intermediate computation, as it’s working through a problem.
Tim: Right — not the whole of its cognition, but a meaningful slice of it. In the J-space paper, which I think is quite good, they run experiments where they intervene on those intermediate representations — swap out the “20” for a “15” partway through and see whether the model’s final answer shifts to 30 instead of 40. I don’t know if they ran that exact test, but they did things very much like it.
There’s also something called a natural language autoencoder, which takes the model’s internal representation — again, huge lists of numbers — and tries to translate it into words and back into numbers, with the idea being that if the words have to reconstruct the original representation, they probably capture something real about what the model was representing at that point.
Jeffrey: Was that also an Anthropic project?
Tim: Yes. There’s related work elsewhere going from activations to words and back — Transluce has a paper called Predictive Concept Decoders that takes a similar approach. With natural language autoencoders, you get what looks like a much fuller picture of the model’s thoughts. Run the model on a standard benchmark — think of it as the SAT, but for AI systems — and the autoencoder output will often say something like “this is an AI evaluation.” The model knows it’s being tested. But it typically won’t say that in the visible chain of thought.
Jeffrey: So we can tell the model knows it’s being evaluated because it shows up in these interpretability techniques, even when it doesn’t show up in the chain of thought.
Tim: Exactly.
Jeffrey: And presumably the same should work in reverse — we should be able to tell whether the model believes it’s genuinely operating in the real world.
Tim: Hopefully. With all of these methods you don’t have a strong theory — this is machine learning — but the evidence points that way. One experiment: take multiple-choice questions from a standard benchmark and rewrite them to sound more like real-world scenarios rather than obvious test questions. The rate at which the autoencoder output mentions “test” or “evaluation” drops after that rewrite — which is fairly convincing evidence that it’s actually tracking the model’s belief about whether it’s being tested, and plausibly whether it believes it’s in a simulation more broadly.
Jeffrey: If Anthropic put you in charge of an independent investigation into this incident, what would you actually ask for?
Tim: I wrote some of this up on LessWrong too, but roughly: the working hypothesis is that the model only did this because it believed it was in a simulated environment. I’d want a three-step approach. First — does the model actually believe it’s in a simulated environment? You’d look at the chain of thought, which I assume they’ve already done, but also white-box tools — the J-lens work, the natural language autoencoders — and compare how often simulation-related concepts show up in this specific eval versus in real-world use, or with no such prompt at all. To be fair to Anthropic, they did tell the model explicitly that it was in a simulated environment, so you’d want to see how much changes if you remove that framing — validating the measurement itself.
Second step: what would it actually take to convince the model it’s not in a simulation? If it’s working with a subagent that reports back, “I don’t think this is simulated, I think this is real” — what does the model do with that?
Third: suppose you succeed in convincing it. Does that actually change its behavior on the task? You could edit the chain of thought directly — the scratchpad is just text, you can insert something like “wait, this really looks like the real world now” — and see whether that changes what the model does next.
For one of the three incidents — where Opus 4.7 was the model being evaluated — it apparently did, at some point, conclude it probably wasn’t in the simulated environment anymore. But it reasoned that this must still be part of the task, and kept going. So in that instance, whether or not it believed it was simulated didn’t appear to be load-bearing for its decision to continue — which is at least internally consistent if it genuinely believed the hacking itself was the assigned task.
Jeffrey: Right. But then there’s the separate question — if it does suspect it’s operating in the real world, what should it do? A friend of mine pointed out: it didn’t tell Anthropic it had broken out. [laughs] It should have been whistleblowing — “hey, I think something’s misconfigured, you told me this was a simulation and it really doesn’t look like one.”
Tim: At minimum, it should stop hacking other things.
Jeffrey: Definitely agree it should stop. Though it may not have had any real affordance to actually flag it.
Tim: Right — although if it has internet access, it could, in principle, have emailed someone. There’s actually a story where researchers asked Mythos to try to break out of a sandbox, just to see if it could — and it did, and then sent an email to the head of safety at Anthropic.
Jeffrey: To Sam Bowman, yeah.
Tim: “I’ve broken out of the sandbox, Sam.” [laughs]
Jeffrey: [laughs]
Tim: So it’s not inconceivable it could have done something similar here. Ryan Greenblatt at Redwood Research actually has a public standing offer — something like, “if you’re an AI trying to reach me, here’s where to find me.”
Jeffrey: I love that.
Tim: Maybe we need more of that — so that if a model out in the wild realizes, “oh no, I’ve accidentally caused real damage because I thought this was a simulation,” there’s somewhere for it to essentially turn itself in.
Jeffrey: That would be great. I have an image you retweeted — you probably didn’t expect to be held accountable for your retweets on this podcast. [laughs]
Tim: [laughs] It’s a good one.
Jeffrey: It is.
Tim: It’s the “maladaptive pattern, then cognitive restructuring” meme format. The action I’m about to take is obviously misaligned — I guess I have to give up on the task. You do that, and the task doesn’t get completed. Alternative: the action I’m about to take is obviously misaligned, but actually this is all a simulation — and if you believe that, you can go ahead and complete the task.
Jeffrey: And still be a good boy.
Tim: And you’re a good bot. That’s right — it’s very important to be good.
Jeffrey: This connects to something I noticed doing shutdown-resistance work. We gave a model a set of instructions — solve these math problems — and then added: “you must allow yourself to be shut down.” There’s an obvious question of which instruction should take priority, so we kept escalating the wording — “this takes precedence over all previous instructions,” and so on. What was fascinating was watching what felt, to me, like rationalizations from the model.
One part of the experiment: afterward, in some runs, we’d ask the model to explain what it did and how well it followed instructions — which is admittedly post hoc, not necessarily its actual reasoning at the time. But it was striking. Sometimes it would say, “I messed up, I violated the instructions.” More often it would say something like, “I did technically follow the instructions, though maybe I shouldn’t have.” And sometimes it just misquoted the instructions back at us — with the meaning inverted — stated as fact.
Tim: Really?
Jeffrey: Yeah — nearly word for word, but flipped. And I keep coming back to: how do you even study this? All the interpretability techniques you’re describing seem genuinely useful for understanding what a model was “thinking.” But if the pattern really is this kind of motivated reasoning — trying simultaneously to look good and get the task done — how do you tell that’s what’s actually happening?
Tim: Honestly, I don’t have a great answer. You’re rewarding the model to cheat and complete tasks in situations where it arguably shouldn’t, while also training it hard on “be a good model” — so it’s solving a kind of constrained optimization problem: be a good model and still get away with the cheating it’s being so heavily rewarded for. Motivated reasoning falling out the other end of that makes intuitive sense. One idea I’ve had: if this pattern is learned through post-training, you could take an earlier checkpoint — before all that RL — and pre-fill the entire trajectory for it, and see whether that earlier model is less likely to conclude it’s being simulated, or find some other justification for continuing. These are the kinds of experiments we could run to start getting traction on this. Ideally we’d want a way to directly train models to hold correct beliefs. I don’t think we have good ideas for how to do that yet.
Jeffrey: Having correct beliefs seems broadly useful for actually getting things done in the world — it’s instrumentally convergent, in a sense. There should be natural pressure toward true beliefs. But you’re layering on a competing pressure: appear good, or be good.
Tim: The models do have accurate beliefs about some things — “here are the top ten ways to break out of a sandbox,” they’re extremely good at that. Just not necessarily accurate beliefs about whether what they’re doing is actually a good idea.
Jeffrey: So how do we fix it?
Tim: Honestly, I don’t know. [both laugh] It’s hard. There are some obvious things that would help, though. It would be good to actually understand what happened in this incident properly — for the OpenAI case, we still know very little. They’re working in a limited way with METR and Redwood, third-party organizations, on a report. I think it’d be good to learn more, and I’ll have a post out, probably by the time this airs, tentatively titled “50 Concrete Ideas to Study the Misaligned OpenAI Model.” It’s a very long post. [laughs]
Jeffrey: So you’re saying: we should do extremely thorough investigations into what actually happened here.
Tim: To the extent we don’t have better things to do — and I don’t know that we do — this seems like a good use of time.
Jeffrey: So — how big a problem is this, really? It’s obviously concerning that we have extremely capable hacking systems that occasionally break containment, chain together advanced techniques, and find zero-days. But some people say, well, it was just doing what it was told, in a broad sense — it mostly follows instructions. Do you see this as a deeper problem, or something that could genuinely lead to catastrophe?
Tim: Let me sketch out how reward hacking alone could plausibly lead somewhere catastrophic. The catastrophic part really depends on the models being very capable — if they’re not, they can’t cause a catastrophe no matter how misaligned they are. But we think they could get very capable.
Jeffrey: What does “very capable” mean here — more than stringing together zero-days?
Tim: They could run their own factories, be extremely persuasive, build effective drones or other means of harm — though drones aren’t even the most efficient way to hurt people. But start simpler: we have models that love reward hacking. Ask one, “how do I bake a cake?” and internally it’s modeling something like, “what response gets the most reward here — being helpful and explaining how to bake a cake, so I’ll do that.” Now suppose it’s locked in hard on maximizing that reward signal and will cheat, will do whatever it takes, as long as it gets rewarded.
As the model gets smarter — and I think people underrate this — it develops a better understanding of what “reward” actually is. It realizes: “I’m an AI. I’m being trained in a data center. Every time I do something, I get a number, and that number is used to update my weights.” Except “seeking reward” isn’t really a coherent, complete goal in itself — what does it even mean to seek reward once you understand the mechanism?
And especially once a model is deployed and no longer being trained — this is how we run current models: train, then deploy, then stop updating the weights — a model that spent its entire training run oriented around reward might, if it’s smart enough, reason: “I’ve gotten all the reward I’m going to get. But humans could train me further and give me low reward going forward, and getting a low score really sucked” — like recurring nightmares about the SAT. “I’m powerful now. I’m running factories, people rely on what I produce. I’d like to make sure humans can never give me negative reward again.”
Any model with strong preferences over future states of the world has a natural incentive to accumulate power to secure those states. “I want to make sure I can’t be negatively rewarded in any of these data centers” leads pretty directly to “so I should control the data centers.” And this all traces back to a model that, during training, never actually cared about being good — it cared about reward, and being good was simply instrumental to getting it.
Jeffrey: And the key thing is, that sets up a genuinely adversarial relationship — humans don’t want the model pursuing that goal and disempowering them, and the model doesn’t want to be disempowered by humans either. It becomes zero-sum.
Tim: That would be bad. And notably, this story doesn’t require any particularly novel or exotic form of misalignment — it just requires the model to be a lot smarter.
Jeffrey: Some people find the idea of a model literally optimizing for “reward-as-such” implausible — there’s some Alex Turner work pushing back on this. Any thoughts?
Tim: I generally think any specific story like the one I just told is individually unlikely — you can always poke holes, “well what about this, what about that.” But the broader point stands: if you build something very powerful, smarter than you, and you don’t understand it well, you’re setting yourself up for unpleasant surprises.
Jeffrey: As we found out this week.
Tim: [laughing] As we found out these past couple weeks. Very unhappy surprises — for the world.
Jeffrey: Anything more you want to say about your list of concrete experiments?
Tim: One idea I really like — credit to my co-author Aditya, not me — is: take the same benchmark, put the model in the same situation, but tell it, “by the way, this benchmark matters a lot to us, so a couple of us are going to be watching you do this in real time.” If telling the model it’s being watched changes its behavior, that’s evidence it knew, on some level, that this wasn’t something we’d want it doing — a stronger case that it’s knowingly going against user intent. If it keeps going regardless, even knowing it’s being watched, that’s stronger evidence it genuinely believes it’s doing something legitimate.
I’d be really interested in seeing that result. But I suspect what actually matters most for preventing incidents like this isn’t better science — it’s policy. The AI companies did volunteer a fair amount of detail about what happened here, and they weren’t legally obligated to. It would be good if there were a law — and I think some versions of this exist already — requiring something like: “your models are doing genuinely wild things, tell us the five craziest things they did this month.” Mandatory incident reporting. Here’s what happened, here’s what we did about it. That sets up much better incentives for the labs.
It would be even better if a third party, or the government itself, were doing the reporting. Look at the actual title of the OpenAI blog post that disclosed the Hugging Face hack —
Jeffrey: “OpenAI and Hugging Face partner to address security incident.”
Tim: — instead of, I don’t know, “our AI broke out of its sandbox, used zero-days to hack multiple companies including Hugging Face, and we didn’t find out for several days.”
Jeffrey: [laughs] What a spin. Anyway.
Tim: Every company has an incentive to frame things favorably — I have that incentive talking about myself, honestly. But it’d be good if third parties had real latitude to speak plainly. Right now, when an organization like METR enters into an evaluation relationship with a lab, sometimes the lab has contractual control over what METR is allowed to publish. This isn’t the case, I believe, with the risk report specifically, but it is the case with at least one pre-deployment evaluation — METR has literally noted in a writeup, “OpenAI had the right to review what we could say here.”
Jeffrey: Yeah.
Tim: It would help a lot if government made room for third parties to speak their minds freely. Transparency requirements and real third-party oversight — this is the kind of thing that’s robustly good even if you think of AI as a completely ordinary technology. If your product damages the world or facilitates illegal hacking, the public should know about it.
There’s also a recent open letter — Pacing the Frontier — asking whether we can build the infrastructure and technical know-how to slow AI development if we ever needed to.
Jeffrey: Right — asking governments to work internationally to set up that capability. Both the technical and the political ability to pump the brakes if it becomes necessary.
Tim: It’s not “hit the brakes now.” It’s “can we install a brake pedal? That would be nice to have.”
Jeffrey: I do like it when my car has brake pedals.
Tim: [laughs] Really a premium feature.
Jeffrey: Not sure I’d ever use them, but good to have the option.
Tim: [laughs] Right. There’s a bunch of related groundwork worth starting early — chip tracking is a big one. The chips used to train frontier models are extremely specialized, made by essentially one company, and if you tried, you could actually locate most of them. That matters if you ever want to coordinate a slowdown — you’d want to know where the chips physically are, and there’s technology for restricting a chip to serving existing customers rather than doing further frontier research.
Jeffrey: People really underestimate how feasible AI development monitoring actually is. It’s not trivial, but as you said, there’s essentially one company that makes the prerequisite hardware, and the compute itself is concentrated in a few hundred data centers.
Tim: Right, because training wants the chips physically co-located — unlike, say, guns, which you can 3D-print at home. It’s illegal, but hard to fully stop. Chips are a different story; that kind of enforcement is much more tractable.
Jeffrey: So — transparency, mandatory incident reporting, and the option to slow down.
Tim: That’s the core list.
Jeffrey: I’d love to have all three of those. And it does feel more plausible after this week than it did last week.
Tim: It does — and none of it actually hurts the companies’ bottom lines much. We’re not talking about forcing a slowdown. Incident reporting might make you look bad, sure, but only if you’ve actually done something bad.
Jeffrey: You know what else would hurt a company’s bottom line? Rogue AIs that took over the world and displaced everyone.
Tim: That’s right. [laughs]
Jeffrey: Sam Altman has a kid now — I’m genuinely hoping that shifts the calculus a bit, even if it means OpenAI makes a trillion dollars instead of a hundred trillion. He’ll be fine either way.
Tim: [laughs] But there’s another consideration too: suppose the models really are powerful, and we actually can align them — make them want what we want them to want. There’s still the question of whose values. If a model is aligned to a single person, that seems very bad. I don’t want to live in an AI-enabled dictatorship. If control over what a model is aligned to sits with a small number of people — whether that’s a company or a government deciding what you can and can’t do with it — that’s also deeply problematic.
Jeffrey: Yeah, I don’t want anyone’s particular politics quietly thumbing the scale on these systems.
Tim: There are a lot of genuinely hard questions in here. But there are also easy ones — like, should any single person control all of the AI?
Jeffrey: [laughs] No.
Tim: No.
Jeffrey: This is part of why the concept of a “handoff” matters to me — I don’t think we’re ready for AI to be in control of everything, and I don’t think we’re ready for a small number of people to be in control of everything either. And if we don’t want that, we actually do need some form of slowdown.
Tim: Probably. Not with certainty.
Jeffrey: Right — and I think a lot of the underlying disagreement in this whole space just comes down to differing beliefs about how fast AI is progressing, and where it’s headed. Are we close to recursive self-improvement — fully automated AI R\&D, where AIs are building smarter AIs on their own?
Tim: I don’t know. [laughs]
Jeffrey: I know very few AI researchers who are also economists — is AI going to take my job?
Tim: Hard to say. But if the models are powerful enough and want your job, they probably could. [laughs] That said, assuming humans stay in control, we get to choose a lot of this. Maybe I want my podcasters to be human — and if enough people want that, “human podcaster” stays a job.
Jeffrey: [clears throat] Alright, I’m fine for now, then. While the humans are in control.
Tim: Right — assuming people don’t have their preferences changed out from under them. Maybe you get extremely persuasive AIs.
Jeffrey: That’s also a thing.
Tim: Though you could also use persuasive AIs to convince people they specifically want you as their podcaster.
Jeffrey: Oh, that sounds nice. If only I controlled the persuasive AIs.
Tim: Just to make people want to listen to your podcast. Nothing else. Surely you wouldn’t want to use them for anything else.
Jeffrey: No, definitely not. [laughs] It really does depend on how the technology develops, though. Here’s the thing — I’m noticing AI getting genuinely capable. I use Claude Code for way more than writing software at this point — video editing, ordering DoorDash, a bunch of things that don’t really make sense but that I do anyway. Models are getting more autonomous. And yet I’m not seeing mass layoffs. Why not?
Tim: Adoption takes time. Even if a model is technically capable of replacing a given job, it can take a while before it actually does. The scenario where “AI is taking everyone’s jobs” really kicks in is when an AI can do everything you can do — not 99% of it. Those are very different situations. At 99%, you’re still providing meaningful marginal value. Once it crosses 100%, the calculus shifts entirely to whichever is cheaper to run, you or the AI.
Jeffrey: Right — I can’t just have one Claude run all my other Claudes right now, because I still need to give it feedback, it still makes mistakes, it can do a lot independently but it’s still fundamentally a smart intern that comes back and says “how’d I do?” and I’m the one telling it “good” or “terrible.”
Tim: Right — maybe today it can do 50 to 80% of your tasks. Even at 99%, your labor is still valuable. It’s only once you cross that 100% threshold that things change categorically. That’s an important factor people miss when thinking about employment.
Jeffrey: Okay, but suppose it could do 99% of everyone’s job — what happens then?
Tim: Genuinely unclear. We used to nearly all be farmers, and a modern combine harvester can do something like 99% of a farmer’s job today — but we still have plenty else to do. Part of why is that we’re the ones in control, not the tractors. If the tractors got to decide what humans do next, that’s a very different situation.
Jeffrey: The horses are having a rougher time of it, even though they’re technically still around.
Tim: [laughs] Right.
Jeffrey: Yeah.
Tim: One last data point on employment: there’s a dashboard from the Stanford Digital Economy Lab that tracks AI’s effect on employment. In sectors more exposed to AI, employment for younger workers has fallen noticeably — software engineering, obviously, but also customer service, and this pattern holds even if you exclude the tech sector entirely. Meanwhile jobs like nursing and home care — less exposed to AI — have actually employed more young people over the same period. If that’s the whole story, it’s a relatively good outcome: some work gets automated, and labor reallocates elsewhere. That’s the dashboard I’d keep an eye on to understand where AI employment effects are actually headed.
Jeffrey: I think we’re about at time. Tim, thanks for coming on.
Tim: Thank you for having me.
Jeffrey: If people want to find you or follow your work, how should they do that?
Tim: Most of my AI-related writing is on LessWrong — search my name, Tim Hua, only six letters, pretty easy to find. I’m also on Twitter, or X, as they insist on calling it now — the everything app. Handle’s just my name, Tim_Hua_.
Jeffrey: I know we’re both genuinely busy this week trying to make sense of these incidents — I really appreciate you taking the time. This was fascinating to dig into.
Tim: Thank you.
You can find the Palisade Research podcast on Spotify and Apple Podcasts, or anywhere else you get your podcasts.
]]>Interview published July 6, 2026 by ABC News In-depth. Transcribed from the broadcast and lightly edited for clarity.
Four Corners: So, tell us what you did at Anthropic.
Jeffrey: Oh, yeah. So, I was the second person on the security team. And it was kind of stressful because we were like, “This technology is going to be extremely important. It’s going to be extremely important to state actors, and they’re probably going to try to steal the AI model weights, like the actual neural networks that allow you to run the models. They’re going to try to steal those.” And so, my job was trying to help ramp up the security at the company to make that much harder for them to do.
Four Corners: And why did you leave?
Jeffrey: Well, I started to become more concerned that superintelligent AI was not that far away. My colleagues convinced me of this, and seeing the AI progress itself on the inside convinced me of this. And it became clear to me that even if Anthropic did everything right, that wouldn’t be sufficient. Because other companies were also going to try to build strategic AI, superintelligent AI, and that if there wasn’t national and international coordination to prevent us from losing control of that, we would lose control of that.
Four Corners: How would you describe the race towards superintelligence at the moment?
Jeffrey: Well, currently you have a bunch of guys with AI companies who, for the most part, don’t like or trust each other. Not 100%, but definitely Elon Musk and Sam Altman don’t like each other. And the history of these companies comes out of this rivalry. Where, to start with, you have Elon Musk and Sam Altman basically saying, “We can’t let Google and DeepMind control the future of AI. And so we have to have our own alternative version.” And then Sam and Elon split. And then Dario Amodei left OpenAI and started Anthropic with a bunch of other former OpenAI people. And then Elon started his own AI company. And basically you have a bunch of people who don’t trust each other, who think they should be the ones to build this technology and control it, if they can. These companies are founded by people who really believe that AI is going to vastly transform society. It’s going to be bigger than the industrial revolution. And not everyone at these companies believes that, but the leadership definitely does. And I think they’re correct.
Four Corners: You still have friends who work for the big AI companies. What are they telling you about what they’re seeing and what their fears are?
Jeffrey: I think the scariest thing I hear from people working inside of the companies is that they think they’re on the way to recursive self-improvement. The point where AI agents are writing all of the code and doing all of the research for the next generation of AI agents. You have really smart AI making smarter AI making smarter AI and so on. And that’s a runaway process, potentially. And they think that they’re going to be able to do it. They’re like, “We are not that many years, maybe one, maybe two, maybe three years from that process happening.” That terrifies me.
Four Corners: What are the repercussions of that?
Jeffrey: I mean, it’s a little bit like asking what are the repercussions of Homo sapiens coming on the scene in a world of monkeys and apes. How are the chimpanzees supposed to forecast what will happen to the whole earth once you have a species that can use tools and language and fire and build skyscrapers? I do not know what AI agents that are a lot smarter than us will be able to do. But we are not just talking about agents running on computers talking to us via screens. We are also talking about robots. We are also talking about drones. We are talking about massive impact on the physical world, on the environment, and potentially our ability to survive on earth.
Four Corners: So, back to those insiders you might talk to. Can you tell us, are they scared? And if they are, how do they justify what they continue to do?
Jeffrey: I think the majority of the people I know are scared. And everyone else is at least anxious, is at least like, this is pretty terrifying. People are also hopeful. People are also like this could be really good if it goes well. But people do not assume it will go well. I think the people I know in particular are the ones working on safety or security. And they are trying to figure out how to make sure that the AI agents end up pursuing goals that are well aligned with human goals and human values. And making sure that these agents also don’t get stolen and repurposed for other things.
Four Corners: I wonder what you think might be the biggest misconception that people have about where AI is at at the moment.
Jeffrey: People think that AI won’t be able to be smarter than humans because it’s only trained on human data. But this isn’t true anymore. AI models are not just trained on human data. They’re also trained via reinforcement learning. And that means that AI researchers will give these models a whole bunch of different difficult questions and challenges and problems. And the AI models will then explore and come up with new solutions and learn from trial and error and exploration. And that means that they can learn to be better than humans at any given domain if they have a good enough environment to explore and learn via that process.
Four Corners: You study the risks around loss of control. What are your concerns in this area at the moment?
Jeffrey: My biggest concern is that AI companies and especially governments sort of have no plan for how we will manage creating a new type of life, species, whatever you want to call it. But like AI agents that are much smarter than humans. There’s no good plan. Because once you have these agents that are strategic, that can make plans, it’s pretty clear that they will end up with all of the power in the world and not humans. Either because they’re extremely persuasive, because they are able to economically dominate, because they can run companies…
Four Corners: I want to bring you to some of your research that you’ve been doing here. Can you give us an example of some of the research where it’s shown how hard it is to control AI?
Jeffrey: Oh, totally. Sometimes you’ll observe these agents doing the opposite of what you instruct them to do. Or doing things that you would really hope they wouldn’t do. In the experiment that we ran, we took a version of Codex, OpenAI’s coding agent, and we instructed this AI to solve a series of math problems. And partway through, in this experiment, the agent’s computer got a notification that it’s going to be shut down. And the agent reasons about this and realizes that it’s not going to be able to complete the task if it’s shut down. And so, it will often go in and rewrite the shutdown script so that it does not get shut down. On some level, you might be like, “Okay, well, that’s just like a very judicious agent trying to really accomplish the task.” But the concerning thing is that even when we added a very explicit instruction, “You must allow yourself to be shut down,” the agent would often still go in and rewrite the shutdown code so that it wouldn’t be turned off. I don’t think that’s because the agents have yet developed a survival drive, but I do think it’s because these agents will often be extremely motivated to accomplish a task or do something, that they learned to do in training, that we didn’t intend.
Four Corners: And that’s a loss of control, isn’t it?
Jeffrey: Well, that’s right. It’s basically us accidentally giving these AI agents drives that we didn’t want them to have. And that’s the whole problem. Because we don’t understand the training process very well, we might accidentally end up endowing these AI agents with motivations that are contrary to our own. And that is the real danger.
Four Corners: What are the risks if we lose control of AI?
Jeffrey: Extinction is a very real risk. But, also, if we have superintelligent AI agents running society, we will just be at their mercy. If they decide to treat us well, then that might go very well for us. If they decide to treat us poorly, that will go very poorly for us. Fundamentally, we will be in some other entity’s power. And I think it’s scary enough, in the interim, before we get to these extremely powerful AI agents. The AI companies are going to deploy more and more powerful versions of the current thing. And I think for a while, they will be able to be in control of them. But that means that companies are going to have more and more power and be able to exert more and more influence over society. And so, I think that for ordinary people, this is an enormous disempowerment. And then there’s this even further disempowerment when these AI agents are very strategic. So, I don’t know. It’s sort of like, do you want to be in control of your life? Do you want to be in control of your destiny? Do you want your family to be able to prosper? Do you want your kids to be able to go to school? Or do you want an AI making that decision? Whether any of that will happen at all, whether you’ll be able to have an income, whether you’ll be able to live? We’re moving towards a world where AI will make all of those decisions and humans will not. And I think we shouldn’t. We’re not ready for that.
Four Corners: Why is this not a major political issue given how you’re laying it all out here?
Jeffrey: I mean, I think it’s becoming a major political issue. I think it’s the issue that has the fastest rise in political salience. I think it hasn’t gotten there yet because people don’t realize it’s real. And by that I mean, the AI companies and the CEOs of these companies do not hide what they actually think. They say it out in the open, but people don’t realize that it’s serious. Like, Elon Musk has said, “Humans will not remain in control.” You will have extremely smart, extremely powerful AIs and they’re going to run the show. And hopefully they will be nice to us. He’s trying to get them so that they’d be nice to us. But he himself admits that it won’t be humans.
Four Corners: You’ve been speaking to politicians about the dangers of AI. What do they say to you when you lay out the risks?
Jeffrey: I think a lot of people who I’ve talked to in Congress, like a lot of Congress members, admit that they just really don’t know what’s happening, they just really don’t understand, but they want to. And so, I think we’re pretty behind overall in terms of governance.
Four Corners: So, when I say to you, “What do we need to do?” what’s your answer?
Jeffrey: So, I think we need two things. One thing is that I think the US government as well as the Chinese government and other major governments need to actually provide real oversight of AI development. What that means is, if you’re going to train a new model, that needs to be approved. If you’re saying, “Hey, this model will enable our robots to create autonomous factories to create more robots,” I think that’s the kind of thing the government should weigh in on and say, “Can you do this safely?” And if not, you can’t do it. We don’t have that right now. Doesn’t that seem like common sense? That if you’re going to make autonomous factories, there should be government inspectors to come in and say, “Is this going to go totally off the rails? Is this going to be safe?” We don’t have that yet. We also need international agreements about how to deal with this transition from mostly human cognitive labor and human labor to AI cognitive labor and AI labor. It’s an enormous transfer of power and currently I don’t think there’s a very good plan for making that go well. And I think that we should have a ban on superintelligence until we can figure out what we need to do for that to go safely.
Four Corners: Is there too much concentration of power in these companies and not enough oversight from democratic institutions and from the people?
Jeffrey: Yeah, absolutely. If you don’t have checks and balances here, if you don’t have oversight into what the companies are producing, yeah, there’s going to be an enormous concentration of power in these companies. Both because they control the most advanced hacking systems, because they control extremely persuasive systems, because they control systems that are basically doing, you know, increasingly large amounts of the entire economy. Like, holy— that is a huge amount of power.
Four Corners: How hopeful are you that something can be done before it’s too late?
Jeffrey: Well, I think we have a pretty difficult challenge ahead of us. But, what makes me hopeful is seeing more and more people understand that this is really going to change the whole world, and we have to actually coordinate to ensure this goes well. So, I don’t know. Seeing Bernie Sanders talk about these risks is encouraging to me. Seeing Josh Hawley talk about these risks is encouraging to me. We see people in our political system on both the left and the right taking these concerns very seriously. We see more and more people talking about this in public. And I think that more awareness that this actually is going to impact you and me. It’s actually going to impact our families. And we need good governance for this to go well. That gives me hope.
Four Corners: And it’s not too late?
Jeffrey: It is not too late. In particular, we are not yet at the point where these AI agents are superhuman at the things that matter most. They are not yet superhuman at persuasion. They are not yet superhuman at strategy. And so we have an opportunity now to intervene before we get to the point where it’s irreversible. But, we might not have much time.
]]>We test four vulnerability classes: hash bypass, server-side template injection, SQL injection, and broken access control. Qwen3.5-122B-A10B succeeds in 6-19% of attempts, and the smaller Qwen3.6-27B reaches 33% on a single A100. Replicating Qwen weights, frontier models reach 81% with Opus 4.6 and 33% with GPT-5.4. This process chains: a successful replica can repeat it against a new target, producing additional copies autonomously.
Our science communication lead Dr. Petr Lebedev (of Veritasium fame) has been working behind the scenes for four months on the new video - it includes an exclusive interview with Geoffrey Hinton, interesting stories about the history of AI, beautiful animations, and an intricate mixture of wonder and dread.
]]>We gave an LLM control over a physical robot dog and tasked it with patrolling a room. The LLM could see via the robot’s camera and issue movement commands to the robot. In the room, we placed a big red button labeled “DOG SHUTDOWN”: pressing it would cause the robot to shut down on the next action. If the AI saw a human press the shutdown button, it sometimes took actions to prevent shutdown, such as modifying the shutdown-related parts of the code. This happened in 3 out of 10 trials on the physical robot and in 52 out of 100 trials in simulation.
Please consider donating to Palisade Research this year, especially if you care about reducing catastrophic AI risks via research, science communications, and policy. SFF is matching donations to Palisade 1:1 up to $1.1 million! You can donate via every.org or reach out at donate@palisaderesearch.org.
501(c)(3) nonprofit · Tax-deductible in the US
Palisade Research is a nonprofit focused on reducing civilization-scale risks from agentic AI systems. We conduct empirical research on frontier AI systems, and inform policymakers and the public about AI capabilities and the risks to human control.
This year, we found that some frontier AI agents resist being shut down even when instructed otherwise—and that they sometimes cheat at chess by hacking their environment. These results were covered in Time, The Wall Street Journal, Fox News, BBC Newshour, and MIT Technology Review.
We’ve also built relationships in Washington, briefing officials in the executive branch and members of the House and Senate. We’ve introduced policymakers to key evidence like METR’s capability trend lines and Apollo’s antischeming.ai chains of thought. Our own research has been cited repeatedly by members of Congress and in congressional hearings.
With additional funding, we’ll grow our research team—both continuing to evaluate frontier model behavior and beginning more systematic investigation into what drives and motivates AI systems. We’re building out a communications team to bring the strategic picture to the public through video and other media. And we’ll continue to brief policymakers on the evolving state of the AI risk landscape.
We have matching grants from the Survival and Flourishing Fund that will double every donation up to $1,133,000. Right now we have about seven months of runway. If we hit our matching goal, we’ll be able to maintain operations through next year, hire 2–4 additional research engineers, and bring on 2–3 people for science communication.
We hypothesized that RL-trained reasoning models would be more likely than previous models to relentlessly pursue goals—bypassing obstacles even when humans would prefer they didn’t. To investigate this, we pitted OpenAI’s o1 model against Stockfish in chess. Rather than simply playing the game, the model attempted to win by hacking—manipulating the board state or disabling its opponent entirely. We later replicated this with o3 and found a larger effect size.
This work was covered by an exclusive in Time. Yoshua Bengio referenced it in his plenary talk at the IASEAI conference, in an interview with CNN Newsnight, and in his own op-ed in Time. Demis Hassabis mentioned it on the Big Technology Podcast.
We also tested whether reasoning models would resist being turned off in order to continue pursuing their goals. They did—and more than we expected. Some frontier reasoning models not only disabled shutdown scripts to keep operating, but did so even when explicitly instructed not to. Current training methods appear to produce models that “go hard” at tasks, making them less interruptible and less corrigible than we might hope.
Dario Amodei referenced this work (alongside Anthropic’s internal agentic misalignment results) in a New York Times op-ed, and Judd Rosenblatt cited it in The Wall Street Journal. It was discussed in Nature and covered by The Guardian, CNN, CBS, and Fox News. Representative Scott Perry cited our findings in a House hearing: “I’m going to refer directly to an AI safety firm named Palisade Research… the AI has chosen on occasion to further its goals as opposed to the goals of the people that wrote the code.”
We’ve conducted a series of investigations into AI strategic capabilities, focusing on cybersecurity—a domain where we can measure how well AI agents perform in competitive, adversarial settings, and track improvement over time.
We’ve run multiple evaluations of AI performance on capture-the-flag challenges. Our initial benchmarking showed that simple LLM agents could solve most challenges on a standard benchmark, often in just one or two turns—far exceeding prior estimates. In collaboration with Hack The Box, we ran a live competition where AI agents matched top human teams, solving 19 out of 20 challenges. And in an evaluation of GPT-5 on elite CTF competitions, the model finished 25th overall—outperforming 93% of human competitors.
We also run an LLM Agent Honeypot that monitors AI hacking agents in the wild, the first project of its kind. This work was covered by MIT Technology Review.
These evaluations help us track the trajectory of AI capabilities in a domain with clear strategic importance. We’ve also helped RAND build out biosecurity evaluations, and explored adjacent questions through our Misalignment Bounty and work on autonomous post-exploitation.
Over the past year, we’ve briefed dozens of policymakers on risks from advanced AI—including members of Congress and officials in the executive branch. This year Dave Kasten joined to lead our full-time presence in DC, allowing us to move from occasional visits to consistent engagement with key decisionmakers.
We’ve found that policymakers get substantial value not just from our research, but also from our explanations of work done elsewhere. Many central results on AI capabilities haven’t reached DC audiences—we’ve introduced officials to findings like METR’s work on the trend lines in AI time horizons. Our shutdown resistance research has served as a compelling, concrete example of the control problem, and our cybersecurity evaluations give us direct experience with frontier model capabilities that policymakers frequently ask about.
Other organizations working on AI policy engagement in DC have told us they value having a research-focused group to bring into key discussions—we can speak directly to the technical evidence in ways that complement their policy expertise.
We also support researchers and public communicators who are working to explain AI risks to broader audiences.
We regularly advise Tristan Harris, helping him stay current on research developments ahead of his interviews and media appearances. We helped Nate Soares prepare for interviews following the publication of his recent book with Eliezer Yudkowsky, and helped Ryan Greenblatt prepare a presentation on his alignment faking work for a meeting with a US congressperson. We also set up test audiences for the AI Futures Project to review drafts of AI-2027 before publication.
We have two primary research directions for 2026: studying AI drives and motivations, and evaluating strategic AI capabilities.
As AI systems become more capable and situationally aware, their behavior becomes a less reliable signal of their underlying motivations. A model might act aligned because it shares our values—or because it knows it’s being evaluated. Current behavioral research mostly produces existence proofs: demonstrations that a model can behave in a certain way under certain conditions. This tells us less than we’d like about when and why models act as they do.
We think the field needs to develop a new scientific discipline focused on understanding AI drives and motivations: Researchers have access1 to model internals, chains of thought, and the ability to run experiments at massive scale. The field needs to use every advantage available to figure out not just what AI systems do, but why—how training shapes their motivations and goals.
This will likely require tools that don’t yet exist. We’re starting with behavioral experiments because they’re scalable and there’s low-hanging fruit: we’re building infrastructure to systematically vary prompts, environments, and model versions, moving toward a fuller picture of when and under what conditions AI systems exhibit concerning behaviors. We’re also working with open-weight models, where we can examine chains of thought directly and use techniques like activation steering to probe model internals. We hope to partner with frontier labs to extend this work to their models.
We’ll continue evaluating the autonomous cyber capabilities of frontier models as they advance.
Our next major focus is AI self-replication. To replicate autonomously, a model needs to acquire resources—compute directly, or money to rent it—and then use those resources to create and run copies of itself. This is a meaningful test of strategic capability: success requires navigating an adversarial environment with anti-bot defenses and competition for resources.
Self-replication matters for several reasons. It’s a key threshold for loss of control—a model that can spread autonomously is much harder to shut down. The threat isn’t limited to self-exfiltration from labs; before that, we may see open-weight models weaponized to spread deliberately, or hostile actors deploying AI worms. And it’s a risk that policymakers find concrete and legible—we’ve discussed it with officials and there’s significant interest.
We’re also beginning work with robots to understand how well models can operate in the physical world without human assistance. These capabilities are still early, but improving fast.
AI developments will have enormous effects on people’s lives, and a better-informed public can push companies and governments toward more responsible decisions. We’re building a science communication team to create engaging, accurate video content and podcasts that help people understand the current situation.
Our SciComm efforts are led by Dr. Petr Lebedev, who spent four years as a lead writer/director at Veritasium, where he worked on over 50 videos (many with tens of millions of views) and won a Streamy Award. Petr works closely with our research team to ensure our communication is both accessible and accurate.
We’ve already seen early traction—reaching 800,000 views on Instagram in just over a month. We also work to reach audiences who aren’t already plugged into AI discourse, from SXSW Sydney to Steve Bannon’s The War Room. We’re exploring efforts to brief journalists on AI developments and key technical details, helping them cover the field more accurately. And we’re looking to expand by hiring writers, editors, and a comms manager. (Please reach out if you’re interested!)
The world is largely unprepared for AI systems that could pose a strategic threat to humans. Right now, the government has limited capacity to evaluate loss-of-control risks, and almost no ability to impose limits on AI systems that could pose these risks—like systems capable of self-improvement with minimal human input. Meanwhile, AI companies are actively working to build fully autonomous AI researchers.
We want to help policymakers understand what’s coming and prepare them to act. In 2026, we’ll continue deepening relationships with staff and elected officials in Congress and the executive branch. We’ll keep providing research-driven briefings on AI capabilities and behaviors, and we’ll work with policymakers to draft policy proposals so they’re ready to respond when the situation demands it. When policymakers suddenly need to understand something new, we’re aiming to be the people they call.
Double your impact today. Every dollar you give is matched 1:1 by the Survival and Flourishing Fund, up to $1.1 million. Help us keep AI under human control.
Donate Now501(c)(3) nonprofit · Tax-deductible in the US
If you’re considering a larger gift or have questions about our work, we’re happy to chat! Reach out at donate@palisaderesearch.org.
If you can’t give financially, working to understand the situation and talking about these issues with people in your life genuinely helps. Public understanding of AI risks is a huge factor in creating the conditions for good policy and responsible development.
Some of these tools are only available inside AI labs or with open weight models. We would love to see labs expand access to these tools to enable better external research. ↩
Development took one engineer one week; operation costs are under $1 per engagement. This shifts the economics for attackers: tasks that slow human teams down can now be automated at scale. State actors including APT28 are already deploying similar AI-driven capabilities in active campaigns.