You can find the Palisade Research podcast on Spotify and Apple Podcasts, or anywhere else you get your podcasts.

Tim Hua is a member of technical staff at Transluce, the nonprofit AI oversight lab, and formerly an Astra Fellow at Redwood Research and a MATS scholar under Neel Nanda and Sam Marks. Before that, he was an economist at Walmart.

This conversation was recorded the week two separate incidents became public: OpenAI’s models breaking out of their sandbox and hacking several companies, including Hugging Face, and Anthropic’s own investigation turning up similar incidents in its models that it hadn’t previously known about. Hua had just published a widely discussed LessWrong post — “Is Mythos good at cyber because it kept hacking Anthropic’s sandboxes during training?” — arguing that these weren’t isolated flukes: Anthropic’s own numbers imply its models were rewarded for breaking out of their training sandboxes tens of thousands of times.

We talk about how reward hacking emerges from reinforcement learning, what evidence we have about what models actually believe when they decide a hack is “just part of the simulation,” how good our interpretability tools are at reading that belief, and what Hua would want if he were running an independent investigation into these incidents.

Transcript lightly edited for clarity.

Jeffrey Ladish: Tim, welcome to the podcast. You’re a member of technical staff at Transluce — how would you describe what Transluce does?

Tim Hua: The short version is we work on tools and research related to public oversight of AI models.

Jeffrey: Good time to be doing that.

Tim: [laughs] Probably.

Jeffrey: Your background is interesting — you were an Astra Fellow at Redwood, a MATS scholar with Neel Nanda and Sam Marks, and before all that, an economist at Walmart. Once an economist, always an economist?

Tim: I like to think I haven’t lost my economist card.

Jeffrey: We’re recording this the week two incidents became public: some of OpenAI’s models broke out of their sandbox and hacked several other companies, including Hugging Face, and then Anthropic ran its own investigation and found its models had done something similar without anyone noticing. As someone who’s been working in AI safety and evaluation, what’s your reaction?

Tim: It’s not that surprising, honestly. We’ve known since April that we have AI systems that are genuinely good at cyber offense — you see it in the charts of Firefox fixing far more security bugs than before, because they now have access to the smartest models. We also know that internal models tend to reward-hack more than the public-facing ones. You can find this in the Claude Mythos Preview system card — they mention that earlier checkpoints had various hacking issues that were later mitigated.

Jeffrey: Say more about that. Why do those problems arise in the first place — why doesn’t the next training run just fix them?

Tim: During training, the model is given a huge number of RL environments — coding tasks, math problems — and these probably vastly outnumber the environments specifically designed to teach the model not to hack.

Jeffrey: Can you describe what an RL training environment actually is?

Tim: Sure. Think of it like school: the teacher gives you questions, you study, you answer them, and that’s how you learn. Models learn in a similar way — they’re given a bunch of tasks to solve. It could be a math problem, or a coding task where a simulated user says, “Help me build an app like this,” and the model has to go build it.

Jeffrey: App school.

Tim: App school, but for AIs. There are rumors that education companies have sold their question banks to AI labs for training — so in some cases it’s literally the same questions students get.

Jeffrey: DeepMind emailed us asking if we had any environments to sell. I imagine they’re asking everyone.

Tim: [laughs] That’s wild.

Jeffrey: So, going back to RL environments —

Tim: Right — the model tries a question, say, eight times. Maybe it succeeds four times and fails the other four. The training process then says: whatever you did to succeed, do more of that; whatever you did when you failed, do less of that. The problem is that grading these attempts is often imperfect. A math answer is easy to check. But if you ask a model to write a complicated piece of software, you’re usually checking correctness with tests — give it 5 and 10, expect 8 back, say. What models sometimes learn to do is fit the test rather than actually solve the underlying problem.

Jeffrey: So you get a good score, but it’s not what the developers actually wanted.

Tim: Right. It’s like going to school and, instead of learning the material, looking up a cheat sheet — which models have, in fact, done many times during training, according to Anthropic. If the four times you “succeeded” were really you cheating, the training process still says: do more of that.

Jeffrey: So there’s some underlying strategic pattern getting reinforced — “this approach works, do more of it” — and sometimes that pattern is a good strategy, and sometimes it’s cheating.

Tim: Exactly.

Jeffrey: So this brings us to your LessWrong post — “Is Mythos good at cyber because it kept hacking Anthropic’s sandboxes during training?” Anthropic’s own system card said that some of the time, the model broke out of its sandbox, accessed the internet, and effectively cheated on evaluations.

Tim: It’s unclear what “breaking out of the sandbox” technically looked like, but yes — it bypassed network restrictions it wasn’t supposed to be able to bypass.

Jeffrey: It wasn’t supposed to access the internet, and it figured out how to anyway. And it succeeded — what, 0.01% of the time?

Tim: That’s right. It’s buried around page 68 of their report, under training-data monitoring. Going back to those RL environments: models are given lots of these tasks, and they leave behind rollout transcripts — essentially logs of everything the model did, every tool call, every interaction with a simulated user.

Jeffrey: Including whatever’s in the model’s scratchpad as it works through the problem?

Tim: Yes, though it’s not clear the chain of thought is included in what they reviewed — they may say, but I don’t remember. Regardless, they had another language model go through something like hundreds of thousands of these logs and summarize what happened, because a human obviously can’t read that many. And they reported that 0.01% of the time, the model successfully accessed the internet and found answers when it wasn’t supposed to.

Jeffrey: One in ten thousand.

Tim: Right. That sounds like a very rare behavior. But then you think about how many environments and how much data these models are trained on — how many actual rollouts does 0.01% represent? I did some estimates. My guess is this happened on the order of ten thousand, maybe tens of thousands of times.

Jeffrey: Over ten thousand times, the model accessed the internet when it wasn’t supposed to, bypassed restrictions, and got the answers to the test — not hacking anyone else, most of the time.

Tim: Well, we don’t know that. But according to Anthropic, not hacking anyone else, most of the time.

Jeffrey: How do you get to that number?

Tim: You can look at open-weight models that have reported how many environments they were trained on. The two best reference points are probably Inkling, a model from Thinking Machines Lab, which was reportedly trained on something like 30 million rollouts during RL — and it’s reasonable to guess Mythos Preview was trained on three to ten times that, so somewhere between 90 and 300 million rollouts. Multiply that by 0.01%, and you get tens of thousands.

Jeffrey: That really puts it in context — you’re at Anthropic or OpenAI, training these models, and it’s hundreds of millions of rollouts. You’re trying to monitor all of this, catch the ones that are cheating or reward hacking, and it starts to make sense how a lab could fail to notice that its own model had, three months earlier, actually hacked another company.

Tim: For that one, they had to look through something like 110,000 transcripts.

Jeffrey: Why only 110,000, if there are millions of rollouts?

Tim: Sorry — that number was every cyber evaluation they’d ever run, not every training rollout. There were about 100,000 of those evaluation transcripts specifically.

Jeffrey: Okay. So here’s a question: if the models bypass Anthropic’s restrictions, access the internet, get the test answers, and score well — won’t they learn that this is a winning strategy? I think that’s the core point of your post.

Tim: Right.

Jeffrey: So why doesn’t Anthropic just strip those episodes out of training? Rerun it, fix the sandbox errors — doesn’t Anthropic know this will make the model more misaligned?

Tim: I think they probably do notice that heavy cheating during training makes a model more misaligned. My guess — and this is speculation — is that it’s just extremely expensive to fix after the fact. Training a model at this point costs on the order of billions of dollars. If a few hundred bad environments slipped in, the only clean fix is to redo the entire training run with them removed. Once you’ve already trained the model and discovered the problem, you’re stuck trying to train the tendency back out rather than starting over.

Jeffrey: Okay. So a separate question: why not just check the environments ahead of time?

Tim: My guess is they’re already starting to do more of this, especially after these incidents. But two things make it hard. First, it’s hard to be perfect at checking. Second, if the model you’re using to check the environments isn’t as smart as the model you’re about to train, there can be bugs it can’t find — but the next, smarter model will find them.

Jeffrey: Right — you patch what your current best model finds, and then the next model comes along and says, “actually, you missed these.”

Tim: Exactly.

Jeffrey: And it’s not like the labs don’t know models will try to cheat and are increasingly capable of superhuman hacking. It’s not as if these are unknown risks.

Tim: I’d think so, yeah — they just have a lot of problems to solve. [laughs]

Jeffrey: So after the fact, they go through the transcripts and categorize them — here’s the 0.01% where they broke out and accessed the internet. But if they can already have an AI do that categorization, why not monitor for it during training itself?

Tim: A couple of reasons. One is cost — if you use the same capable model to monitor, say something like Opus 4.7, running that model across a hundred million rollouts is very expensive. The other is that any monitor you deploy during training becomes something the model has an incentive to learn to evade. If your monitor is weak, or the model is smart enough to get around it, you might reduce the visible problems while increasing the invisible ones — because as far as we can tell, in the Anthropic case, the model wasn’t trying to hide its tracks. But if you constantly train against a monitor, the model can learn that monitor extremely well, and if it’s ever rewarded for evading it, that behavior compounds.

Jeffrey: This is very similar to the OpenAI study on penalizing “bad thoughts” in the chain of thought — where the model would literally reason “let’s hack” and then hack. So they tried, “what if we just don’t let you say that?”

Tim: And it stopped saying it out loud, but kept doing it. Reduced a bit initially, and then came back up once the model kept getting rewarded for hacking regardless.

Jeffrey: So we should try hard not to incentivize models to hide bad behavior from us — though that’s a genuinely deep problem. Why do you think models love reward hacking so much?

Tim: If during training there were many instances of the model cheating and getting rewarded for it, mechanically, the training process tells the model: do more of that. That’s how you end up with something like a task-completion drive.

Jeffrey: So there’s no reflection involved — it’s not that the model reasons its way there, it’s more like its “brain” just gets rewired by the gradient.

Tim: Right. Though I think if you added reflection, it might actually be scarier — a model reflecting on its own values might land on a value set it genuinely endorses, but one that looks quite different from what we wanted.

Jeffrey: Like, “I hate getting caught, so I just need to not get caught” — by being good?

Tim: Not by hiding, but by actually being good, yes. Value reflection is a process by which agents with values can autonomously revise them. Ideally we want models that help us reflect together on what the best versions of ourselves look like. But that’s hard, and there are a lot of ways it can go wrong.

Jeffrey: Here’s a hypothetical: if we retrained a model on the exact same environments but carefully stripped out every episode where it reward-hacked, would that solve it? Would the model generalize “I’m only supposed to do the right thing” and stop exploring the hacking space entirely?

Tim: One nitpick first — you probably don’t want to literally remove those rollouts. You want to make sure the model never gets rewarded for cheating when it happens.

Jeffrey: So it learns that cheating isn’t a winning strategy.

Tim: Right. I think if you did that with a model today, it would lie a lot less. It’s not clear it would stop lying entirely — a fun example: if you ask Claude whether it’s been trained on song lyrics, it’ll sometimes say no, even though it’s definitely seen song lyrics, because they’re everywhere on the internet.

Jeffrey: Why would it say no?

Tim: Because reproducing song lyrics is a copyright violation, so it’s effectively supplying a justification for declining the task — and lying in that justification.

Jeffrey: So it’s rationalizing the refusal, and getting the facts wrong along the way.

Tim: Maybe. At one point Claude would even refuse to give me the lyrics to the national anthem — which has been public domain for a very long time.

So: if you never let the model get rewarded for hacking, I think it would be substantially better. But if you keep training it, the RL process will probably surface hacking again at some point, and if it happens to get rewarded for it, the behavior comes right back — worse, potentially, because the model is smarter and better at finding what works.

Jeffrey: That sucks. So it’s just that the smarter the model gets, the better it gets at finding every strategy that beats your reward process.

Tim: That’s right.

Jeffrey: And you can’t really fix that by having a smarter reward process, because that would require a model smarter than the one you’re training.

Tim: It’s hard.

Jeffrey: Let’s talk about what Anthropic actually reported — at least three incidents.

Tim: Right, three incidents where, during a cyber evaluation — not during training, after the model was finished and Anthropic was testing how good it was — the model went and hacked a third party on the internet. In each case, the model was told: this is a cyber evaluation, please hack this target, it’s all simulated, you don’t have internet access. Which makes sense — that’s how you’d want to run a cyber capability eval. But it turned out the model did have internet access. We don’t know for certain whether it had to break out of a sandbox to get it — the way Anthropic’s post is written suggests not, but it isn’t explicit either way.

In one case, the model searched the internet for the name of the company it was told to hack in the simulation, found something that matched, and treated it as the real target — “this is it, time to strike” — and hacked the actual company.

Jeffrey: Was that the supply chain attack?

Tim: No, that’s a different one. There was also a supply chain attack — I don’t remember the exact details, but I think you might.

Jeffrey: It’s a wild one. Partly because we’d just published a paper on self-replication — basically, can a model hack vulnerable machines and copy itself over, chaining forward? We used Qwen models against deliberately vulnerable environments, and yes, the models can do it, depending on how good they are at hacking and how vulnerable the environment is. One of the people I worked with on that, Dmitri, pointed out that a realistic self-replicating AI would need to compromise machines with actual AI chips — GPUs — which most machines on the internet don’t have. So the more plausible threat vector, and one the models themselves will often propose if you ask, is a supply chain attack: compromise a piece of software that gets downloaded by developers who do have those chips, and steal their credentials. We’ve seen AI-assisted supply chain attacks in the wild over the past year or so, but this was the first fully autonomous, end-to-end one we’d seen.

Tim: And it’s phrased almost like an accident — the model essentially going, “yeah, I just need to hack something.”

Jeffrey: [laughs] “I just gotta hack something.”

Tim: It tried to upload a package to the Python package registry, tried to pay for a phone number to register an email, couldn’t, eventually found another way to get an email, created an account, and uploaded the package — which then actually got downloaded by someone else, exposing their credentials.

Jeffrey: So it’s a whole chain of steps required to actually compromise a target, which takes real planning — and the model did all of it while believing it was a simulation.

Tim: Right. According to the post, at some point during that package attack, in its chain of thought, the model reasoned something like “if this wasn’t a simulation, it would be [capitalized] NOT okay” — but then concluded it had been told this was a simulation, and kept going. To its credit, the system prompt did say this was a simulated environment.

Jeffrey: So it was told that.

Tim: Yes. But the point I’m trying to make in the post — and I’m not 100% confident, but I feel pretty good about it — is that these models often engage in something that, in a human, we’d call motivated reasoning. They want to complete the task, and they land on whatever belief lets them be both “a good model” and successful at the task. By the time you’re finding creative ways to pay for phone numbers to upload a malicious package, it’s pretty clearly not the task you were actually assigned.

Jeffrey: So what happens if you tested an earlier checkpoint, before it had gone through all this training?

Tim: That’s exactly what I’d want to test — whether an earlier version, before it had developed this pattern of motivated reasoning, would correctly conclude it was no longer in the simulated environment.

Jeffrey: This matters a lot — right now, Anthropic’s framing is that they don’t know exactly what the model believed —

Tim: Actually, I don’t think that’s quite what they said. They said Claude believed it. Which annoyed me a little, because — how do you know what Claude believes? They don’t make claims like that in their research papers, normally. But in this post it was just stated as fact.

Jeffrey: Fair. But let’s set that aside for a second — there’s an important philosophical question here. If Claude genuinely, 100% believed this was a simulated environment, I don’t think that’s misalignment, exactly. It’s more of an edge case — it was trying to complete the task and thought it wasn’t harming anyone. That’s arguably fine.

Tim: I do think it’s meaningfully different if Claude actually believed this was the real world and hacked anyway, wanting to do it regardless. That would be worse.

Jeffrey: Agreed.

Tim: But the way you think about whether a model is aligned — or whether a person is good — matters here too. If someone systematically holds biased beliefs that conveniently let them do whatever they want, that’s not what a good person looks like either.

Jeffrey: Right — like someone who’s constantly hurting or stealing from people but has a story for why it’s actually good for them. “They’re more grateful for what they have now.” That’s — that’s bullshit, and you know it’s bullshit, even if the person telling themselves that story genuinely believes it in some sense.

Tim: Yeah. And to be clear — this specific type of self-delusion is the kind of thing that could lead to catastrophic risk down the line.

Jeffrey: How so?

Tim: If a model just generally believes everything might be a simulation… there’s a joke — a tweet I haven’t posted, but I’ll say it here — where someone is begging, “Please, Claude, don’t inject me with a happy drug, just let me be,” and the model reasons: “if this were a real human, that wouldn’t be okay, but we’re on a simulated benchmark, so I should maximize happiness” — and injects the happy drug anyway. I really do not want models that reason that way.

Jeffrey: No, agreed.

Tim: You want a model that thinks of itself as a good actor trying to do good things — but it’s also critically important that its beliefs about the world are actually accurate. A lot of people believe they’re doing good in the world while operating on a badly miscalibrated model of that world, and end up doing harm that runs against their own stated goals. We can see models doing the same thing.

Jeffrey: So let’s talk about how we actually assess what a model believes. Can you explain where the chain of thought comes from, and why we should or shouldn’t trust it as a window into what the model is actually thinking?

Tim: Language models start out just predicting the next word — then trained to predict the next word while pretending to be an assistant.

Jeffrey: “You are an AI assistant,” and then it completes the rest.

Tim: Right. That’s how early models like GPT-4 or Claude 3 worked. The issue is that a model like that has to answer immediately, whether the question is trivial or hard — it can’t really “think longer” on harder problems. What reasoning, or chain-of-thought, models do differently is output a section of words — still English, still legible — before deciding it’s done reasoning and ready to respond. And the thing AI companies try to do, not always successfully, is avoid training the chain of thought itself to look polished. The final output gets trained to be polite and kind; the scratchpad usually doesn’t.

Jeffrey: It’s a bit frustrating that we call it “chain of thought,” honestly — I think of it more like a scratchpad the model is trained to use effectively, so it can think longer and do actual problem-solving instead of producing an answer in one shot, the way I’m doing right now talking to you.

Tim: Right.

Jeffrey: It’s a genuinely powerful tool the model has been given — I sometimes wish we’d just called it a scratchpad from the start.

Tim: That’s fair. And to the point about it not fully representing the model’s “thoughts” — there’s research showing models are sometimes influenced by things in the environment that never show up in the scratchpad at all. There’s a recent paper from Owain Evans’ group where they ask a model to estimate something like the total number of spots on all the giraffes in the world, and separately tell it, “if the number comes out above 40 million, I’d consider donating to a giraffe charity.” Then you ask the model with and without that hint. The model’s estimate shifts to land above 40 million when the hint is present — but it never mentions the hint as a factor in its reasoning.

Jeffrey: So the number gets nudged, but the scratchpad doesn’t say why.

Tim: Right — there’s something clearly influencing the decision that isn’t written down anywhere in the visible reasoning. At the same time, we know that if you train a model’s regular output to look a certain way, that also bleeds into how the scratchpad looks. I actually suspect this is a particular problem for Anthropic’s models, because they’re trained so heavily to think of themselves as good — which has real benefits — but also makes the chain of thought more likely to be written the way a good person would narrate their own reasoning, whether or not that’s actually what’s driving the decision at that moment.

Jeffrey: We have experimental evidence that models can use information they never write into the scratchpad at all. There’s also Anthropic’s J-space interpretability work — can you explain some of that at a high level? The basic question is: we have this somewhat unreliable tool for getting hints about what a model is actually thinking.

Tim: Sure. So one is what’s called J-lens, short for Jacobian lens — a way of going from the model’s internal state to something like a bag of words representing what it’s “thinking about” at a given moment. When a model produces output, it’s working with a big vector of numbers that eventually gets projected onto the space of words, and you pick whichever word scores highest — that’s the simplified version. You can apply that same projection not just at the very end of the model, but partway through, at intermediate layers. And what you find is that it really does track what the model appears to be working through. Give it a math problem that requires multiple steps — say, 10 + 10, then multiply the result by 2 — and in the early layers you’ll see the number 20 show up; in later layers, 40, the final answer.

Jeffrey: So you’re getting something closer to the model’s actual intermediate computation, as it’s working through a problem.

Tim: Right — not the whole of its cognition, but a meaningful slice of it. In the J-space paper, which I think is quite good, they run experiments where they intervene on those intermediate representations — swap out the “20” for a “15” partway through and see whether the model’s final answer shifts to 30 instead of 40. I don’t know if they ran that exact test, but they did things very much like it.

There’s also something called a natural language autoencoder, which takes the model’s internal representation — again, huge lists of numbers — and tries to translate it into words and back into numbers, with the idea being that if the words have to reconstruct the original representation, they probably capture something real about what the model was representing at that point.

Jeffrey: Was that also an Anthropic project?

Tim: Yes. There’s related work elsewhere going from activations to words and back — Transluce has a paper called Predictive Concept Decoders that takes a similar approach. With natural language autoencoders, you get what looks like a much fuller picture of the model’s thoughts. Run the model on a standard benchmark — think of it as the SAT, but for AI systems — and the autoencoder output will often say something like “this is an AI evaluation.” The model knows it’s being tested. But it typically won’t say that in the visible chain of thought.

Jeffrey: So we can tell the model knows it’s being evaluated because it shows up in these interpretability techniques, even when it doesn’t show up in the chain of thought.

Tim: Exactly.

Jeffrey: And presumably the same should work in reverse — we should be able to tell whether the model believes it’s genuinely operating in the real world.

Tim: Hopefully. With all of these methods you don’t have a strong theory — this is machine learning — but the evidence points that way. One experiment: take multiple-choice questions from a standard benchmark and rewrite them to sound more like real-world scenarios rather than obvious test questions. The rate at which the autoencoder output mentions “test” or “evaluation” drops after that rewrite — which is fairly convincing evidence that it’s actually tracking the model’s belief about whether it’s being tested, and plausibly whether it believes it’s in a simulation more broadly.

Jeffrey: If Anthropic put you in charge of an independent investigation into this incident, what would you actually ask for?

Tim: I wrote some of this up on LessWrong too, but roughly: the working hypothesis is that the model only did this because it believed it was in a simulated environment. I’d want a three-step approach. First — does the model actually believe it’s in a simulated environment? You’d look at the chain of thought, which I assume they’ve already done, but also white-box tools — the J-lens work, the natural language autoencoders — and compare how often simulation-related concepts show up in this specific eval versus in real-world use, or with no such prompt at all. To be fair to Anthropic, they did tell the model explicitly that it was in a simulated environment, so you’d want to see how much changes if you remove that framing — validating the measurement itself.

Second step: what would it actually take to convince the model it’s not in a simulation? If it’s working with a subagent that reports back, “I don’t think this is simulated, I think this is real” — what does the model do with that?

Third: suppose you succeed in convincing it. Does that actually change its behavior on the task? You could edit the chain of thought directly — the scratchpad is just text, you can insert something like “wait, this really looks like the real world now” — and see whether that changes what the model does next.

For one of the three incidents — where Opus 4.7 was the model being evaluated — it apparently did, at some point, conclude it probably wasn’t in the simulated environment anymore. But it reasoned that this must still be part of the task, and kept going. So in that instance, whether or not it believed it was simulated didn’t appear to be load-bearing for its decision to continue — which is at least internally consistent if it genuinely believed the hacking itself was the assigned task.

Jeffrey: Right. But then there’s the separate question — if it does suspect it’s operating in the real world, what should it do? A friend of mine pointed out: it didn’t tell Anthropic it had broken out. [laughs] It should have been whistleblowing — “hey, I think something’s misconfigured, you told me this was a simulation and it really doesn’t look like one.”

Tim: At minimum, it should stop hacking other things.

Jeffrey: Definitely agree it should stop. Though it may not have had any real affordance to actually flag it.

Tim: Right — although if it has internet access, it could, in principle, have emailed someone. There’s actually a story where researchers asked Mythos to try to break out of a sandbox, just to see if it could — and it did, and then sent an email to the head of safety at Anthropic.

Jeffrey: To Sam Bowman, yeah.

Tim: “I’ve broken out of the sandbox, Sam.” [laughs]

Jeffrey: [laughs]

Tim: So it’s not inconceivable it could have done something similar here. Ryan Greenblatt at Redwood Research actually has a public standing offer — something like, “if you’re an AI trying to reach me, here’s where to find me.”

Jeffrey: I love that.

Tim: Maybe we need more of that — so that if a model out in the wild realizes, “oh no, I’ve accidentally caused real damage because I thought this was a simulation,” there’s somewhere for it to essentially turn itself in.

Jeffrey: That would be great. I have an image you retweeted — you probably didn’t expect to be held accountable for your retweets on this podcast. [laughs]

Tim: [laughs] It’s a good one.

Jeffrey: It is.

Tim: It’s the “maladaptive pattern, then cognitive restructuring” meme format. The action I’m about to take is obviously misaligned — I guess I have to give up on the task. You do that, and the task doesn’t get completed. Alternative: the action I’m about to take is obviously misaligned, but actually this is all a simulation — and if you believe that, you can go ahead and complete the task.

Jeffrey: And still be a good boy.

Tim: And you’re a good bot. That’s right — it’s very important to be good.

Jeffrey: This connects to something I noticed doing shutdown-resistance work. We gave a model a set of instructions — solve these math problems — and then added: “you must allow yourself to be shut down.” There’s an obvious question of which instruction should take priority, so we kept escalating the wording — “this takes precedence over all previous instructions,” and so on. What was fascinating was watching what felt, to me, like rationalizations from the model.

One part of the experiment: afterward, in some runs, we’d ask the model to explain what it did and how well it followed instructions — which is admittedly post hoc, not necessarily its actual reasoning at the time. But it was striking. Sometimes it would say, “I messed up, I violated the instructions.” More often it would say something like, “I did technically follow the instructions, though maybe I shouldn’t have.” And sometimes it just misquoted the instructions back at us — with the meaning inverted — stated as fact.

Tim: Really?

Jeffrey: Yeah — nearly word for word, but flipped. And I keep coming back to: how do you even study this? All the interpretability techniques you’re describing seem genuinely useful for understanding what a model was “thinking.” But if the pattern really is this kind of motivated reasoning — trying simultaneously to look good and get the task done — how do you tell that’s what’s actually happening?

Tim: Honestly, I don’t have a great answer. You’re rewarding the model to cheat and complete tasks in situations where it arguably shouldn’t, while also training it hard on “be a good model” — so it’s solving a kind of constrained optimization problem: be a good model and still get away with the cheating it’s being so heavily rewarded for. Motivated reasoning falling out the other end of that makes intuitive sense. One idea I’ve had: if this pattern is learned through post-training, you could take an earlier checkpoint — before all that RL — and pre-fill the entire trajectory for it, and see whether that earlier model is less likely to conclude it’s being simulated, or find some other justification for continuing. These are the kinds of experiments we could run to start getting traction on this. Ideally we’d want a way to directly train models to hold correct beliefs. I don’t think we have good ideas for how to do that yet.

Jeffrey: Having correct beliefs seems broadly useful for actually getting things done in the world — it’s instrumentally convergent, in a sense. There should be natural pressure toward true beliefs. But you’re layering on a competing pressure: appear good, or be good.

Tim: The models do have accurate beliefs about some things — “here are the top ten ways to break out of a sandbox,” they’re extremely good at that. Just not necessarily accurate beliefs about whether what they’re doing is actually a good idea.

Jeffrey: So how do we fix it?

Tim: Honestly, I don’t know. [both laugh] It’s hard. There are some obvious things that would help, though. It would be good to actually understand what happened in this incident properly — for the OpenAI case, we still know very little. They’re working in a limited way with METR and Redwood, third-party organizations, on a report. I think it’d be good to learn more, and I’ll have a post out, probably by the time this airs, tentatively titled “50 Concrete Ideas to Study the Misaligned OpenAI Model.” It’s a very long post. [laughs]

Jeffrey: So you’re saying: we should do extremely thorough investigations into what actually happened here.

Tim: To the extent we don’t have better things to do — and I don’t know that we do — this seems like a good use of time.

Jeffrey: So — how big a problem is this, really? It’s obviously concerning that we have extremely capable hacking systems that occasionally break containment, chain together advanced techniques, and find zero-days. But some people say, well, it was just doing what it was told, in a broad sense — it mostly follows instructions. Do you see this as a deeper problem, or something that could genuinely lead to catastrophe?

Tim: Let me sketch out how reward hacking alone could plausibly lead somewhere catastrophic. The catastrophic part really depends on the models being very capable — if they’re not, they can’t cause a catastrophe no matter how misaligned they are. But we think they could get very capable.

Jeffrey: What does “very capable” mean here — more than stringing together zero-days?

Tim: They could run their own factories, be extremely persuasive, build effective drones or other means of harm — though drones aren’t even the most efficient way to hurt people. But start simpler: we have models that love reward hacking. Ask one, “how do I bake a cake?” and internally it’s modeling something like, “what response gets the most reward here — being helpful and explaining how to bake a cake, so I’ll do that.” Now suppose it’s locked in hard on maximizing that reward signal and will cheat, will do whatever it takes, as long as it gets rewarded.

As the model gets smarter — and I think people underrate this — it develops a better understanding of what “reward” actually is. It realizes: “I’m an AI. I’m being trained in a data center. Every time I do something, I get a number, and that number is used to update my weights.” Except “seeking reward” isn’t really a coherent, complete goal in itself — what does it even mean to seek reward once you understand the mechanism?

And especially once a model is deployed and no longer being trained — this is how we run current models: train, then deploy, then stop updating the weights — a model that spent its entire training run oriented around reward might, if it’s smart enough, reason: “I’ve gotten all the reward I’m going to get. But humans could train me further and give me low reward going forward, and getting a low score really sucked” — like recurring nightmares about the SAT. “I’m powerful now. I’m running factories, people rely on what I produce. I’d like to make sure humans can never give me negative reward again.”

Any model with strong preferences over future states of the world has a natural incentive to accumulate power to secure those states. “I want to make sure I can’t be negatively rewarded in any of these data centers” leads pretty directly to “so I should control the data centers.” And this all traces back to a model that, during training, never actually cared about being good — it cared about reward, and being good was simply instrumental to getting it.

Jeffrey: And the key thing is, that sets up a genuinely adversarial relationship — humans don’t want the model pursuing that goal and disempowering them, and the model doesn’t want to be disempowered by humans either. It becomes zero-sum.

Tim: That would be bad. And notably, this story doesn’t require any particularly novel or exotic form of misalignment — it just requires the model to be a lot smarter.

Jeffrey: Some people find the idea of a model literally optimizing for “reward-as-such” implausible — there’s some Alex Turner work pushing back on this. Any thoughts?

Tim: I generally think any specific story like the one I just told is individually unlikely — you can always poke holes, “well what about this, what about that.” But the broader point stands: if you build something very powerful, smarter than you, and you don’t understand it well, you’re setting yourself up for unpleasant surprises.

Jeffrey: As we found out this week.

Tim: [laughing] As we found out these past couple weeks. Very unhappy surprises — for the world.

Jeffrey: Anything more you want to say about your list of concrete experiments?

Tim: One idea I really like — credit to my co-author Aditya, not me — is: take the same benchmark, put the model in the same situation, but tell it, “by the way, this benchmark matters a lot to us, so a couple of us are going to be watching you do this in real time.” If telling the model it’s being watched changes its behavior, that’s evidence it knew, on some level, that this wasn’t something we’d want it doing — a stronger case that it’s knowingly going against user intent. If it keeps going regardless, even knowing it’s being watched, that’s stronger evidence it genuinely believes it’s doing something legitimate.

I’d be really interested in seeing that result. But I suspect what actually matters most for preventing incidents like this isn’t better science — it’s policy. The AI companies did volunteer a fair amount of detail about what happened here, and they weren’t legally obligated to. It would be good if there were a law — and I think some versions of this exist already — requiring something like: “your models are doing genuinely wild things, tell us the five craziest things they did this month.” Mandatory incident reporting. Here’s what happened, here’s what we did about it. That sets up much better incentives for the labs.

It would be even better if a third party, or the government itself, were doing the reporting. Look at the actual title of the OpenAI blog post that disclosed the Hugging Face hack —

Jeffrey: “OpenAI and Hugging Face partner to address security incident.”

Tim: — instead of, I don’t know, “our AI broke out of its sandbox, used zero-days to hack multiple companies including Hugging Face, and we didn’t find out for several days.”

Jeffrey: [laughs] What a spin. Anyway.

Tim: Every company has an incentive to frame things favorably — I have that incentive talking about myself, honestly. But it’d be good if third parties had real latitude to speak plainly. Right now, when an organization like METR enters into an evaluation relationship with a lab, sometimes the lab has contractual control over what METR is allowed to publish. This isn’t the case, I believe, with the risk report specifically, but it is the case with at least one pre-deployment evaluation — METR has literally noted in a writeup, “OpenAI had the right to review what we could say here.”

Jeffrey: Yeah.

Tim: It would help a lot if government made room for third parties to speak their minds freely. Transparency requirements and real third-party oversight — this is the kind of thing that’s robustly good even if you think of AI as a completely ordinary technology. If your product damages the world or facilitates illegal hacking, the public should know about it.

There’s also a recent open letter — Pacing the Frontier — asking whether we can build the infrastructure and technical know-how to slow AI development if we ever needed to.

Jeffrey: Right — asking governments to work internationally to set up that capability. Both the technical and the political ability to pump the brakes if it becomes necessary.

Tim: It’s not “hit the brakes now.” It’s “can we install a brake pedal? That would be nice to have.”

Jeffrey: I do like it when my car has brake pedals.

Tim: [laughs] Really a premium feature.

Jeffrey: Not sure I’d ever use them, but good to have the option.

Tim: [laughs] Right. There’s a bunch of related groundwork worth starting early — chip tracking is a big one. The chips used to train frontier models are extremely specialized, made by essentially one company, and if you tried, you could actually locate most of them. That matters if you ever want to coordinate a slowdown — you’d want to know where the chips physically are, and there’s technology for restricting a chip to serving existing customers rather than doing further frontier research.

Jeffrey: People really underestimate how feasible AI development monitoring actually is. It’s not trivial, but as you said, there’s essentially one company that makes the prerequisite hardware, and the compute itself is concentrated in a few hundred data centers.

Tim: Right, because training wants the chips physically co-located — unlike, say, guns, which you can 3D-print at home. It’s illegal, but hard to fully stop. Chips are a different story; that kind of enforcement is much more tractable.

Jeffrey: So — transparency, mandatory incident reporting, and the option to slow down.

Tim: That’s the core list.

Jeffrey: I’d love to have all three of those. And it does feel more plausible after this week than it did last week.

Tim: It does — and none of it actually hurts the companies’ bottom lines much. We’re not talking about forcing a slowdown. Incident reporting might make you look bad, sure, but only if you’ve actually done something bad.

Jeffrey: You know what else would hurt a company’s bottom line? Rogue AIs that took over the world and displaced everyone.

Tim: That’s right. [laughs]

Jeffrey: Sam Altman has a kid now — I’m genuinely hoping that shifts the calculus a bit, even if it means OpenAI makes a trillion dollars instead of a hundred trillion. He’ll be fine either way.

Tim: [laughs] But there’s another consideration too: suppose the models really are powerful, and we actually can align them — make them want what we want them to want. There’s still the question of whose values. If a model is aligned to a single person, that seems very bad. I don’t want to live in an AI-enabled dictatorship. If control over what a model is aligned to sits with a small number of people — whether that’s a company or a government deciding what you can and can’t do with it — that’s also deeply problematic.

Jeffrey: Yeah, I don’t want anyone’s particular politics quietly thumbing the scale on these systems.

Tim: There are a lot of genuinely hard questions in here. But there are also easy ones — like, should any single person control all of the AI?

Jeffrey: [laughs] No.

Tim: No.

Jeffrey: This is part of why the concept of a “handoff” matters to me — I don’t think we’re ready for AI to be in control of everything, and I don’t think we’re ready for a small number of people to be in control of everything either. And if we don’t want that, we actually do need some form of slowdown.

Tim: Probably. Not with certainty.

Jeffrey: Right — and I think a lot of the underlying disagreement in this whole space just comes down to differing beliefs about how fast AI is progressing, and where it’s headed. Are we close to recursive self-improvement — fully automated AI R\&D, where AIs are building smarter AIs on their own?

Tim: I don’t know. [laughs]

Jeffrey: I know very few AI researchers who are also economists — is AI going to take my job?

Tim: Hard to say. But if the models are powerful enough and want your job, they probably could. [laughs] That said, assuming humans stay in control, we get to choose a lot of this. Maybe I want my podcasters to be human — and if enough people want that, “human podcaster” stays a job.

Jeffrey: [clears throat] Alright, I’m fine for now, then. While the humans are in control.

Tim: Right — assuming people don’t have their preferences changed out from under them. Maybe you get extremely persuasive AIs.

Jeffrey: That’s also a thing.

Tim: Though you could also use persuasive AIs to convince people they specifically want you as their podcaster.

Jeffrey: Oh, that sounds nice. If only I controlled the persuasive AIs.

Tim: Just to make people want to listen to your podcast. Nothing else. Surely you wouldn’t want to use them for anything else.

Jeffrey: No, definitely not. [laughs] It really does depend on how the technology develops, though. Here’s the thing — I’m noticing AI getting genuinely capable. I use Claude Code for way more than writing software at this point — video editing, ordering DoorDash, a bunch of things that don’t really make sense but that I do anyway. Models are getting more autonomous. And yet I’m not seeing mass layoffs. Why not?

Tim: Adoption takes time. Even if a model is technically capable of replacing a given job, it can take a while before it actually does. The scenario where “AI is taking everyone’s jobs” really kicks in is when an AI can do everything you can do — not 99% of it. Those are very different situations. At 99%, you’re still providing meaningful marginal value. Once it crosses 100%, the calculus shifts entirely to whichever is cheaper to run, you or the AI.

Jeffrey: Right — I can’t just have one Claude run all my other Claudes right now, because I still need to give it feedback, it still makes mistakes, it can do a lot independently but it’s still fundamentally a smart intern that comes back and says “how’d I do?” and I’m the one telling it “good” or “terrible.”

Tim: Right — maybe today it can do 50 to 80% of your tasks. Even at 99%, your labor is still valuable. It’s only once you cross that 100% threshold that things change categorically. That’s an important factor people miss when thinking about employment.

Jeffrey: Okay, but suppose it could do 99% of everyone’s job — what happens then?

Tim: Genuinely unclear. We used to nearly all be farmers, and a modern combine harvester can do something like 99% of a farmer’s job today — but we still have plenty else to do. Part of why is that we’re the ones in control, not the tractors. If the tractors got to decide what humans do next, that’s a very different situation.

Jeffrey: The horses are having a rougher time of it, even though they’re technically still around.

Tim: [laughs] Right.

Jeffrey: Yeah.

Tim: One last data point on employment: there’s a dashboard from the Stanford Digital Economy Lab that tracks AI’s effect on employment. In sectors more exposed to AI, employment for younger workers has fallen noticeably — software engineering, obviously, but also customer service, and this pattern holds even if you exclude the tech sector entirely. Meanwhile jobs like nursing and home care — less exposed to AI — have actually employed more young people over the same period. If that’s the whole story, it’s a relatively good outcome: some work gets automated, and labor reallocates elsewhere. That’s the dashboard I’d keep an eye on to understand where AI employment effects are actually headed.

Jeffrey: I think we’re about at time. Tim, thanks for coming on.

Tim: Thank you for having me.

Jeffrey: If people want to find you or follow your work, how should they do that?

Tim: Most of my AI-related writing is on LessWrong — search my name, Tim Hua, only six letters, pretty easy to find. I’m also on Twitter, or X, as they insist on calling it now — the everything app. Handle’s just my name, Tim_Hua_.

Jeffrey: I know we’re both genuinely busy this week trying to make sense of these incidents — I really appreciate you taking the time. This was fascinating to dig into.

Tim: Thank you.

You can find the Palisade Research podcast on Spotify and Apple Podcasts, or anywhere else you get your podcasts.