OpenAI is looking for a new head of preparedness, someone that can help the chat GPT maker avoid a lot of the catastrophic events we worry about with AI.
Now, this is an interesting story because they previously had a head of preparedness who left.
Many people from their safety team have left and a lot of people say this is due to Sam Altman not prioritizing That part of the company and basically just pushing out models without doing all of the red teaming.
This is a multi billion dollar company, so I don't think that's 100% the full story here.
Today on the show.
We're going to get into what they're looking for.
How much Sam Altman is personally willing to pay for this new head of preparedness why?
Now is the time and everything he's been saying about this on X.
We're gonna get into all of that, but before we do, I wanna say a big thank you to the sponsor of today's episode, which is Delve.
If compliance is something that's slowing down your deals, whether that's SOC 2 HIPAA GDPR, We'll see you next time.
Over a thousand fast-growing companies trust Delve to close deals faster and stay compliant as they scale.
I'll leave a link in the description to Delvecom if you're interested in learning more or booking a demo.
All right, let's get into what's going on with OpenAI.
Sam Altman himself has personally posted on X saying that quote We are just now seeing models get so good at computer security they are beginning to find critical vulnerabilities.
Okay, there's a lot going on here.
And I'll read more of his tweet.
He said a couple other things about this role.
But essentially what they're looking for is someone that can help, you know, prevent all of the catastrophic events.
We've been terrified that AI could pose, you know, risk in the future.
And I think this is interesting because there are a bunch of Things happening, happening with security, where we see OpenAI has said that they actually like basically fine tune and train models to try to breach security.
And then they use those and and to try to like harden their models and then they, you know, put in safeguards so that people can't use OpenAI models to become like hackers and and breach security.
What's interesting to me in this is that they are actually going and training the AI to be able to do this in the first place, which I mean is kind of crazy.
But maybe you need to be able to do that to be able to control it.
But one shocking thing they found while they were doing this is that AI is getting so good.
It's discovering security vulnerabilities.
It's social engineer situations to be able to get data, to be able to get information, to be able to hack and to crack into things that humans weren't able to do.
In other words, they had like literal, a red team of hackers that are like trying to use AI and chat GPT to like hack into things.
And then they had these real people going up against actual, just like training the AI model to be a hacker.
And the AI that was trained to be a hacker was doing better than the actual people.
And when I say better, it was essentially thinking of new vulnerabilities, really elaborate complex, multi-step ways to get data and to hack into things that people were not coming up with.
And so this is definitely part of the big concern that Sam Altman is pointing out here and that he's worried about.
He said, quote we have a strong foundation of measuring growing capabilities, but we are entering a world where we need more nuanced understanding and measurement of how those capabilities could be abused and how we can limit those downsides, both in our products and in the world, and in a way that lets us all enjoy the tremendous benefits.
These questions are hard, and there is little precedent.
A lot of ideas that sound good have some real edge cases.
Now, the thing that I do think is interesting here and it's a great point is we obviously all love ChatGPT.
We obviously all want this technology to get better.
It's helping us a lot with work.
It's saving us a lot of time on a number of tasks and menial things that I'm not having to do anymore, that I used to do.
But at the same time, there are real risks.
So we have to figure a way to balance some of the risks.
And, you know, if we want the AI models to get better, they're also going to get better at areas that are, you know, areas of concern.
And so how do we how do we mitigate that?
And that's who they're hiring for.
Sam Altman finished off his tweet by saying if you want to help the world figure out how to enable cybersecurity defenders with cutting edge capabilities while ensuring attackers can't use them for harm ideally by making all systems more secure and similarly for how we release biological capabilities and even gain confidence in the safety of running systems that can self-improve, please consider applying.
This will be a stressful job and you'll jump into the deep and pretty much immediately.
So This job is not for the faint of heart.
If you go over to their careers page and you look, you know they give you an outline about the team and the role and they say what you're going to be doing in the role.
One thing that is interesting is the compensation for this particular job here is $555,000 a year.
And they also have equity along with that.
So I mean it's a.
You know it's a very generous pay package, but it is obviously very stressful and entails a lot.
When they say you know what you'll actually be doing in the role.
They said own OpenAI's preparedness strategy end to end by building capability evaluations, establishing threat models and building and coordinating mitigations.
Lead the development of frontier capable evaluations, ensuring that they are precise, robust and scale across rapid product cycles.
Oversee mitigation design across major risk areas, ensuring safeguards are technically sound, effective and aligned with underlying threat models.
So one thing that I do think is interesting here with this role is, I mean very publicly, Sam Altman is going out saying look, we're going to pay over half a million dollars a year, plus you know compensation, which I would imagine.
This is like a million dollar a year, maybe 750 800, a million a year, maybe more with the compensation package, with the stock options, probably over a million, if I'm being honest.
So, you know, great compensation for this role.
But what does that mean?
I think because this is so publicly being posted online, you could say one of two things.
Sam Altman is signaling to everyone, look, I'm very serious about this.
We really want to hire the best person.
Also, it's going to be a very public-facing person, I would imagine, just given the nature of the role.
And if anything goes wrong at OpenAI, if any of these systems are used for safety, you know that whoever gets this role is going to be pointed at like oh my gosh, XYZ person didn't do their job, because this is what they were supposed to do.
Now, One thing that I do think is interesting here is they already had a head of preparedness.
They first announced that they were creating kind of this role in 2023.
They said that it was going to be responsible for studying potential catastrophic risks, whether they were immediate, like phishing attacks, or more speculative, like nuclear threats.
So that was kind of when it was first when it first came out.
But less than a year later, they reassigned their head of preparedness, who was Alexander Madry, to a job focused on AI reasoning.
So the person that was in charge of that, they reassigned them less than a year later.
And now they're trying to fill the role again.
And it's interesting, like I'm not sure if this was because Alexander himself requested to move to something different, or if they just needed help over there, and he had a lot of experience.
I think there's, you know, definitely other safety executives at OpenAI that have left the company or they've taken new roles outside of preparedness and safety and kind of moved it around, as I think that got a little bit deprioritized last year.
Google Gemini was putting out a lot of updates.
You know, they were concerned about meta, they were concerned about Grock, they were concerned about, like a lot of other players, including Claude.
And it felt like they definitely kind of put this on the back burner because they didn't want to spend all their time on safety when they were.
It felt like they were kind of falling behind on some of the model features and they were just focusing on getting the model out as fast as possible.
Now they're going to bring someone in and that's going to be their role.
But that person is going to take a lot of like personal I wouldn't say the personal liability, because obviously everything at the end of the day is opening eyes.
You know responsibility, but that person is going to be very public facing.
If anything goes wrong, they will 100% feel a lot of the heat on that.
They also recently updated their preparedness framework saying that it might, and they said that they're going to adjust it.
It's you know, and some of their safety requirements could be adjusted if a competing AI lab released a high risk model without similar protections.
So they're like, look, we're make our models really safe.
But like if Gemini or Grok come out and their model is crushing it, but they don't have the same protections, then we'll just like dial back the safety on it so that we could be competitive with them, which is definitely a very interesting point.
I know a lot of people will find concerning.
Right now, I think Sam Altman.
He alluded to this in his post, but a lot of these generative AI chatbots have faced a lot of scrutiny around specifically, mental health.
There's a bunch of recent lawsuits that allege that ChatGPT enforces users' delusions, it increased their social isolation and it even led some to suicide.
OpenAI is definitely at a critical moment where they have to get this right.
I think it's not an easy thing.
OpenAI did say that they're continuing to work on improving ChatGPT's ability to recognize signs of emotional distress and to connect users to real-world support.
I think this is amazing.
Obviously, this is one of these situations where you have to learn and figure out what can go.
As you see things that go wrong, you have to fix them and try to improve them.
And I think that that is the exact right approach I would like to see more and more from OpenAI.
So this is going to be interesting to see how this rolls out and hopefully, to see that their safety platform is robust but also doesn't slow down the innovation that a lot of us are using in our day-to-day lives.
Thank you so much for tuning into the podcast today.
As always, make sure to go check out AIboxai, my own startup, to get access to all of the latest models and build incredible tools.
If you're not a coder.
And also go check out the sponsor of today's episode, which is Delve.com.
There's a link in the description to Delve and I'll see you in the next episode.
