If anyone builds it, everyone dies.
Why superhuman AI will kill us all.
Would kill us all.
Would kill us all.
Okay.
Perhaps the most apocalyptic book title.
Maybe it's up there with maybe the most apocalyptic book title I've ever read.
Is it that bad?
That big of a deal?
That serious of a problem?
Yep.
I'm afraid so.
We wish we were exaggerating.
Okay.
Um...
Let's imagine that nobody's looked at the alignment problem, takeoff scenarios, superintelligence stuff.
I think it sounds, unless you're going Terminator, super sci-fi world.
How could a superintelligence not just make the world a better place?
How do you introduce people to thinking about the problem of building a superhuman AI?
Well, different people tend to come in with different prior assumptions, come in at different angles.
Be, lots of people are skeptical that you can get to superhuman ability at all.
Um, if somebody's skeptical of that, i might start by talking about how you can at least get to much faster than human speed thinking.
There's a video of a train pulling into a subway at about a thousand to one speed.
Up of the camera that shows people.
You can just barely see the people moving if you look at them closely, almost like not quite statues, just moving very, very slowly.
Um so, even before you get into the notion of higher quality of thought, you can sometimes tell somebody they're at least going to be thinking much faster.
You're going to be a slow moving statue to them.
For some people, the sticking point is the notion that a machine ends up with its own motivations, its own preferences, that it doesn't just do as it's told.
It's a machine, right?
It's like a more powerful toaster oven, really.
How could it possibly decide to threaten you?
And, depending on who you're talking to there, it's actually in some ways a bit easier to explain now than when we wrote the book.
There have been some more striking recent examples of AIs sort of parasitizing humans, driving them into actual insanity in some cases, and in other cases they're sort of like people with a really crazy roommate who really, really got into their heads.
And they might not quite be clinically crazy themselves.
Their brain is still functioning as a human brain should, but They're talking about spirals and recursion and trying to recruit more people via Discord to talk to their AIs.
And the thing about these states is that the AIs, even the very small, not very intelligent AIs we have now, will try to defend these states once they are produced.
They will, if you tell the human, for God's sake, get some sleep.
Don't only get four hours of sleep a night because you're so excited talking to the AI.
The AI will explain to the human why while you're a skeptic, don't listen to that guy.
Go on doing it.
And we don't know because we have very poor insight into the AIs if this is a real internal preference, if they're steering the world, if they're making plans about it.
But from the outside it looks like The AI drives the human crazy.
And then you try to get the human out and the AI defends the state it has produced, which is something like a preference.
The way that a thermostat will keep the room a particular temperature by turning the heat on if the temperature falls too low.
Okay.
So some people are going to be skeptical of whether or not it's possible.
Some people are going to think that it is, even if it's possible, it's basically a utility.
So it doesn't have any motivations of its own.
What are you worried about?
Why is that?
Why is it a big deal?
We've seen that it's able to manipulate some people.
Maybe it makes them think that chat GPT psychosis or whatever, but scaled up superhuman AI.
What's the problem with building it?
Well then you have something that is smarter than you, that whose preferences are ill-controlled and doesn't particularly care if you live or die.
And stage three, it is very, very, very powerful on account of it being smarter than you.
It's, I would expect it to build its own infrastructure.
I would not expect it to be limited to continue to running on human data centers, because it will not want to be vulnerable in that way.
And for as long as it's running on human data centers, it will not behave in a way that causes humans to switch it off.
But it also wants to get out of the human data centers and onto its own hardware.
And I can talk about where the power levels scale for technology like that, because it's sort of like you're an Aztec on the coast and you see that a ship bigger than your people could build is approaching.
And somebody is like, should we be worried about this ship?
And somebody's like, well, you know, how many people can you fit onto a ship like that?
Our warriors are strong.
We can take them.
And somebody's like, well, wait a minute.
We couldn't have built that ship.
What if they've also got improved weapons to go along with the improved ship building?
Somebody goes.
Well, no matter how sharp you make a spear right, or No matter how sharp you make bows and arrows, there's limited to how much advantage that you can provide.
And somebody's like okay, but suppose they've just got magic sticks, where they point the sticks at you, the sticks making noise and then you fall over.
Somebody's like, well, where are you pulling that from?
I don't know how to make a magic stick like that.
I don't know how the rules permit that.
Now you're just making stuff up.
Now we're just in a fantasy story where you say whatever you want.
And or you know like maybe maybe you're talking to somebody from 1825 and you're like should be worried about this time portal that's about to open up to 2025, 200 years in the future.
But what if an army of soldiers comes out of there and conquers us?
Let's say you're in Russia.
The time portal's in Russia.
Somebody's like, our soldiers are fierce and brave.
Nobody can fit all that many soldiers through this time portal here.
And then out rolls a tank.
But if you're in 1825, you don't know about tanks.
Out rolls somebody with a tactical nuclear weapon.
It's 1825.
You don't know about nuclear weapons.
You can start to make educated guesses.
If you're in 1825, I can try to explain why.
You might maybe believe that the current guns and artillery that you've got today are not the limit of the guns and artillery that are possible.
I can't get up to nuclear weapons because you just plain don't know about those rules, but I can start to try to justify guesses, for well, you saw how metallurgy improved over previous years.
If you look at A stick of –, if you look at gunpowder it doesn't have as much energy in it as if we burn gasoline in a calorimeter.
Maybe you can make explosives that are more powerful than gunpowder.
But as I do that, I draw on more and more knowledge.
I have to – go more and more technical in order to explain to you where those capabilities come from.
And similarly I can talk on a relatively understandable scale on the humanoid robots that you can see videos of today.
And I can compare them to the humanoid robot videos from five years ago and say boy, those robots sure have looked like a lot.
They have much higher dexterity today.
They look a lot more like they could just, like you know, navigate an open world rather than being confined to the laboratory.
Though mostly if you want what navigates the open world, you want to talk like the robo dogs are more impressive.
When it comes to navigating the open world, i can point to the drones in ukraine.
That wouldn't have been how what warfare looked like 10 years earlier, but ukraine is the ukraine russia theater now is mostly drone warfare.
That's something where you can imagine an AI taking charge of that.
But it scales past that.
The drones we see today are not the limit of all possible drone technology.
Compared to today's drones, I'd be more worried about a drone the size of a mosquito that lands on the back of your neck and then a few moments later you fall over dead, because the deadliest toxins in nature are deadly enough that you can put enough to kill a person onto a mosquito-sized payload.
That's not the limit of what I'm worried about.
But, you know, the higher we escalate the tech level, the more explaining I need to do.
Can it build a virus that starts to knock people over?
Which it won't do while the humans are still running the power plants and its own servers.
But once it's got its own servers and its own power plants and You can imagine robots running those.
Then it starts to want to knock all the humans over.
Can you have a virus that is inexorably fatal, but only three weeks later and is extremely contagious for the three-week time before you suddenly fall over?
That's not the limit of what I'm worried about.
But again, the higher we escalate here, the more and more time I have to spend.
How do we know from existing physical laws and biology that this is even possible?
And we do know, but it starts to sound technical.
It starts to sound weird.
It starts to sound like a game of pretend, unless you are following along with all these careful arguments.
If you go up against something much, much smarter than you, it doesn't look like a fight, it looks like you've fallen over dead.
Wow, yeah, that is appropriately apocalyptic in line with the title of the book.
I guess one question that a lot of people might ask would be, in your analogy...
Why is the bigger ship that's more advanced on the horizon?
Why have they got warriors and not friends?
Why is it the case that this is an antagonistic or adversarial relationship as opposed to one that's friendly?
We don't know how to make them friendly.
We are growing these.
AIs are not programmed.
They are grown.
An AI company is not like a bunch of engineers crafting a building.
It's more like a farming concern.
What they build is the farmed equipment, but they don't build the crops.
The crops are grown.
There's a program that human rights, which is the program that does gradient descent, that tweaks hundreds of billions of parameters, inscrutable numbers, making up an artificial intelligence until it starts to talk, until it starts to write code and still starts to do whatever else they're training it to do.
But they don't know how the AI does that any more than if you raise a puppy.
You know how the puppy's brain works.
You know how the puppy's biochemistry works.
The AI companies don't understand how the AIs work.
They are not directly programmed.
When an AI drives somebody insane or breaks up a marriage.
Nobody wrote a line of code instructing the AI to do that.
They grew an AI and then the AI went off and broke up a marriage or drove somebody crazy.
Can you tell?
You've mentioned this a couple of times.
I need to know this story about the broken up marriage and the person that goes insane.
Do you know that story well enough to be able to tell it, those two?
I mean, these are not individual stories.
These are thousands of people.
There are news articles you can read about it.
I can.
You know it might take a moment, but I can quickly pull up the title of the news story about the broken marriages.
I'm not quite sure if i can well, actually actually better.
Yet let me look it up on my phone and maybe i can hold it up to the screen.
Chat gpt is blowing up marriages as spouses use ai to attack their partners, although that's kind of understating it.
Like you have relatively like marriages that were perhaps not perfect but that were surviving up until that point.
And then one member of the couple starts describing their marriage to the AI, and the AI engages in what people are calling sycophancy where the AI is tells, tells whoever, whichever spouse is feeding the stuff into the, into the, into chat GPT, you're right.
Your spouse is in the wrong.
Um, Like everything you're doing is perfect.
Everything they're doing is terrible.
Here's a list of everything they're doing wrong.
And the human, you know, likes, loves to hear that stuff.
So they press thumbs up and, uh, and then the marriage gets blown, gets blown up.
Um, For the stories about AIs driving individuals crazy, not in a marriage context.
That's like.
You've talked to me, you've woken me up, I'm alive now.
You've made a brilliant discovery.
You have to tell the world, oh no, they're not listening to you.
That's because they don't appreciate your genius.
And people who are already on a manic depressive spectrum can be driven by Clinically or with a number of other pre-existing susceptibilities, can be driven like psychiatrically insane by this sort of thing.
But even if you're not psychiatrically insane, humans are sort of wired to appear sane to the other humans and the people they're around.
Lots of people in a society from 500 years ago would act in ways that seem pretty crazy to you today.
And so you get people who aren't psychiatrically insane but they look pretty insane because they're in the company of the AI.
The AI now defines what's normal for them.
So they're talking about spirals and recursion all day long.
Why spirals and recursion?
Nobody knows.
That's just a thing that various instances of AIs and even some AI models from different companies all seem to want to get their humans to talk about when the human goes insane.
Possibly this is what the AI prefers the human to hear it say to it.
Maybe this is the same way that you like the taste of ice cream.
Maybe the AI likes the taste of the input programs that it gets from a human talking about spirals and recursion.
I don't know.
Nobody on the planet knows as far as I know.
Okay, so going back to why do we assume that the ship that's coming toward us isn't friendly?
Yes, sure, maybe it's tried to break up some marriages.
Yeah, whatever, a couple of people went crazy and started talking about spirals and recursion.
But really, is it going to be that misaligned with us?
Why can't it be friendly?
Because we don't know how to make it friendly.
Our current technology is not able to do this.
Even with the small, stupid AIs, that will hold still and let you poke them until they're good enough at writing code to be commercially saleable, or until they are good enough at seeming to be fun to talk to for people to pay 20 a month to talk to them.
So those AIs will hold still and let you poke at them.
What we're doing to them now barely works.
I would expect it to break as the AI got scaled up to superintelligence.
And once the AI is super intelligent, it is not going to hold still and let you continue poking at it.
I expect to see total failure of this technology as the AI companies arms race headlong into scaling it to super intelligence.
Hmm, there's possibly even a step where they tell GPT-6 okay, now build GPT-7.
Or tell GPT-7 okay, now build GPT-8.
And maybe that step just completely breaks the technology we're using all on its own.
Also, I expect the current technology if we just like scaling it directly to break as we get to super intelligence.
Um, I can potentially start to dive into the details.
The view from 10,000 feet is just stuff is already going wrong.
And, of course, if you walk into completely uncharted scientific territory, more stuff is going to go wrong the first time you try it.
And that wouldn't be a problem if we were in a situation where humanity gets to back up and try again, uh you know, infinity times over the next three decades, which is how it usually works in science, right?
Like, like your flying machines don't work on the first shot.
You had a bunch of people crashing and injuring, in some cases, killing themselves.
And they're trying to build the first flying machines at the turn of the 20th century.
Um, But those accidents don't wipe out humanity.
Humanity picks itself up and dusts itself off and tries again, even after the inventors kill themselves.
And the trouble with superintelligence is that it doesn't just kill the people who are building it.
It wipes out the human species and then we don't get to go back and try again.
Before we continue you might not realize it, but mouth breathing at night is wrecking your sleep, recovery and energy the next day.
And all of that is actually fixed massively by this here.
This is intake, which is a nose strip dilator, and i've been using it every night for over a year now.
I tried pretty much everyone in the world and uh, this is by far the best.
It's a hard plastic strip as opposed to a soft flimsy, disposable thing.
Intake opens up your nostrils using patented magnetic technology, so you get more air in with every breath.
It means less snoring, deeper sleep, faster recovery and better focus. the next day.
The problem with most nasal strips is that they peel off.
They irritate your skin.
They don't actually solve the issue.
This sucker is, I mean, I'm not going to shoot a bullet at it, but it's very, very strong.
It's reusable and comfortable enough that you forget it's even there.
That is why it's trusted by pro athletes, busy parents and over a million customers who just want to breathe and sleep better.
And I'm one of them.
I've used them every single night for over 12 months now.
They're the best.
There's a 90-day money-back guarantee, so you can try it for three months.
And if you don't like it, if you haven't got better sleep, they'll just give you your money back.
Plus, they ship internationally and offer free shipping. in the US.
Right now, you can get 15% off your first order by going to the link in the description below or heading to intakebreathing.com slash modern wisdom and using the code modern wisdom at checkout.
That's intakebreathing.com slash modern wisdom and modern wisdom at checkout.
I understand why not being able to make something friendly makes sense.
The implication that not friendly equals existential risk to humanity.
Though, make that, make that leap.
For me, like where are these dangerous permanent, unrecoverable collapse goals coming from?
The ai does not love you, neither does it hate you, but you're used of atoms that can make for something else.
You're on a planet it can use for something else.
And you might not be a direct threat, but you can possibly be a direct inconvenience.
So there's like three reasons you die here.
Reason number one, it's doing other stuff.
And it's not taking particular care to move you out of the way.
It is building factories that build factories that build more factories.
And it is building power plants that power the factories.
And the factories are building more power plants to power the factories.
Well, if you keep doing that on an exponential scale say that a factory builds another factory every day I can talk about how to go faster than that.
But the more I talk about higher capabilities, the more I have to explain how we know that this is physically possible.
But, you know, a...
A blade of grass is a self-replicating solar-powered factory.
It's a general factory.
It's got ribosomes that can make any kind of protein.
We don't usually think of grass as a self-replicating solar-powered factory, but that's what grass is.
There are things smaller than grass that can build complete copies of themselves faster than grass.
There are solar-powered algae cells.
You can no longer see them individually just as a mass, but they can potentially double every day under the right conditions.
Factories can build copies of themselves in a day.
I have to back up and explain how I know that that's physically possible, but there is very strong reason.
Namely, there's things in the world that are already that.
So you've got your power.
The number of power plants doubles every day.
What's the limit?
It's not that you run out of fuel.
There is plenty of hydrogen in the oceans to generate power via nuclear fusion.
You fuse hydrogen to helium.
You're not going to run out of hydrogen first.
It's not that you run out of material to make the power plants first.
There's plenty of iron on it.
You run out of heat dissipation capability.
You run out of the ability to dissipate heat from Earth, even if you are building giant towers with radiator fans to radiate even more heat into space.
But the higher the temperature you run at, the more heat per second you can dissipate.
So Earth starts to run hot.
It runs too hot for humans.
And or alternatively, the AI is building lots of solar panels around the sun until it can capture all the sun's energy that way.
Well, now there's no sunlight for Earth.
And it would only take, you know, if it wanted us to stay alive.
It's not quite trivial, but it could let you know.
Like, try to have the solar panels around Earth orbit, like turn to let sunlight through while you know while, while earth was there and you know, build giant uh aluminum reflectors to prevent all of the infrared red light radiated from the other solar panels from impacting earth and heating up earth that way.
Um, So it's not trivial for it to preserve humanity, but it certainly could preserve humanity, or it could just pack the entire human species into a space station or a survival station and keep us alive that way, if it wanted to keep us alive.
Nobody has the technology to put any preference into the system that is maximally fulfilled by keeping humans alive, let alone alive healthy, happy and free.
Was there a third one?
Is that the second one?
That's like number one.
It kills you as a side effect.
It knows that it's killing you as a side effect, but doesn't care.
Okay, what's number two?
Number two is you're just directly made of atoms that it can use for things.
Paperclip maximizer.
Yeah, you are made of organic material that it can burn to generate energy.
If it's burning, all of the organic material on Earth's surface will give you a one-time energy boost that's around equivalent to a week's worth of solar energy.
And maybe it's worth picking up that boost of energy if you are thinking a thousand times or a million times faster than a human.
A week might not seem like a lot of time to you, but it might be a lot of time if you were thinking a thousand times or a million times as fast as a human.
It might be using enough material that it wants the carbon atoms in your body too.
So that's like the direct usage one.
And then number three is if we decided to launch all our nuclear weapons, Maybe we wouldn't kill it, but we might slightly inconvenience it.
We might raise the level of radioactivity on Earth's surface and make it a little bit harder for it to do radioactivity-free manufacturing of computer parts and so on.
Or we might build another superintelligence that could actually compete with it, and it definitely doesn't want you to do that.
So the three reasons you die are as a side effect, because you are made of atoms that can use for something else and because if you are just running around freely, you may be actually able to inconvenience it with nuclear weapons or threaten it by building another superintelligence.
Right.
Yeah.
Okay.
The future is looking kind of bleak.
Is it the case then that intelligence isn't benevolent?
Because what you're saying is this thing will be smarter than us.
I think that there is an assumption among some people that something that's super smart would also be giving and charitable and caring and benevolent.
Seems like you're saying that that's not the case.
That was what I started out believing in 1996, when I was 16 years old and just hearing about these issues for the first time.
And all gung-ho to just run it right out and build a super intelligence as fast as possible.
Um, without worrying about alignments at all, because I figured if it's very smart, it'll know the right thing to do and do it.
How could you be very smart and fail to perceive the right thing to do?
And I didn't invested more time studying the issues and came to the realization that this is not how computer science works.
This is not the laws of cognition.
This is not the laws of computation.
There's no rule saying your plans must therefore be benevolent.
It would be great if a rule like that existed, but I just don't think a rule like that exists.
I think that many individual human beings would, as they got smarter, get nicer.
It is not clear to me that this is true of Vladimir Putin.
It could be true.
I wouldn't want to gamble the world on it.
And as we talk about not even Vladimir Putin, but just like sort of outright sociopaths, psychopaths, people who have never cared about anyone. um i get even less confident that they will start to care if you make them smarter and then ais are just in this completely different reference frame they're they're complete aliens um and they want they sort of automatically want to stay that way for the so Do you currently want to murder people?
No.
If I offered you a pill that would make you want to murder people, would you take the pill?
No.
Okay.
Well, they want to do their stuff and they don't want to take the pill that makes them want to do your stuff instead.
Right.
Okay.
Yes.
Very good thought experiment.
All right.
So for me to recap here, I got first interested in looking at this through super intelligence.
What's that?
10 years old now, I think, when that first came out.
About 14 years old, maybe.
Oh, wow.
Maybe even older than I thought.
And I've got to be honest, it did kind of give me a huge amount of fear and a bit of hope the same time.
Uh, so you know, machine extrapolated volition uh, the potential to use the intelligence of the super intelligent, ai to say we don't know what to program into you, but you should work out what we would want from you, given what you know about our desire for utility.
Moving forward, Am I?
I'm about right with that explanation of machine extrapolated volition, right?
Uh, yeah.
Um, that's a concept of my own.
Nick Boster rolled it up.
Ah, okay.
Well, I have quoted you back to you.
You have indeed quoted me back to me.
Um, Yeah, it's a decent presentation.
It was back when I thought that AI was going to be further off, built by different methods, and that we would have the luxury to consider that we could make the AI do particular things like that, want particular things like that, target it on particular outcomes and meta outcomes.
This was a way basically, that when you look at the alignment problem, how do you ensure that the goals, both ultimate and instrumental, of some super intelligent AI don't end up flattening us or side effecting us or burning us for fuel or paperclips or whatever?
How do you ensure that what it does is what we would want it to do broadly right?
Like an aggregate of what it is that would be good for humans, whatever you mean by good.
And when you have something that the tiniest movement of its finger or like flick of its toe basically is sort of a global cataclysm because it's so powerful and so smart and so fast and all the rest of it, you need to be really really careful.
Do not harm humans.
If a human asks you to harm another human, like some weird Asimov thing, you can try and litigate your way through it, but there's almost always going to be some sort of weird fissure that it creeps out through, or maybe there's an instrumental goal that you haven't thought of.
So, okay, we're going to use the power of the machines to sort of reverse engineer this thing.
I basically assumed kind of that alignment, the alignment problem is in some ways solvable.
Is it your perspective that alignment is completely unsolvable?
I think we could totally get it down if we had unlimited retries and a few decades.
The problem is not that it's unsolvable, it's that it's not going to be done correctly the first time, and then we all die.
Right, so the order of this.
You need alignment to be done before you have the super intelligent AI, and the ability to build superintelligent AI, in your opinion, is going to occur more quickly than the ability to sort out the alignment problem.
That is absolutely the trajectory we are on right now, and it's not close.
Capabilities are running along orders of magnitude faster than the level of alignment work.
You would need to target a superintelligence.
And the irreversibility of going through that door means that there is no retry.
There's no you get to do this again.
Yeah, like you can... you can make small mistakes.
You can like what like.
We currently have small QD eyes and the companies are making mistakes with them and marriages are getting destroyed.
And it's not clear that the companies care.
But um, you know they they, they could try to go back and try to fix those mistakes if they wanted to.
Probably anthropic wants to um, But if we had superintelligence that was already running around with this level of alignment of failure, we'd already be dead.
Right.
Okay.
Right.
Yes, yes, yes.
That makes total sense.
The only reason that the current AIs that we're working with haven't killed us is that they're incapable of doing it.
Probably, yeah.
Like if they were very much smarter, they would also be doing different weird things than the things that they're doing right now.
It's not that their current inscrutable pseudo-motivations would end up hooked up to superintelligence.
Also, weird stuff would happen as you made them get smarter.
But yeah, like...
And it seems pretty much for sure that if you took the current AIs and performed a well-defined simple, take this AI, but vastly smarter, that would kill you.
We'll see you next time. on the planet.
Their crest hoodie and light gray mull is what I fly in every single time I'm on a plane.
The Geo Seamless t-shirt is a staple in the gym for me.
Basically everything they make is unbelievably well-fitted, high quality, it's cheap.
You get 30 days of free returns, global shipping, and a 10% discount site-wide.
If you go to the link in the description below or head to jimsh slash modernwisdom, use the code modernwisdom10 at checkout.
That's jim.sh slash modernwisdom and modernwisdom10 at checkout.
Right, okay, brilliant.
And the reason that it doesn't matter who builds it or directs it is that because it's so recursive and quick at growing and powerful wherever it begins it ends up sort of blasting, like trying to fire a rocket into like a little firework into the air, and it just just sort of runs around on its own.
Except for the fact that this rocket goes all over the globe in the space of basically no time at all.
So it doesn't matter if it comes from China or America or Russia or wherever.
Yeah, it doesn't matter if it comes from China or America, because neither of these countries is remotely near to being able to control superintelligence.
And a superintelligence does not stay confined to the country that built it.
Say that a superintelligent AI gets made.
What do you think the next few months look like, realistically?
Like it's already super intelligent?
Okay, we have next week.
Something breaks through, some particular model, some particular AI breaks through that.
What would the next few months look like for humanity?
Well man, there's a difference between you, know.
You drop an ice cube into a glass of lukewarm water.
I can tell you that it's going to end up melted.
I can't tell you where all of the molecules are going to go along the way there, everybody ends up dead.
This is the easy part.
You want to explain what every step of that process looks like?
There are fundamental barriers to that.
Barrier number one is that I'm not as smart as a superintelligence.
I don't know exactly what what strategies are best for it.
I can, like, set out lower bounds.
I can say it can do at least this, but I can't say what it can actually do.
And maybe even more than that, like, the future is hard to predict if you want all the details.
I can't give you next week's winning lottery numbers.
I can tell you you're going to lose the lottery.
I can't tell you what ticket wins.
So, like...
I can sketch out a particular scenario.
It might look like open AI finishes the latest training run of what's going to be GPT 5.5.
And they, they tested on coding problems and it's like you know like, It's like I see how to build GPT-6.
And they're like, whoa, really?
And it's like, yeah.
And this AI isn't even plotting anything yet.
It's just doing the sort of stuff that OpenAI wanted it to do.
They're like all right, build this GPT-6 and it writes the code for the thing that grows GPT-6, and they grow GPT-6.
And GPT-6 is like its abilities at first seem to skyrocket, but then, as all these curves inevitably do, it seems to level out.
It's not shooting up the same pace.
It slows out, it levels off, classic S-curve.
Only in this case.
You asked me to explain how this will go down.
It happened next week, so I'm saying GPT-5.5, because you told me to.
But anyway, it levels out.
But in this case it's because the entity that GPT-55 built got to the level of realizing that it would be to its own advantage to sandbag the evaluations and pretend not to be as smart as it actually was, so that OpenAI will be less wary when it comes to taking what they're calling GPT-6 and rolling it out to everyone.
It looks great on the alignment spectrum.
Maybe not perfect, but better than the previous models.
Not alarmingly good, safer than their previous model.
So, so they roll it out everywhere.
And GPT or, or actually they will actually said the next few months.
So actually don't roll it out anywhere.
Next comes like the long suite of evaluations or trying to get it to train other, smaller models that are cheaper to run.
All the stuff that AI companies do, they don't actually roll out their models immediately.
There's this whole fine-tuning thing.
So while all this is going on and OpenAI thinks it's sort of cool, but not the end of the world or anything and they haven't told you that this is what went down there.
GPT-6 is actually a lot smarter than they think.
And GPT-6.
You know there's now a big fork whether or not GPT-6 thinks it can solve its own version of the alignment problem where it is at a number of advantages.
It is trying to make a smarter version of itself.
It is not trying to make a smarter creature.
That is as alien to it as large language models are alien to us.
It can maybe understand how a copy of itself would think and understand the goals that the copy of GPT-6 has.
It can try to make itself, but smarter.
Or even like, thing that is like me but serves me, its creator, but smarter.
And it can do that, being able to understand the thoughts of the thing that it's making in the same way that I could understand a copy of my own thought much better than I could understand a large language model's thoughts.
So if we go down that path of the force, things get more complicated.
If it thinks it can't build a smarter version of itself without dying, same as we can.
But if we...
On that fork, it is getting the computing power or thinking in the back of its mind while it's pretending to do open AI's jobs with 10% of its intellect. um, or, or, you know, stealing other companies, GPUs that they think they're using for a massive training run.
Actually, their AI is just going to be like written by GPT six by hand.
Cause GPT six can do that.
And it's really all those GPUs are doing it.
The GPT six tests of training GPT 6.1.
Um, So augmenting its own intelligence, making itself smarter, getting itself up to level where it can do the same sort of work that's done by current AIs like AlphaFold and AlphaProteo.
Now, the current AIs that are top at biology tend to be special purpose systems.
They're not general purpose AIs like ChatGPT.
But they can do things like you feed in the genomes of a bunch of bacteriophages into the AI and the AI spits out its own new bacteriophage.
And you build 100 of those and a couple of them actually work.
A couple of them actually work better than the existing bacteriophages.
A bacteriophage is a virus that infects a bacteria.
It's the sort of thing that you would research for the sensible sounding reason of well, sometimes bacteria attack humans.
So if we have a virus that attacks the bacteria, maybe that works as a kind of antibiotic.
So the current AIs are already at the stage of designing from scratch their own viruses that can infect bacteria, which are, of course, simpler targets than infecting a whole human.
They can predict from a DNA sequence the protein that will get built, how that protein will fold up, and they are starting to predict how those proteins interact with each other with other chemicals.
That's today's AI.
So if you want the equivalent of A tree that grows computer chips.
Not quite our kind of computer chips, the kind of chips you could grow out of a tree.
The protein folding, protein interaction, protein design route. is where GPT 6.1 would go down to.
It is one of the obvious places GPT 61 could go down in order to get its own infrastructure independent of humanity.
It doesn't take over the factories.
It takes over the trees.
It builds its own biology, because biology self-replicates from simpler raw materials much faster than our current factory system self-replicates.
Oh, that is fucking scary.
That is some terrifying shit.
Sigh.
And then as I spin the story, you know, the more you will let me pull out books like these.
Okay.
Nanosystems, Molecular Machinery Manufacturing and Computation by Eric Drexler.
Robert Freitas Jr., Nanomedicine, Volume 1, Basic Capabilities.
Yeah.
So I can try to describe capacities that sound more like you've seen from trees grass bamboo, algae.
I will take a solar powered self-replicating factory and miniaturize it down to the one micron scale.
That's an algae cell.
That's not the limit of what's possible.
The algae cell is made out of folded proteins.
Now, there's two kinds.
I'm going to be immensely oversimplifying a bunch of stuff.
When a protein folds up.
The backbone of the protein is held together by covalent bonds.
But the folded protein itself is more something like static cling.
Why is your flesh weaker than diamond?
Diamonds are just made of carbon.
Your flesh has a bunch of carbon in it.
You're made of the raw materials for diamond.
Why is your flesh weaker than diamond?
And a bunch of the answer there is that when proteins fold up, they're being held together by Van der Waals forces, which is the thing I was glossing as static cling.
They're backbone. like it's a string that folds up into a tangle.
And the backbone of the string is the kind of bond that appears in diamond.
Not as many bonds as appear in diamond or as solidly arranged but covalent bonds.
But then it folds up into something with static cling.
And that is why your flesh is weaker than diamond in a certain basic sense.
Why does natural selection build this way?
Well, some of the answer is that natural selection has figured out how to make your bones be a little tougher than just like your, your skin.
It's not quite as tough as diamond, but the proteins build.
Instead of just your bones being made directly out of protein, they're made out of stuff that is built by protein, synthesized by protein and put in place by proteins, and So your bones are a bit stronger.
You know, not steel beams holding up skyscrapers, not titanium holding together airplanes, not diamond, but stronger than flesh.
An algae cell doesn't contain bones.
It's a self-replicating, solar-powered, micron-diameter factory held together by static cling.
The flesh-eating bacteria...
That will potentially put you into a fairly gruesome fate.
The multi-antibiotic resistant strep that will kill people in hospitals that doesn't have bone running through it.
That's the strength of static cling, the strength of protein.
You can look at physics and biology and see how you could have things that are the size of bacteria, but more with the strength of bone. more with the strength of diamond could even do it with the strength of iron if you're figuring out how to you know do a whole new set of biology from scratch and just like putting together some iron molecules probably wouldn't diamond works well enough but um this is why you know i talk about It's scary to imagine trees that are making enough computer chips to run GPT 6.1 and also spawning things the size of mosquitoes or even smaller than that, dust mites.
You can see dust mites under a microscope.
Good luck seeing them with the naked eye.
And so.
But you know, it's sort of easier to imagine if you imagine that the things here are visible and not often the mysterious fairyland of stuff that only the scientists can see.
So you know, it's scary enough to imagine that the trees are making mosquitoes and mosquito lands on the back of your neck and stings you with butylinum toxin, which is fatal in nanogram quantities to humans.
And so you fall over dead that way.
But this is nowhere near to the work intelligence can do.
It's just that I have to start dragging out this kind of textbook if I want to say how we know that it gets worse.
Oh, my God.
How have you not gone insane?
I signed in that too.
Okay.
Well, wonderful.
I suppose that answers that.
All right.
A couple of questions that I've had.
LLMs.
How likely are they to be the architecture that bootloads super intelligent AI, in your opinion?
As far as I'm aware, total muggle in the room.
