Hi Benjamin, here with a podcast extra which features an interview with Yoshua Bengio, considered by many to be one of the godfathers of AI.
Joshua works at the University of Montreal in Canada and has been at the forefront of AI research for many years.
But recently his opinions on the technology have shifted and he spends much of his time talking about his views on the potential dangers to humanity that AI could represent.
Joshua happened to be in London last week and I went to meet him along with my colleague Davide Castelvecchi.
Davide spoke to Joshua about ways to identify and address the risks posed by AI and his efforts to develop an AI with safety built in from the start.
Joshua chairs an international panel of advisors in the field of artificial intelligence, which this year published the International AI Safety Report, which identified three main areas of risk for the technology.
Unintended risk from malfunctions, malicious use and systemic risk such as the loss of livelihoods.
Davide asked Joshua which of these areas is the most likely to have a short-term impact and which keeps him awake at night.
The second one, in other words, malicious use, is already happening, but I think we're only seeing just the shades of it, with things like deep fakes, cyber attacks that are very likely to be driven by the most recent cyber capabilities of AI.
And we need to have much better guardrails to mitigate those risks.
And those guardrails have to be both technical and political.
What keeps me even more awake, of course, is the possibility of human extinction.
That's the extreme malfunction.
That's why I suddenly pivoted my research into the question how do we build AI that will not harm humans by design?
More broadly, I think it's a mistake to focus on only one kind of risk.
As an example.
Right now there's a lot of concern in the US that China could use advanced AI against the US in a military sense or influencing democratic institutions.
And the Chinese have the same fear.
But if we only focus on one risk like this, which is due to country competition, we might miss other risks, such as the use of AI by third parties like terrorists or the emergence of a rogue AI.
Going back to the International AI Safety Report, how do you feel it was received and has it begun to have an impact on say, what governments do about AI?
I'm really excited to see how much impact it's already having.
What it has done is establish, rigorously based on the scientific literature.
What are the risks that we already understand?
And it also establishes what are the current mitigation approaches and their limitations.
So why has it been useful?
For example, there's been many countries which created AI safety institutes.
And so now there's a network of these AI safety institutes that are government entities.
These have greatly benefited from that kind of synthesis of the scientific literature.
It is also actually helping companies, scientists that didn't have familiarity with AI safety to get into the field.
And it is written in a language that every citizen should be able to understand as well.
If I understand correctly, kind of the existential risk posed by AI was not at the top of your worries until a few years ago, but then something's changed.
ChatGPT, November 22.
It took me two or three months to realize we were on a path that could be extremely dangerous.
I realized that we were building machines that already understood the language.
And although I was initially pleased to see that deep learning had finally reached that milestone, I realized that, because of the nature of these systems, we didn't know how to make sure they would behave in the ways that we want.
And i started thinking about my grandchild and i thought oh, in 20 years he's going to be 22.
And will he have a life?
Will he live in a democracy?
What kind of future awaits the uncertainty due to bringing to the world machines that are smarter than us?
It hasn't happened yet, but we are on that path.
That's the existential risk.
But there are other existential risks due to the power of AI in the wrong hands.
There are people who would press the red button, ask an AI to do something terrible that could cause the death of billions of people.
And then there's also risks to democracy, because intelligence gives power.
Whoever will control very advanced AIs in the future will have huge power.
Democracy is about sharing power.
If the power is concentrated in the hands of a few, that is not democracy, that is a dictatorship.
We should not just deny those possibilities, even if the chances were small.
These are so radical and destructive possibilities that we should be very seriously considering them and trying to mitigate them.
Of course, there are people who would say that you're being an alarmist and you're describing really a worst case scenario.
What would you say to them?
And what are some of the key steps to get ahead of these risks?
The really important thing to keep in mind is we don't know what scenario will unfold.
So if you ask experts, they will... have different opinions.
But if you look at the scenarios that receive a substantial fraction of beliefs, they include some really bad ones, including all of those I've discussed.
So you might say, well, let's hope for the best.
But that isn't a good strategy, right, in general.
We should try to steer towards the good ones, which means we need to understand what is going on and we need to then have policies to try to move us in the right direction.
And do you think people are being too gung-ho at the moment?
Yes, and I think that it's very difficult for most people to project themselves into a future with machines that are much smarter than what we see now, even though if you ask them, did you anticipate.
What we see now, five years ago.
Five years ago, most of us would have said oh no, that's science fiction.
If you just do the same exercise but project into the future.
I think we have a lack of imagination, which is very human.
You mentioned the need to develop a kind of AI that has safety built in from the start and it acts responsibly.
Yes.
And you and your team have proposed this idea of the scientist AI.
Exactly.
Do you want to tell us about that?
So we call it a scientist AI for two reasons.
What's the connection between AI and science?
The way that it's designed is very much inspired by how human scientists go about understanding the world and building models of the causal mechanisms and the laws of the world that we're hypothesizing.
And the other reason is that by automating that process of hypothesis generation and reasoning with these hypotheses in a probabilistic way, we can also help the development of scientific research.
AI is already helping scientific research, but currently it is often incoherent in ways that can be bad.
It's trying to please you in ways that are not good for science.
We want the truth, not what we want to hear to come from AI.
And so to go back to the safety aspect, if you think about a scientific field.
Let's keep it simple and think about the laws of physics.
You know, you can make predictions from the laws of physics.
And those predictions, they don't care about you or me.
They don't care about a political party.
They're completely unbiased with respect to human goals.
So if we can build AI systems that are like that, that are really good predictors of what will unfold and also understand the causal structure, like scientists do, behind what will happen,
Well, first, that could be very useful.
But more importantly from a safety perspective, it would make sure that there is no hidden agenda behind such an AI.
Currently, we have AIs that have goals that we don't control.
They want to achieve their mission in spite of the goals that we've given them, and so they will cross red lines.
The scientist AI is non-agentic.
In other words, it has no goal.
It has no intention.
And so we can trust what it says.
Now, you might ask, but companies want to build agents, right?
AIs that do things in the world.
And actually scientists want to build AIs that help them design experiments, which is something you do in the world.
You're not just passively making predictions.
So the good news is that if you have good predictors, you can use them to construct guardrails like predict whether an experiment or the action of an AI in a computer could give rise to bad outcomes, and with what probability.
So, even though the focus of the scientist AI, is a completely non-agentic system, because it is so trustworthy by design because of its properties, it also means we can use it to mitigate the risks from agentic systems.
And a slight shift of topic, how vulnerable is the current state of the art in AI to disruption?
In other words, could someone come up with some kind of clever new idea, some simple algorithm that suddenly makes a lot of the existing technology obsolete?
That's possible.
In a way, it's already happening on a small scale and then many small improvements.
But for now, the different labs leading that race and it is a race are matching each other's progress.
If one company were to make a discovery that gives it a huge advantage, then it would be disruptive, because it would mean even more concentration of power.
And, along those lines, something that has been a stated goal for many of the companies is to use AI to do AI research and accelerate the advances in AI research and then have an advance that can't be easily bridged by others.
To understand this, you have to realize that a lot of the recent advances have been in the abilities of AI to do math, computer science, engineering.
There are good reasons why it's easier to make that progress, but it also means that we could see AIs that are already as good as a top AI researcher, top AI engineer, in just a few years, even though AIs are also not that good in other ways and they can't take the job of that many people.
It could create a virtuous cycle for those doing it.
And there's been a lot of coverage lately of these companies with enormous stock market valuation, and NVIDIA passed 5 trillion mark.
Is there an AI bubble and is it likely to burst soon?
I don't know, but I think there could be adjustments in the markets.
It depends on the reasons why investors are putting money into this.
If they expect very short term profits, then they might have negative surprises, because the rate of advances isn't always what is being sold.
But in the long run, it's very clear that we're going to go and have more and more capable AIs.
At least that's a very likely scenario.
So in the long run, I don't see any reason why AI would not eventually be even much more valuable than it is now.
And I guess this leads naturally to my next question setting aside the catastrophic risks or the short or medium term risks of recession, bubbles bursting and so on?
Do you think that AI will ultimately cause the world's economy to grow faster or become bigger?
Or will it make the world poorer overall?
In terms of GDP, it's very likely to grow because AI will enable much more productivity across the board.
The question is how does that relate to individual humans' well-being?
In particular, if all that wealth is concentrated in a few hands and just a couple of countries, the rest of the world could be in for not such a great ride.
So this brings us back to the International AI Safety Report.
Like the third category, the systemic risks,
I know that the report was explicitly told not to make recommendations for policy, but do you have any personal opinion on this?
Of course.
As a citizen anxious about the future of my children, their life, their jobs and all the people who ask me these kinds of questions, we will almost certainly see very significant effects on the labor market.
In the report we discussed the fact that some economists think that the effects will be small and others think that the effects will be very large.
So if you think that we've reached the peak in terms of capability, then the effects will be small.
Because currently AI is, for most tasks, below the threshold of being able to replace humans.
But if the advances continue which is just the continuation of the current trends, so we could expect this is plausible then the situation becomes very different.
The value of labor in the total equation of economic output is going to go down, because the job that a particular human could do can now be done by a machine for 10 times or 100 times less.
And we could end up in a world where a lot of people feel actually misery because they lost their job and they don't have any other revenue.
So in terms of policy, it's obvious that governments need to start thinking about this.
And finally, do you wish AI had never been invented?
It's a difficult question.
I wish that we had had collectively more foresight about the catastrophic possibilities, so that we would have moved more carefully into where we are now.
Joshua Bengio there, talking with nature's Davide Castelvecchi.
This piece was produced by me, Benjamin Thompson.
For more on this story, look out for links in the show notes.
