This episode is brought to you by Focus Features.
On March 27th, Focus Features invites you to be a part of the most explosive movie of this year's Sundance and South by Southwest film festivals.
The AI doc, or how I became an apocalyptic optimist, is being called supremely entertaining and the most urgent movie of our time.
The AI doc, or how I became an apocalyptic optimist, rated PG-13, only in theaters March 27th.
For Scientific American Science Quickly, I'm Kendra Peer-Lewis, in for Rachel Feltman.
In 1997, Deep Blue, a supercomputer built by IBM, did the unexpected.
It defeated chess giant Garry Kasparov at his own game, leading to a flurry of headlines about whether Deep Blue was truly intelligent and if computers could now outthink humans.
The answer, at least then, was mostly no.
But it's now 2026 and we have a growing number of generative AI models that are once again making us wonder can machines outthink us?
To dig into this question, a group of researchers aren't turning to chess this time.
They're looking to math.
You wrote a piece that's talking about the challenges of AI and math.
Before we kind of get into the meat and potatoes of that piece.
I have maybe a more basic question for you.
For those of us who may be peaked with high school algebra, when you're talking about AI and math problems, what are the kind of math problems we're really talking about?
That's actually a lot of what this story is about.
Is that the kind of questions that mathematicians ask and spend their time thinking about kind of don't really sound like or have anything in common with the problems that we work on for homework in math class?
If you've recently taken a math class, you're used to problems that have answers, right?
And the answer is like a number or something.
And you hand in your homework and the teacher can check that number if it's the right number or the wrong number and they give you a grade.
But what research mathematicians are doing is trying to prove that statements are either true or false about the mathematical universe.
So what does that mean?
Like, you know about triangles and squares and basic shapes.
I did graduate from kindergarten, yes.
That's about as far as I made it to.
There's way more complicated shapes that exist in many dimensions and have weird curvatures that you can't even picture in your mind.
But mathematicians are able to say things about them, right?
Using equations and using proofs, they're able to learn about these objects that we can't actually see or picture.
So now that we kind of know what math is In your piece, you note that LLMs have had some mathematical wins, like Google Gemini DeepThink achieved a gold-level score on the International Mathematical Olympiad, and that AI has solved multiple Erdos problems.
Why isn't that enough to show AI's math prowess?
Yeah, I mean the thing about most of these so-called benchmarks.
That's what they call them for a lot of reasons.
AI companies have fixated on mathematics as like the next thing to prove that LLMs can think or to take a step towards intelligence.
But most of those examples, like you said, they have more in common with the kind of test questions and homework problems that we were just talking about.
Not really looking like research math, right.
Which is more about proving statements about the world and exploring that world, posing questions that are interesting.
So in a way, all of those accomplishments are very impressive.
It's crazy that a computer can win gold on the math IMO.
But it doesn't say much about whether and to what extent a computer can advance mathematics right on its own or even with the help of a human.
Kind of like the difference between a really good calculator.
And a mathematician.
Exactly.
Yeah.
Like mathematicians have come across in the history of mathematics.
New tools have been invented time and again that have been useful for mathematicians and have accelerated things.
And one of the big questions here is this just another one of those tools, or is this going to fundamentally revolutionize how mathematics is done, at a level that we've never seen before?
And it's kind of too early to say.
And one of the ways it seems that people are trying to suss out whether AI is kind of just a giant calculator, is or can really advance math, is this first proof challenge that was put together by a group of 11 mathematicians.
Can you explain what this challenge was?
Yeah.
So these mathematicians who are like luminaries in their various fields of mathematics and they cover a broad range of subfields in mathematics, they wanted to rectify this situation where we don't really have a good sense of how good AI is at posing and solving real research math problems.
All of them have had this anecdotal experience where LLMs have gotten a lot better in just the last few months at interrogating mathematical questions kind of in the way a mathematician would and at proposing proofs and methods of proof that seem to bear out in some situations.
But then they also hallucinate a lot and they propose a lot of very confident nonsense.
So these mathematicians, who, by the way, don't work for AI companies, right?
They decided to get together and pose actual research questions that they are trying to solve for their own mathematical research right.
So each of them has papers that are coming out with proofs, and each of them took a little section of that proofs.
The way mathematicians do proofs is they break them up into smaller theorems, right?
So if you wanted to prove that 7 is bigger than 3, you might first prove that 7 is bigger than 5 and then prove that 5 is bigger than 3, right?
And that's kind of how mathematicians work.
And these smaller proofs are called lemmas.
What these mathematicians did is they each took from an upcoming paper a lemma that they proved as part of their bigger proof and picked it out of that paper, posed it as a problem for an LLM, and did all of this before uploading that paper to any online place so that it's not in the training data of the LLMs right.
Because any math problem that I could pose an LLM has probably been posed before and probably an answer exists on the internet.
So these are real cutting edge research questions.
And if an LLM can solve them, then it would be substantially able to contribute to the Yeah.
So, for this first round, different AI companies using their best models and a lot of mathematicians on staff tried their hand at the problems.
And we can't really see the practice that they put into place.
We can't see, in some cases, their full transcript with the chatbots.
We don't know to what extent they consulted with human mathematicians.
And, as one of the first proof team, Lauren Williams, said to me, once there's humans involved in the process at all, it becomes really hard to say how much the humans are doing and how much the AIs are doing.
So the team really wanted this originally to just be like you ask an AI the question, see if it answers the question.
So they did this before the challenge with publicly available chatbots.
And the chatbots were able to answer two out of these 10 questions, which is impressive, but to some extent it shows that this is a real difficult challenge that we're giving to the AIs.
This tiny quarter of the internet that only I pay attention to went really crazy trying to solve these problems.
It shows that there's this growing online community of mathematicians and kind of math enthusiasts who maybe aren't research mathematicians, who are trying to use LLMs to do pure mathematics.
And this community really tried their hand at these problems and produced a lot of proofs posted on social media and Discord servers.
The first proof team posed these questions and they uploaded the answers in an encrypted form and told the community that they would decrypt in one week.
So they gave the world a week to try to answer as many of the questions as they could.
And this online community went crazy trying to do so, produced a lot of proofs.
A lot of them instantly from my reporting were clearly garbage.
Mathematicians who I talked to said, yeah, most of these proofs are nonsense.
But some of them had some promise.
So OpenAI initially claimed that it had solutions to six of the problems.
Pretty quickly, a mathematician found a problem with one of those.
So it was down to five.
The rest of those seem to have held up.
So OpenAI seems to have gotten five correct with its unknown process.
Google Gemini also released its results and it did similarly.
It got six out of 10 correct.
And some of those were different ones than OpenAI did.
The active online community and some research mathematicians who were trying their hand got a couple of questions as well.
Questions nine and 10, which the researchers said were answerable by AI.
Other people produced those answers.
There's a few things that were striking to me about these results.
One is that there was this huge discrepancy between what people with publicly available models can do and these in-house efforts of these giant companies.
Right,
It's a big difference to get one or two correct than to get.
The other thing is that people aren't using one LLM.
They're using what they call a scaffold.
So they'll have an LLM and then they'll have a bunch of other LLMs, systematically interrogate its answer and go back and forth with it, right?
This is allowed.
It's not a human in the loop, but it's a bunch of AIs all talking to each other in some way.
And it seems like this is a way to boost the performance of these LLMs.
They do much better at sussing out some of the nonsense and producing a real proof.
There was a quote in the piece that I thought was interesting, which was that it said that when it got the correct answers that the LLMs were using almost like 19th century style math.
And I was wondering about that quote and like, what does 19th century style math mean?
Yeah, this is a really important point.
AI seems to, at least right now, do math a little differently.
And in a way that's a little less impressive to at least some of the mathematicians.
In many cases, the AI will produce a proof that gets to the same conclusion as the mathematician's proof that decrypted that Friday.
But it does it in a much more circuitous, roundabout way and with a lot of brute force, in a way that isn't as aesthetically pleasing to mathematicians.
Mathematicians.
Sometimes, when they describe what they're doing, they sound more like artists than scientists, right?
They really like to have what they call a beautiful proof, something that, when you read it, you really understand why that statement at the end must be the case.
And AI tends to produce these proofs where every step makes sense and you get to the end and you see the statement, so you believe it, but you don't see the whole picture.
And maybe the AI never saw the whole picture.
Where do you think it goes from here?
One of the researchers, Mohamed Abouzaid, said this thing about 19th century mathematics because...
When mathematicians prove something, they'll often do it by coming up with some new mathematical concept that distills the truth and is easier to work with than anything that existed before.
So this is an abstract object like a tesseract.
AIs don't seem to prefer to do that.
They're very happy to work with existing tools and just assemble them in new MacGyver-y ways.
But it's not clear that that will lead to new discoveries.
A lot of times, those tools that mathematicians invent along the way to a proof give them a deeper understanding of the mathematical universe and lead to more results.
So, at this point at least, it's not clear if AI is capable of that kind of creative style of mathematics.
But there's counter examples.
There's at least one other proof.
On one of the servers where people are discussing these results, multiple mathematicians reviewed it, not only said it was correct, but quite beautiful.
And it accomplished the proof in a way that they never would have thought of.
So it's not clear that this is something that is always going to be the case about AI.
Maybe it just needs to keep getting better.
That's interesting and a little bit creepy.
The next round is going to tell us a lot more.
The First Proof team is working with AI companies to establish controls on the way that they do the questions.
So, whatever answers we get, we won't have to take with so much of a grain of salt, because And that will really tell us where the models are at and whether these in-house systems are actually much better than what's on the public market.
And also the fact that we now have the system of iterated rounds.
We can see the LLMs evolve over time.
So where does this go from here?
I don't know.
There's mathematicians who will tell you that mathematics will never be the same, that AI will be solving some of the biggest problems in mathematics in the next few years.
And there's mathematicians, who I talk to, who were even convinced by this first proof, first round, that timeline is going faster than they thought prior.
What I'm hearing is that Terminator was a documentary.
Yeah, about the future, I guess.
There's also plenty of mathematicians who will tell you that AI can never do what humans do about math, which is direct curiosity in new directions, and that the best it can ever be is a tool mathematicians use just like a calculator.
I have trouble not being bummed out when I imagine a future where AI is solving the big problems in math, like isn't part of the excitement that humans solve the problems, but Multiple mathematicians have pushed back on that.
They'll say, no, they just want to know things about the mathematical universe.
They don't care whether an AI tells them or they do.
One mathematician used this example, this thought experiment from a Borges story, the Library of Babel.
So he's saying, imagine a world where we could just have access to any mathematical truth.
We had a giant library that contained all the proofs you could ever have.
And his point was that any mathematician he knows would be ecstatic to be in that library and would get right to work trying to understand things.
The point is that the job of a mathematician isn't going anywhere.
It's maybe an exciting time for mathematicians.
For me, it's hard imagining a future where I won't have the human side of the story.
Definitely like reporting on a big math proof will be less exciting if I don't hear about the person who was stuck late at night at her desk like struggling through a problem, beating her head against the ground until she came up with that.
Like moment of illumination and also collaboration.
Like the stories of mathematicians meeting up at conferences and having that key discussion over coffee that leads to like a fundamental breakthrough.
So I hope humans stay in the loop.
I do too.
Thank you so much for taking the time to speak with us today.
Thanks so much for having me, Kendra.
That's it for today.
See you on Friday when we explore the science.
Grainger knows that as the procurement coordinator at a manufacturing plant, you can count on auto reordering to have the PPE you need on hand.
Call 1-800-GRAINGER, click grainger.com, or just stop by.
Grainger, for the ones who get it done.
