This message comes from Greenlight.
Parents rank financial literacy as the number one most difficult life skill to teach. With Greenlight, the debit card and money app for families, kids learn to earn, save, and spend wisely.
Get started risk -free at greenlight .com slash npr.
You're listening to Short Wave from NPR.
It seems like these days generative AI is everywhere.
It's in my Google searches, it's suggested as a tool on TikTok, it's running customer service chats.
And there's a lot of forms that generative AI can take, like it can create images or video, but the ones that have been in the news recently, DeepSeek R1, OpenAI's ChatGBT, Google Gemini, Apple intelligence, all of those are large language models.
A large language model is kind of like the predictive text feature in your phone, but on steroids.
Large language models are statistical beasts that learn from example, of human -written texts, and learn to produce text that is similar to the ones that the model was taught to them.
That's Ilya Shumailov.
He's a computer scientist and he says in order to teach these models, scientists have to train them on a lot of human written examples.
Like, they basically make the models read the entire internet.
Which works for a while, but nowadays, thanks in part to these same large language models, a lot of the content on our internet is written by generative AI.
If you were today to sample data from internet randomly, I'm sure you'll find that a bigger proportion of it is generated by machines.
But this is not to say that the data itself is bad.
The main question is how much of this data is potentially downstream dangerous." In the spring of 2023, Ilya was a research fellow at the University of Oxford and he and his brother were talking over lunch. They were like, okay, if the Internet is full of machine -generated content and that machine -generated content goes into future machines, what's gonna happen?
Quite a lot of these models, especially back at the time, they're relatively low quality.
So there are errors and there are biases.
There are systematic biases inside of those models and thus you can kind of imagine the case where rather than learning useful context and useful concepts, you can actually learn things that don't exist. They are purely hallucinations.
Ilya and his team did this research study indicating that eventually, any large language model that learns from its own synthetic data would start to degrade over time, producing results that got worse and worse and worse.
In simple, theoretical set -ups we considered, you're guaranteed to collapse.
So today on the show, AI model collapse.
What happens when a large language model reads too much of its own content, and could it limit the future of generative AI?
I'm Regina Barber and you're listening to shortwave the science podcast from NPR.
This message comes from Charles Schwab.
When it comes to managing your wealth Schwab gives you more choices like full service wealth management and advice when you need it.
You can also invest on your own and trade on thinkorswim.
Visit Schwab .com to learn more.
This message comes from NPR sponsor Acura.
The all -new Acura ADX is a compact SUV built for versatility.
Available all -wheel drive for weekend escapes, Google built -in for seamless connectivity, as well as a Bang & Olufsen premium sound system for high -quality audio.
The all -new Acura ADX is crafted to match your energy.
Visit Acura .com slash ADX to learn more.
Google is a trademark of Google LLC.
This message comes from Schwab.
At Schwab, how you invest is your choice, not theirs.
That's why when it comes to managing your wealth, Schwab gives you more choices.
You can invest and trade on your own.
Plus get advice and more comprehensive wealth solutions to help meet your unique needs.
With award winning service, low costs, and transparent advice, you can manage your wealth your way at Schwab.
Visit Schwab .com to learn more.
Okay, Ilya, before we get into the big problem of like model collapse, I think we need to understand why these errors are actually happening.
So can you explain to me what kinds of errors do you get from a large language model and like how do they happen?
Why do they happen?
So there are three sources.
Three primary sources of error that we still have. So, the very first one is basically just data associated errors.
Usually, those questions along the lines of, do we have enough data to approximate a given process?
So if some things happen very infrequently in your underlying distribution, your model may get a wrong perception that some things are impossible.
Wait, what do you mean by, they are impossible?
Like an example I've seen on Twitter was if you Google for a baby peacock, you'll discover pictures of birds that look relatively realistic, but they are not peacocks at all.
They are completely generated and you will not find the real picture, but if you're trying learning anything from it, then of course you're gonna be absorbing this bias.
Right, you're like telling me now that there's like a lot of fake baby peacock images, But machines don't know that, right?
They're just gonna think, great, this is a baby peacock.
And also there's not that many, like, real baby peacock images to compare to.
Exactly. And those are the kinds of errors that you don't normally see that often because they are so improbable, right?
So, and if people are gonna start reporting things to you and say, oh, your model is wrong here, they're likely to notice things that average are wrong.
But if they're wrong in some small part of the internet that nobody really cares about, then it's very unlikely that you will even notice that you are making us.
And usually this is the problem because as the number of dimensions grow, you will discover that the volume and the tails grow disproportionately.
Not just babies, but babies' birds, not just baby birds, but baby peacocks.
Yeah, exactly. So as a result, you'll discover that you need to capture quite a bit.
Okay, okay, so that's one kind of problem, a data problem.
What are the other two?
On top of it, we have errors that come from learning regimes and from the models themselves.
So, on learning regimes, we are all training our models.
All of them are structurally biased.
So basically, to say that your model is going to be good, but it's unlikely to be optimal.
So it's likely to have some errors somewhere.
And this was the error source number two.
Error source number three is that the actual model design would shape and form your model should be taken, is very much alchemy.
Nobody really knows why stuff works, we kind of just know empirically stuff works.
It's like a black box.
We don't know how it's making these decisions, we don't know where, like you said, in that order, it's fixing those decisions, you know.
Yeah, which parts of the model are responsible for what?
We don't know the fundamental underlying bias of a given model architecture.
What we observe is that there is always some sort of an error that is introduced by those architectures.
Right, right, okay.
So, the three places errors could come from is like one, the model itself, two, the way it's trained, right.
And three, the data or the lack of data that it's trained on.
Exactly, and then we also have empirical errors from, like for example, hardware.
So, like we also have practical limitations of hardware with which we work, and those errors also exist. Let's talk about like how those errors build.
What happens when they start to build upon each other?
Like, can you describe that outcome to me?
Yes, certainly. So what we observe in simple theoretical models is that— so two main phenomenons happen.
The very first phenomenon that happens is it's really hard to approximate improbable events in part because you don't encounter them very often.
So you may discover that you're collecting more and more data, and a lot of this data looks very similar to what you already possess.
So you're not discovering too much information.
But importantly, you're not discovering those, like, infrequent data points.
So those tail events, they kind of disappear.
And then the other thing that happens is that the first time you made this error and underestimated your improbable events, when you hit a model on top of this, it's unlikely to recover from this taking place.
Okay, so like over time you start to like lose the more unique occurrences and like all the data starts to like look more similar to the average.
Originally improbable events are even more improbable for the subsequent model and it kind of like snowballs out of control until the whole thing just collapses through to near zero variance.
So instead of this bell curve, you just have a point in the middle.
You just have a whole bunch of stuff in the middle.
Exactly. And the thing is, you can theoretically describe this.
It's actually very simple.
And you can run these experiments like however many times you want.
And you'll discover that even if you have a lot of data, if you keep on repeating this process and the rate at which this collapses you can also bound, you end up always in a state where your improbable events kind of disappear.
In practice, when we grab large language models, we observe that they become more confident in the predictions that they're making.
So basically, the improbable events here are going to be things that the model is not very confident about.
And normally, it will not make predictions about it.
So when you're trying to generate more data out of a language model in order for another language model to learn from it, over time, basically, it becomes more and more confident.
and then it basically join the generation setup it gets stuck very often in this repetitive loops.
I know this isn't exactly the same but it makes me think of the telephone game.
You know, when you tell somebody a phrase or like a couple sentences and then like the next person tells a person the same two sentences and then like the next person says the same two sentences and it usually gets like more and more garbled as it goes down the line.
I think this is a comparison kind of, kind of works.
Yes, so this is the first thing.
It's the improbable event.
And then the second thing that happens is your models are going to produce errors, so misunderstandings of the underlying phenomena.
And as a result, what you will see is that those errors start propagating as well, and they are relatively correlated, like if all of your models are using the same architecture, then it's likely to be correlated wrong in the same kinds of way, so whenever it sees errors, that may amplify the same narratives that it's observing.
I mean, I'm looking at some of the like image output of these models that are trained on their own data right now and we'll link these images in the show notes, but I'm looking at somebody's handwriting of like 0 -9 and you know it's not perfect.
It's handwriting. But like as it gets regenerated by the models over and over like 15 times, they're just dots, right?
Like they're not distinguishable.
You can't even tell their numbers, like which one is which. Yeah, so approximations of approximations of approximations end up being very imprecise.
As long as you can bound the errors of your approximations, it's OK, I guess.
But yeah, in practice, because machine learning is very impiric, quite often we can't.
Oh. I love these images.
This is so good, Ilia.
It's so good. Yeah.
So an important thing to say here is that the settings we talk about here are relatively hypothetical in a sense that we are not in the world in which, you know, today we can build a model and tomorrow they disappear.
That is not going to happen.
We already have very good models and the way forward is having even better models and there is no doubts about it.
OK. So, like you said, you know, ChatGPT isn't going to disappear tomorrow.
what are researchers doing to avoid the problem of model collapse?
Like, as a computer scientist, what do you think the solution is?
— I mean, there are many different solutions.
You'll find a lot of different papers that are exploring what are the most effective mitigations.
And it's mostly data filtering of different kinds and basically making sure that the data that ends up being ingested by the models is representative of the underlying data distribution.
And whenever we hit this limit and we see that our model diverges into some sort of a training direction, the trajectory that is making the model worse.
I promise you people will stop training of the models, retract back a couple of steps, maybe add additional data of a certain kind and then keep on training.
because we can always go back to previous models.
Nothing's stopping us.
And then we can always spend more effort getting high quality data.
Or paying more people to create high quality data.
Yeah, so model collapse is not gonna magically kill the models tomorrow.
We just need to change the way we build stuff.
So this is not all doom and gloom.
I am quite confident we will solve this problem.
I like that perspective.
Ilia, thank you so much for talking with us today.
Thank you very much for having me.
It was a pleasure. If you want to see some of the images I was looking at, you know, see the consequences of AI model collapse for yourself, we'll link to those in our show notes.
Also make sure you never miss a new episode by following us on whichever podcasting platform you're listening from.
This episode was produced by Hannah Chinn and edited by showrunner Rebecca Ramirez.
Hannah and Tyler Jones checked the facts.
Jimmy Keeley was the audio engineer, Beth Donovan is our senior director, and Colin Campbell is our senior vice president of podcasting strategy.
I'm Regina Barber. Thank you for listening to ShoreWave, the science podcast from NPR.
Music Support for NPR and the following message come from Edward Jones, What does it mean to live a rich life?
Maybe it's full of brave first leaps, tearful goodbyes, and everything in between.
And with over a hundred years of experience, your Edward Jones financial advisor can help.
Edward Jones, Member SIPC.
This message comes from Greenlight.
Ready to start talking to your kids about financial literacy?
Meet Greenlight, the debit card and money app that teaches kids and teens how to earn, save, spend wisely, and invest. With Greenlight, you can send money to kids quickly, set up chores, automate allowance, and keep an eye on your kids' spending with realtime notifications.
Join millions of parents and kids building healthy financial habits together on Greenlight.
Get started risk -free at greenlight .com slash NPR.
This message comes from Mint Mobile.
Mint Mobile took what's wrong with wireless and made it right.
They offer premium wireless plans for less, and all plans include high speed data, unlimited talk and text, and nationwide coverage.
See for yourself at mintmobile .com slash switch.
