Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz.
Ian Buck is here with us today.
Ian is Vice President of Hyperscale and High-Performance Computing here at NVIDIA, and he's here to discuss mixture of experts, the architecture powering the world's leading frontier models and how extreme co-design can both drive down the cost of generating intelligence today and future-proof your AI platform for whatever advances come tomorrow.
Ian, welcome.
Thanks so much for taking the time to join the podcast.
Thanks, Tom.
Glad to be here.
So let's jump right into it.
What is mixture of experts, MOE as we call it?
Why does it matter?
If you look at the top 10 open models on artificial analysis right now on their leaderboard, they all share the MOE architecture.
So can you explain kind of in lay terms what MOE is and why it's suddenly become the standard for frontier AI?
Yeah, it's a great question.
I think it's a term that is used in industry and amongst AI researchers, but it's not really understood.
What does mixture of experts mean?
We've all heard of neural networks, and that's what these neural networks are.
They're neurons, they're parameters, they're components of an AI model.
And when AI got started and really became in the zeitgeist of the world, the neural network was simply.
Each parameter represented a neuron of the model.
And we heard about a 1 billion parameter model and a 10 billion, a 100 billion, now a trillion parameter models.
Those are basically the neurons of the AI brain that you activate when you ask ChatGPT a question.
But something happened along the way.
As these models got smarter and smarter and smarter, they naturally got bigger and bigger and bigger.
In fact, two years ago, when LAMA first came on the scene, there was A 7B Lama and then there was a 70B Lama and now we have a 405B BB billion parameter model.
And that makes them smarter.
They have more information, they understand more things, and they give you better answers.
But there was a problem.
As they got smarter and smarter and smarter.
To get the answer you actually had to ask and activate every neuron in that brain.
So, as a result, while the models are getting more and more intelligent, they're also getting slower and slower, because you have to ask every neuron and calculate every neuron and perform all the math on every neuron on a GPU.
And then it wasn't one GPU, it was lots of GPUs and even more.
Along the way, researchers came up with this idea and they realized, just like a human brain, we probably don't need all of these neurons to ask every question.
Simple questions, probably just a few neurons.
Or different parts of the brain may encode different information.
Let's just activate those.
So to make the AI cheaper, the tokens, which is the piece of data that's flying through, that eventually becomes a word on the screen the tokens cheaper, let's only activate the neurons we need to activate.
And that's what mixtures of experts is.
Instead of having one big model, we actually split the model up into smaller experts.
Same number of total parameters.
But now we train the model to only ask the experts that probably know that information along the way.
And that's part of the training process to build that model.
Once you do that, you can have a model which has maybe 100 billion parameters, 100 billion neurons, but we only ask or activate about 10 billion.
That's a compression mechanism.
It's a way of making AI cheaper, but still being able to encode all the possible information and answer all the questions.
So today, most models today, are achieving higher and higher intelligence scores by taking advantage of having lots of experts and able to have the model as it comes up to the answer.
Ask only the right experts in order to get the right answers.
To put some numbers behind it, You know, we have that LAMA 405B, 405 billion parameter.
That's one big model.
You know, on leaderboards like artificial analysis you mentioned, you know it gets an intelligence score of about 28.
28 is just a weighted score of the benchmarks they tested.
Sure.
But all 405 billion parameters are getting active.
Right.
Now fast forward to like a modern open model like OpenAI's GPT-OSS model.
It has 120 billion parameters, actually a little bit smaller in total parameters.
But when you ask the question, it only activates on the order of about 5 billion parameters.
So instead of 405 billion parameters and all that math and all that cost, it actually only needs to activate about 5 billion parameters.
That's like a 10 to 1 or beyond compression.
Yeah.
It's making it cheaper.
And then it gets an intelligence score of 61.
So it is going from 28 to 61. going from $405 billion parameters to $5 billion parameters.
Way cheaper.
It's not a 10x cheaper.
It's still complicated.
And we can talk why these MOEs are complicated to run.
But artificial analysis does measure the cost to run the benchmark.
So like how to run and calculate the intelligence score.
For LAMA-405B, I think that currently it costs about $200. for them to actually ask a cloud service to get all the answers to create that score.
They asked GPT-OSS the same thing.
Its tokens are cheaper, and it only cost about 75 bucks.
So MREs are making models, allowing models to get bigger smarter, allowing them to get cheaper and, as a result, advancing AI.
Now, of course, across the board, all the leaderboards, they're all these mixture of expert models.
Right.
Correct me, bring you back on track if I if I get off here with the questions.
But from kind of a layperson, use that, that word standpoint.
If I'm trying to wrap my head around this idea of mixture of experts, are the experts divided up in ways that I might think about knowledge?
You know, this expert handles math and this one handles science and this one handles I don't know visual understanding.
Yeah, it's a great question.
That is the art of training these things.
In fact, AI, it's not hard-coded in there.
They don't train a separate model for doing math questions and a separate model for telling you how to make a pizza.
The beauty of AI is that the algorithms that these researchers and scientists and companies like anthropic and open ai and everybody else have figured out is that they can just give it the data and they encourage the model to, to sort of camp, to identify and create these little pockets of knowledge.
It's not prescriptive, it's just the data that they're seeing.
It naturally clumps the activity of these different questions to different experts And then in front of those experts, there's this thing called a router.
And the router actually is able to just look at the string of questions, like what answer is forming, how is it thinking?
And then be able to predict.
You know what this one probably goes to that guy or this other guy.
In fact, today's experts.
They may have on the order of dozens of experts on every layer of the model.
And there's a little router between, and they may actually ask not just one expert but, like at every layer, they may ask two experts or eight experts.
And then there's another unit model which listens to all the experts.
This guy says, I'm pretty sure I got the right answer.
Maybe I got the right answer.
I don't know.
I don't know.
I don't know.
Combines the answer and then goes to the next one.
So that's actually the architecture of it.
It's kind of like you could train one person, one brilliant scientist.
You train an Einstein to be able to answer any question.
That's really hard.
It takes a lot of energy.
That's a very expensive person to hire and have on staff.
Instead, maybe I can hire a couple of domain experts or teach a couple of different people some stuff.
And I just give them all that question.
They can all answer it very quickly in parallel.
And the combined knowledge.
And that's actually how we work today.
We don't work in one person is not a company.
Companies exist because we have all this expertise around.
And the MOE method is basically applying that to AI.
So the models are all trained that way.
There's all sorts of training methods to create the condition where information activations can start grouping and gathering together.
And you can train these little routers and combiners.
And then you just do that and Multiple, multiple layers.
And sure enough, at the end of it, you've got a chat model like GPT-OSS or Kimi K2.
Yeah.
Now, MOE isn't new to 2025.
The idea of the architecture has been around for a few years.
So was it being used?
Has it been being used all along and we just weren't so aware of it?
And then why has it kind of come to prominence lately?
Yeah.
The idea of experts is not new in machine learning.
Before AI there was an idea of combining multiple machine learning models together and how to do that statistically to improve the accuracy.
There's all sorts of history and math around that.
Applying it to AI, though, is relatively new.
There are early versions of we now we now know we're chat gpt.
They were a mixture of expert models, but they were not public, publicly known.
Okay, it really wasn't until um, the deep seek moment uh, which is about a year ago, or really blew the doors open like uh because, deep seek, those researchers uh, were the first to really build a world-class moe based model.
People have written papers about it, but it was One that actually competed and demonstrated the intelligence scores.
That competed even with the closed source models.
And it was a beast.
It was awesome.
It had 256 experts in every layer.
I mean, it did every single optimization.
And as a result, it was extremely cheap to run.
Incredibly complicated but cheap to run.
Because it was so easy it took MOE all the way to the extreme.
And maybe many people think it's kind of where OpenAI was, you know, with the original GPT.
So now, once we had that moment, The first time DeepSeq was run on even GPU systems it actually didn't run that well because we didn't have the infrastructure or even the software to run it that well.
The DeepSeq engineers had written all this custom code to make it run awesome.
But at that point, every model, every researcher realized, hey, this thing's real.
We now can see how we do it.
They made the whole thing open.
They published the paper.
It's a brilliant paper, and it shows the opportunity for MOE.
And since that moment, you can see that every model now has shifted to building MOEs.
DeepSeq sort of shined a light on how to do it, how to train it, how to do inference and deploy it, and sort of kicked off that revolution of MOEs that we've been enjoying.
Right.
So we know the DeepSeq moment was huge, as you just said, for many reasons.
Is that kind of?
Are we going to look back and say like hey, the lights went on then and you know, new things will come?
But for the moment, is everything MOE?
And if not, why?
What's kind of the I don't know, the decision making process?
When would you train a model to be MOE and when would you not?
I think all the models that really are focused on providing an intelligent response.
It makes a lot of sense why they're MOE.
You want to do your best to encode as much knowledge into the neural network so it just knows things.
You don't need to, on pencil and paper, write two plus two to work out that it's four.
So the more neurons you can throw into a holistic model, it gives innate knowledge.
It doesn't have to work that out in a reasoning chain or such things.
So there's a huge advantage to having models be bigger as long as we don't increase the cost.
And that's why MOEs, we want to be able to push the limits of only activating ten percent five percent three percent of the neurons more and more experts and you can see that in the research and the way the models are evolving they're really pushing the limits um of seeing some of the modern models you know they'll have 300 400 experts they're trying to combine now getting all those experts and all that communication is complicated we'll talk about that yeah um but it is innate by you know having that the foundation model with all of those experts allows them to then apply all the other techniques of inference, of reasoning.
It allows models that are smaller to be distilled and fine-tuned for specific tasks.
It creates a foundation for the rest of the AI models around the world.
Certainly some of the smallest models you know for the more dedicated individual use cases.
You know, I've got to put a box around a stop sign or I've got a ring doorbell that uses AI to detect if it's a squirrel or not a squirrel.
You know, those small models may not, you know, they need to do one specific thing.
Probably I can get it, squeeze it down.
I don't need to go to, you know, the complexity of an expert system.
But anything that wants to be agentic, any kind of agent, and pretty much most of the AIs that we interact with purposefully they're all MOEs.
Because they can be thrown, and they need to know, and they need to be able to reason about a wide variety of different stuff.
And it makes AI cheaper.
It lowers the cost per token.
So there's always a driving cost and a continuous like let's increase intelligence and let's lower costs.
We can do both with MOEs.
I was going to ask you about that because there's this, it seems like there's this focus happening now.
You know, generative has progressed far enough and certainly it's.
It's everywhere you know, including the news, the business section, if you will.
There's a shift, kind of from you know the biggest models, raw speed, you know the highest scores to, as you said.
How much does this cost and can we get it to be cheaper while being just as smart, if not more intelligent?
So we're calling it tokenomics, right?
So not in the sense of blockchain or crypto tokens, but you know, as you mentioned, AI systems generating tokens, right reasoning tokens, output tokens, what have you?
So if we're focused on bringing the cost down, how does a more complex system and I'm kind of inferring here a little bit, but I would imagine it's more expensive to train to architect, to train perhaps, not to run, but total cost?
How does a more expensive kind of premium system actually drive the total cost down.
Yeah.
There's a wonderful symbiotic relationship that happens in the market between the AI hardware and the models that are being created to serve AI.
They inherently and they kind of have to make sense.
If the hardware offers a certain level of connectivity, a certain GPU performance, a certain memory size, obviously building an AI model that's even bigger is going to be hard to take to market or even not possible to efficiently train.
So you know, since the beginning of the original Kepler GPUs that were used for those first CAT AIs to today's modern GB200 GB300 NVL72X, you can see a pattern where you know, with every new platform we advance the state of the art or what the capabilities of what NVIDIA is able to offer the compute performance, the memory performance, the connectivity, the IO.
We'll talk about NVLink later.
Those things enable the next wave of building to train the next model, but also to do inference.
You know, it's the they add complexity.
You know, when we started we were doing PCIe cards, little basic graphics cards that plugged into the server equivalent of your PC and use the floating point calculations and the graphics memory and do that computation.
And they were great.
When the AI revolution took off, we saw that by adding more floating point calculations and building a bigger GPU, adding things like HB and memory, adding things like increasing the power beyond what a typical PCIe slot will do, we often would increase the performance of what was capable in the AI, not by the just the percentage of more flops or memory bandwidth, but by X factors.
And that's really because the AI models they were able to build were bigger, smarter and could run more efficiently and could do more things.
You know, TCO, people talk about TCO as the cost.
You know, TCO actually is just, it's not the goal.
Like, in and of itself, it's just the lowest cost.
You want the lowest cost?
You know, buy one GPU.
The goal is actually to improve intelligence, And intelligence per dollar, the cost of that intelligence.
Or if we're at the same level of intelligence say this 60 score from artificial intelligence are we reducing the cost of that intelligence over time?
The tokens that people need to buy are the cost in order to run it.
That's really the goal in every generation of NVIDIA architecture.
We're looking to figure out what technologies can we incorporate expand, double down on, invest in or pull from the community or pull from our partners in order to deliver X factors of performance improvement where the model, even the existing models, like the current MOEs, could get an X-factor performance improvement, while only you know, we're not afraid to add more cost and more technology on a per-GPU basis.
You know, the HP memory, it's a lot more expensive than the old-school graphics memory.
But it only increases the cost in percentages.
Because you now have HBM and because you have the bandwidth that it offers and can connect to that much floating point, you can deliver an X factor in total end-to-end performance.
And we saw that actually.
When DeepSeeker R1 came out, the GPU at the time was the Hopper H200 system.
Hopper had eight GPUs in a server.
They were all connected with MVLink through an MVLink switch.
So we could effectively build one giant GPU of eight GPUs working as one.
That was really important.
The model was so large it couldn't really fit on a single GPU.
It had to use multi-GPU.
And the researchers that built DeepSee took great advantage of that.
It also had MVLink capability.
So we could actually put every expert on different GPUs.
And you could see that.
You could parallelize the work, running things even more efficiently and faster.
And because as those experts all had to talk to each other, they would do that over MVLink.
So that was very important.
Before we had MVLink, you know, you would have to send things over a PCIe bus and only one could talk at a time.
And it was much slower.
Because we have MVLink, all those GPUs can talk to every other GPU at full speed.
It's a totally unblocked, you know, literally at gigabytes and terabytes a second of bandwidth, without any concern for collision.
It was critical for those deep-sea creatures to get good performance.
So obviously it also happened at a time which now we can say is when we're in the heart of bringing and building what is now the GB200 and VL72, where we scaled up the number of GPUs.
We can connect from just eight GPUs in a server to 72 GPUs in an entire rack, a 9x multiple.
Yeah.
Now, that's a lot more GPUs.
So did the cost go up?
And it certainly obviously that many GPUs, entire rack with GPUs, versus servers is a lot more money.
Sure.
In fact, we actually even had to add more technology because we had to take all those NV switches and build a separate NV switch plane.
It does cost more.
But because we did that, we can actually parallelize.
You can take all those experts and instead of having to try to make it all fit and work within only eight GPUs, we could actually get all 72 GPUs working as one.
And that improved performance of just going generation to generation.
Being able to further paralyze and run all those experts across.
It could actually increase the performance so much that we actually got a 15x improvement on running DeepSeeker 1 versus only adding percents about 50 more total cost on a per GPU basis.
Wow, okay.
That actually generated a 10x reduction in the cost per token.
Right, right, right, right.
So we do have to add more technology.
We want to keep running more technology.
NVIDIA is a technology company, but we turn that technology back into performance which, in the net of it, reduces the cost per token, because those 72 it's that much faster.
And as a result, they can actually run, get more out of that rack, more out of the per GP basis.
And we've taken it down from what was Hopper.
It cost about a $1 to get her a million tokens, roughly a million words.
It's now down about 10 cents.
So people look at the rack and they say it's really expensive.
But the way you do that is actually you've put all that investment in MVLink, in all the connectivity, in all the next generation software.
And you also do all that software work to make it all work really well.
And generation over generation, you get that multiple, the 10x multiple in the reduction in cost.
That's just one model.
That same story is playing out for BOSS and everything else.
And those are models that were built and trained and designed for Hopper.
Right.
You know we're entering into the.
You know the starting season, models come out that are trained on Blackwell and you're going to see that.
You know.
Now raise the bar and go even further.
So this, this is the virtuous cycle that we've been working so furiously to make, help make happen.
We add um.
You know we might add percents in terms of cost and complexity on a per GPU basis, but we aim at every generation to deliver X factors of performance.
And as a result, we dramatically lower the cost of per token by 10X.
As I'm listening to you describe, you know, NVLANC and the advances in Getting the experts, getting the GPUs to communicate and kind of act as one.
I can't help but think, like, we need NVLink for, like, Teams meetings so we can get everybody.
We're able, instead of talking over each other, just communicate as one at the speed of light.
That's right.
I'm speaking with Ian Buck.
Ian is Vice President of Hyperscale and High-Performance Computing at NVIDIA and we're discussing mixture of experts and why it's become the architecture as it has been for a while, but now getting public prominence, if you will the architecture behind so many leading frontier models and what goes into not only architecting and training the models but the infrastructure that really makes them hub.
And Ian, I wanted to ask you you talked about this a little bit, as I said with you know NVLink and all the technologies you kind of alluded to as you were describing the MOE architecture.
But what is it specifically about these NVIDIA systems that make them such a good and such a unique fit for these complex MOE models and are able to achieve, as you just described, this lowering cost of intelligence measured per token?
Yeah, it's an interesting and understandable.
It goes back to the original idea about having experts.
We're reducing the cost per token by not turning on every neuron, but only turning on the ones we need.
It's a cost savings.
And we talked about LAMA, the 405 billion parameter LAMA model, that in order to use it you got to activate all 405 billion of those neurons, even though they're not all needed.
Look at GPT-OSS it's 120 billion parameters still a lot, but you only need about 5 billion parameters.
It is smart and is a cost saving measure, only does five.
She also notices, though.
So that's like a 10x less actually more than 10x 1 of the number of neurons we're actually doing math on.
The cost isn't, unfortunately, on GPT-OSS, it's not 1%, actually.
It is X factor slower, it's about 3x less cost, but it's not 1% less cost.
There's a hidden tax to MOE, and it's all about how those experts need to communicate with each other.
In order to get MOEs to run efficiently.
Those experts are all doing their math very very, very fast and they all need to communicate with each other very very, very quickly.
And one of the challenges with MOEs is and as we go and get sparser and sparser and sparser, which makes the models more and more valuable and we're saving more and more cost is Can we make sure that all that math is happening and all those experts can talk to each other without ever going idle, without ever waiting for a message?
You're buying those GPUs, you're paying for them so they can do the math they need to do, not to sit around and wait for someone else to send them something.
Or worse, the network that connects all these GPUs gets gummed up, and now everybody's sitting idle, and that's going to go straight to the bottom line of the cost.
Yeah yeah, so that's the key part.
Uh, and the hidden cost anomaly is communication.
We've looked at, you know, can we make it work with just point to point, like maybe i can just connect this gpu with this gpu and this gpu with that gpu.
It'll be a much lower cost to actually just directly wire them up.
But there's a limit to how much i can do that.
If i take one gpu and i connect it to four well, this gpu now is it's io is split four ways and i can only do that so far, and Even with our hopper systems we had eight.
And there was an NVSwitch chip.
We built another chip specifically for this.
But we can't scale beyond that eight because that's the chip.
So if you have point-to-point or a Taurus-like network, you're fundamentally limited by how much MOE, how cheap you can make those tokens.
Because the hidden cost is communication.
And if you try to go bigger than the neighboring or point-to-point connection or some kind of loop or message passing thing or use a fabric like Ethernet, they weren't designed for this.
The best answer is no compromises.
I want this expert, this GPU, to be able to talk to every other expert at full speed, no limitations, no worry about congestion.
I need a network.
I want to connect these things so there's nothing blocking.
Yeah.
And that's what MVLink is.
In fact, that chip that we built is specifically designed to make sure that every GPU and all of its terabytes a second of bandwidth can talk to every other chip at full speed and never compromise on the maximum IO bandwidth we can get out of every GPU.
We did that with Hopper, with 8-Way and one of the big innovations, and obviously it took a lot of engineering to make that 72 racks, every one of those 72, every one of those GPUs at full speed, no constraints.
And you can see that taking off.
You can see the benefit that allows people to go even further and build even bigger models.
The Kimi K2 model is even bigger than the GPT-1.
We now have open source trillion parameter model Kimi K2, yet it only uses 32 billion parameters when you ask it a question.
That's like a 3% activation of the brain.
But it's incredibly complicated.
It's 61 layers.
Over 340 experts, they all got to talk to each other.
And as a result, we now have open models that are trillion parameter scale levels of intelligence.
And the cost is all comparable to what, and even lower than what, we could ever possibly have with a fully dense model.
It's possible because of that emulating connectivity.
NVIDIA is committed to like, let's keep going down that path, build.
We have some of the world's best certies engineers, signal processing engineer, wire engineers, mechanical engineers to make all that work without having costs explode and make it all connected.
Every one of those GPUs, by the way, is connected with a copper wire to one switch, to another switch.
There's a reason why it all sits in the rack is because we're running at 200 gigabits per second on every one of those wires.
It's PAMP4 signaling, so it's like four bits per wire.
It's a 0, 1, 2, 3, and 4, not a 0, 1.
We've gone past the binary at this point.
And it's going so fast, it's actually...
Its wavelength is about a millimeter, I think.
So we're pushing the limits of physics, keeping it all nice and tight and also doing everything in copper for low cost.
We're super happy with GB200 and what it's been able to do for inference and just keeping the cost and driving the cost of tokens down down, down while intelligence goes up up, up.
So is this getting into what we call extreme co-design?
Yeah.
One of the joys of working at NVIDIA is that we're the one company that works with every company in AI.
Right.
Yes.
And you know we work with them in building their data centers and getting the latest GPUs to them and explaining the NVL72 architecture and building and help build a lot of the software that they use.
We have teams working on PyTorch, on JAX, on SG Lang, on VLM and all the other software that's out there.
And as these model makers are building new models or pushing the limits some inside NVIDIA actually now, but all around the world we can co-design with them how to take maximum utility out of those 72 GPUs, to manage that hidden cost of communication, to make sure every GPU is running at 110 on computing on the fewest possible neurons and doing that seamlessly and incredibly fast, all the while thinking about the next model.
You know what's that next GPT, that next vision model, next video model, the next Sora?
And making smart decisions about how to Add more bandwidth, more communication, more MVLink and the right kind of floating point, and all doing so without blowing out cost or blowing out power and leveraging all the work that they've done up to date so that it can be applied moving forward to the future.
This is the extreme co-design that we do at NVIDIA, and some of the folks that I get to work with and probably watching this get to enjoy, not just to have the fastest and be the fastest, but also to reduce the cost.
Because you talked about tokenomics.
If just our software alone could increase performance by 2x.
You've now reduced the cost per token by 2x direct to the user and the customer or whoever's going to deploy this AI.
I was on a call this morning.
We got a model from a customer.
They wanted some help.
We applied the latest NVFP4 techniques, the latest kernel fusions, the latest NVLink communication, io overlaps within two weeks.
We did, we hit 2x on their model and gave them the code back and and you know, and we're not done.
There's, there's so many places where we can optimize.
I think a lot of people get confused.
They see a gpu, a certain number of flops, and they say oh, that's better or faster.
I'll tell you this stuff's pretty complicated manage and run 72 gpus with 348 experts and all the different kernels and all the different ai and all the different math.
We didn't even talk about KV caching and reasoning models and all the tricks and techniques.
That's an end-to-end problem.
It requires extreme co-design between the hardware what's out of the possible the model builders themselves and the dense and deep software stack that run on it.
NVIDIA actually has more software engineers than hardware engineers specifically for that purpose.
Right.
Yep.
So to kind of zoom out for a second because we've been talking about and kind of get hearkening back to what you just said about, you know, thinking about what's next,
We've been talking about MOE in the context of language models predominantly, you know now, and the GB200 and VL72 is really well suited to that architecture.
But is there a risk of focusing too narrowly on this single model trend of MOE?
What happens when we get, you know, sort of beyond MOE?
What happens?
Is the architecture still well suited?
Is the cost of tokens still going down?
How do you think about that going forward?
And how does the...
The design that NVIDIA has today, how is it ready for whatever the next trend might be?
Well, there's one clear trend in AI is that intelligence creates opportunity.
As the models get smarter, as they start to learn new things or as they specialize in certain areas, They create opportunities to advance that industry, that science, that application, or just make computers more productive for you and I every day.
And in order to do that, we need to make the models smarter themselves.
We need to use techniques like reasoning, which is only going to generate more tokens.
And the only way to advance the state of the art of AI is Well, there's lots of ways.
One way NVIDIA can help is just reduce the cost of tokens.
And doing that MOE, it's just an optimization technique.
If you don't need all the neurons, don't waste time computing on them.
That's an idea.
That's not unique to... lms and chatbots that's that's just a good idea so we see um it may materialize in different ways and how these networks and experts want to communicate or the shape of the models are actually diversifying in lots of ways there's lots of different techniques x mixture of experts is certainly one of them that will stick around for a while there's lots of other hybrid approaches and other things that people are talking about or trade-offs that you can make in order to reduce cost But we see MOEs happening not just in chatbots, but similar sparsity MOE applications being done in vision models and video models.
As the models are expanding into science and not just generating tokens which turn into words that you and I talk about, but work on proteins or working on material properties or understanding, or working on things like in robotics or path planning or logic or business applications.
All of those will benefit from having a large, intelligent model that can be sparsely optimized to only use and leverage the part that is needed for that particular question, that particular use case.
You can always go back down to the squirrel detector and a doorbell.
But there's usually a benefit to having a model that is actually able to reason about or has some multimodal aspects maybe listen to what's going on, see the things around it and be able to make intelligent decisions smartly.
That is going to continue to grow.
And NVIDIA is not just working on MOEs.
We've got lots of different irons in the fire.
There's lots of different models.
The models are diverse.
I get to work in HPC as well.
The whole supercomputing community has now embraced AI, building all sorts of models for simulating physics and simulating weather and things that look nothing like chatbots.
But they're going to use MOEs.
They're going to use every trick in the book because the opportunity is huge.
The ability to revolutionize, like biology, to do drug discovery to, for cancer research alone is uh, an investment that the whole world's making right now.
And they can take these ideas and take our platform and apply them to uh their domain.
Their problem uh, to take a open source model or a general model and fine tune it to be a science model or a, um uh, an application specific model or a business model.
That is possible because they're starting from a really intelligent model that can be, that can learn or be used to turn to each another model To make things possible.
So I'm super excited about MOEs.
I'm super excited and will continue to work on reducing the cost per every token.
And while that may make our technology bigger smarter, more complicated at times and will make it more expensive, it is going to deliver X factors in capability improvement, intelligent and, as a result, dramatically lower the cost per token.
Ian, for listeners who want to dive in further, we could talk about this all day, but you have things to go build and customers to take care of and all that good stuff.
Where can listeners go online?
What's the best place to start to dive into MOEs, to the infrastructure you've been talking about, to any and all of it.
Check out GTC.
One of the things that we started this conference a few years ago over a decade, I guess I was there for the first one.
It's called the GPU Technology Conference.
It's not a business conference, although obviously many business people show up.
It's not a demo conference.
It's a developer conference.
And if you want to learn more, go check out GTC.
We put all the presentations online.
Jensen's keynote is wonderful.
He'll explain it even better than I can.
And we actually do a few a year now.
I encourage you to check out GTC.
Go see the old ones.
And if you're going to be in San Jose in March, please come and check it out and attend.
There's tons of sessions at every level, from beginner to deep dive if you want to go down to the hardware.
All the NVIDIA experts will be there.
All of the different developers are going to be there.
It is kind of the go-to place to go learn and also present your work on what you can do with GPUs and the state of the art of AI.
Check it out.
Perfect.
Ian Buck, again, thank you.
And for what it's worth, Jensen's an amazing presenter.
You did a great job explaining all this.
So we appreciate you taking the time.
And as always, all the best to you and your teams on continued progress.
Thank you.
