Welcome to the podcast.
I'm your host, Jaden Schaefer.
Today on the podcast, Microsoft has made a huge announcement when it comes to AI chips.
They've announced a really powerful new chip for AI inference.
So today on the show I want to break down it's called Maya 200, what it does, why it's a big deal for what we're going to be seeing with AI in the future.
Before we get into the podcast, I wanted to mention if you want to build AI tools without knowing how to code, without being a developer like myself, I would love for you to try out my platform, AIboxai.
We have a vibe tool builder where you can describe a tool that you'd want to create, whether that is.
I just created one that creates profile pictures for people, or you upload an image of yourself and it has this right kind of lighting and it kind of creates all of these different things that you're looking for.
And, you know, these are great for business portraits or all sorts of other headshots for LinkedIn or other platforms as well.
But I just created this tool without being a developer.
I put in a prompt and it linked together a whole bunch of different AI models to create this perfect tool for me.
So If you want to be able to build tools like this without knowing how to code, go check out AIboxai and give it a try.
We have over 40 of the top AI models, everything from Anthropic to DeepSeek to Google Meta Mistral OpenAI Perplexity XAI, Quen tons of image, audio and text models on there.
You can build some amazing tools without knowing how to code.
So go check it out.
Now let's get into the episode.
Like I was saying, Microsoft, they just launched their newest custom AI accelerator.
It's called the Maya 200.
So This is their very purpose built.
It's a silicon platform and they are aimed at one of the most expensive and also it's one of the most complex parts of modern AI systems.
If you're looking at this from kind of like an operational perspective and that is large scale inference.
The Maya 200 is the successor of the Maya 100, which Microsoft.
They actually launched that one back in 2023 as it was kind of like their first serious in-house AI chip that they were creating.
This new generation, now that they've made the 200, is a really big step forward.
So there's a couple of things that it does.
Number one is just raw performance and then also how tightly the chip is integrated into Microsoft's kind of broader cloud and also AI stack.
So, according to them, the Maya 200 has more than 100 billion transistors and it's capable of delivering up to 10 petaflops of performance in a 4-bit precision and roughly 5 petaflops in 8-bit, which is a massive increase over the last generation.
And I think it's really trying to optimize for just running larger language models efficiently and doing this in production.
It's interesting to me seeing Microsoft get into the chips game.
There's a lot of competitors in this space, but not a lot of competitors that could really compete at this level.
And Microsoft, I think sees just how much money they'll have to spend, let alone, you know not to mention just how they're not able to customize everything the way they like if they're using outside suppliers for this.
So it's interesting for me seeing them get into this.
And for those that are curious, right inference is essentially just the process of executing a training AI model to generate outputs, as opposed to training, which involves teaching the model in the first place, right?
So we have inference, which is getting it to generate for you.
And what's interesting is I think we talk a lot about the GPUs involved from NVIDIA.
If you want to train an AI model, and just you know how intense that can be.
And yes, it does cost a lot of money, it is very intense.
But I think it's also important to remember there are millions of people around the world using these AI models.
And we also need to optimize the tech stack for people that are generating stuff.
So I think we'll training oftentimes gets a lot of kind of like the headlines and People talk about it a lot because it's basically this kind of massive upfront compute demand, right?
Like in order to train one of these models, you're spending millions and millions of dollars.
I think inference is quietly becoming a really dominant cost center for a lot of these AI companies because their models are, you know, getting deployed to millions of users.
That's chatbots.
And then if you look at Google, that's like all of the search tools.
You have co-pilots for Microsoft and a bunch of others and a lot of the enterprise software.
So every query autocomplete or you know, generated paragraph.
Every bit of that is consuming compute power and cooling.
So, as a result, even like a very small efficiency gain at the chip level can translate into some really big cost savings at cloud scale.
So it's interesting because this is, you know, obviously something Microsoft's concerned about, but every other AI company should be, and is concerned about this as well, because they need to make those, you know, they need to make the cost savings not just when they're training the model, but when they're actually generating stuff.
Microsoft.
Right now they're betting that, this new kind of Maya 200.
It's going to be a really big shift in that financial equation.
They said that the chip is going to be designed to essentially run today's largest frontier models, so you can imagine the ones that they partnered with, like OpenAI, and they're going to be able to do that on a single node while leaving enough headroom to, you know, accommodate larger and more demanding architecture in the future, which is kind of interesting.
Right, they're not just looking at what is OpenAI, what do, what do our AI models need today?
They're looking at what is it going to need in the future.
So I think, because they kind of have this design, it's very forward looking.
I think this matters because model sizes are continuing to grow.
And I think, as companies are increasingly, you know, expecting lower latency, they're expecting kind of this always on AI service rather than a batch style workload right.
Like you're not going to.
I think in the olden days or olden days, but like in the past, it was kind of like hey, we need like an AI model that's going to go and run through and do this massive project, it's going to get this huge batch done for us and then we're going to be done when you're looking at like how consumers and how the enterprise is using it today, people are pinging this all day, every day, always.
It needs to always be on.
No one wants latency and so it needs to kind of accommodate for that.
Not just like this huge fluctuating, big usage and then a lull it's.
It's kind of like we're we're getting this constant, steady usage.
So, beyond just like the performance, i think the power efficiency is a really key part of all of this.
Data centers.
Right now, they're already straining against energy constraints.
We have even to the top levels of the government talking about um look, you guys need to be building uh power, power generation or, you know, power creation in some way alongside your data centers, because there just isn't enough and that cost gets passed on to the consumer, like if you live in an area with a And they're all getting subsidized.
You are paying for essentially your power is going to be more expensive.
So I think the AI workloads are getting really intense right now.
But essentially, by designing this chip and creating its own silicone, Microsoft can tune Maya specifically to its data center layouts, which is a really interesting thought.
Microsoft being one of the biggest players buying and building these data centers, they could build a chip that's specifically designed for how they structure and run their data centers.
And that's like the cooling systems and it's kind of the software framework and they can do all of this to reduce any sort of wasted power and they can smooth out the deployment at scale.
So I think that's a really interesting vertical integration.
It's difficult to achieve with any sort of off-the-shelf GPU alone that you might get from NVIDIA or anyone else.
And so I think this kind of Maya 200 chip is also reflecting a really big shift in the whole industry, right?
The whole world's largest cloud providers.
More and more they're getting into designing their own chips to try to reduce their reliance on NVIDIA.
And let's be honest, NVIDIA's GPUs have become basically the backbone of the AI boom.
But I think it also like remains that they're very expensive, they're very supply constrained, it's hard to get them, and so i think google was kind of pioneering this whole approach.
Uh, years ago they had their tensor processing unit, their tpus, which are now on.
You know, they're offered as a cloud service rather than kind of a standalone hardware.
Amazon also followed up and kind of copied google.
They did tranium and Inferentia.
It's kind of their in-house accelerators for training and inference.
And then recently they also rolled out a new generation that they were kind of aiming at improving some of the price performance and you know, for like larger models and stuff.
So we see Google doing it.
We do see Amazon with AWS doing it.
So it kind of only makes sense that we're seeing Microsoft get more serious about this.
And I mean, they already had this, the 100 version of this chip.
Now this is the 200.
I think Microsoft is now really solidly positioned with Maya kind of as like a peer for some of those other alternatives from Google and Amazon.
And so in their big announcement they said that it delivered roughly three times the FP4 performance of third generation Amazon Tranium chips and exceeded the FP8 performance of Google's seventh generation TPUs.
So I think, while those types of comparisons often depend on specific workloads, if we're being 100 honest I think they do show that Microsoft's in, like they're really trying to be competitive, not just internally but also across the broader AI kind of cloud market.
They know that this isn't just them that it's going to be using these for training.
They're going to have other customers and other people doing this.
I think it's really important to remember.
Maya is not, you know, being treated as sort of an experimental side project.
Microsoft says that the chip is already powering internal workloads, which includes models developed by its super intelligence team and also some of their core features of Copilot.
So saying that right now their AI assistant that is, you know, on like opening or on office on Microsoft?
Windows on all of their enterprise tools, it is using this.
So by deploying this internally first, I think Microsoft can kind of validate the performance in the reliability, the cost savings, and then they can go and kind of roll this out to other people.
And it's honestly, I mean, that's the greatest validation.
Microsoft's a massive company.
They have millions and millions of users on their products and their copilot is used by millions of people every day.
So You know, if they're like look, if it's good enough for us, it is definitely good enough for other AI companies.
I think as of this week they started inviting internal developers and some academic researchers or some frontier AI labs to experiment with it and experiment with their software development kit.
And I think that's kind of just basically showing that Microsoft is putting out the groundwork for Maya to become a first class compute option within Microsoft Azure, their cloud platform.
And they're going to do this alongside GPUs and other accelerators.
So if this is successful, it's going to give a lot of different customers that they have more flexibility in how they run AI workloads.
It's going to give Microsoft a lot more control over one of like.
This is really one of the most strategic kind of important layers of the AI stack that They're not going to have to rely on their competitors to give this to them.
And so I think all of that together, it's going to be less about winning some sort of benchmark war with the Maya 200.
And it's going to be more about kind of this long term leverage as the inference workloads continue to scale and the margins are getting tighter.
And so, if you want to own the silicone beneath all the software, I think that is going to prove to be one of the most you know, one of the best advantages in the next phase of the AI race.
So I think Microsoft is really well positioned for that into the future.
Thank you so much for tuning into the podcast today.
If you enjoyed the episode and if you learned anything new, it would help the show a tremendous amount.
Honestly like it'd be a huge help if you could leave a rating and review if you have not left one already.
I read them all.
I appreciate them.
But most importantly, it really helps the show to be shown in the algorithm to more amazing people like yourself that are learning about AI.
I'm trying to get the word out and help everyone learn together.
So if you wouldn't mind doing me a huge favor and helping everyone else learn about AI.
Drop a review on the show.
And also make sure to go check out AI box, dot AI if you want to build AI tools and if you are not a developer.
All right.
Thanks so much for tuning in and I'll catch you in the next episode.
