Welcome to the NVIDIA AI Podcast.
I'm Noah Kravitz.
Today we're talking micro data centers with Ben Suter, Director of RD at EPRI, the Electric Power Research Institute.
The relationship between AI, data centers and energy grids is an increasingly important one, to say the least.
In a moment, we'll talk about how micro data centers can help strengthen that relationship.
But first, a quick note about GTC San Jose.
Join us at the world's premier AI conference.
GTC San Jose is online and in-person March 16th through the 19th.
From physical AI and AI factories to agentic AI and inference.
GTC 2026 will showcase the breakthroughs shaping every industry.
Learn more and register at nvidia.com slash GTC.
Ben Suter, welcome.
Thank you so much for taking the time to join the NVIDIA AI podcast.
Really glad to have you here.
Yeah, great to be here, Noah.
Super excited.
So Ben, to kind of set the table before we dive in for listeners who don't know EPRI, can you briefly explain well first who you are and what you do and, as part of that, what EPRI is and what EPRI does?
Yeah, absolutely.
So, EPRI is sort of a unique organization.
We're a 501c3 not-for-profit.
It's an independent institute focusing on RD, collaborates with more than 400 companies across more than 40 countries and really drives innovation to ensure sort of the public has reliable and affordable energy.
So really awesome mission statement and been a really exciting place to work.
You've been at EPRI for a couple of decades now.
Yeah, I've been here a while.
I just crossed over the 20-year mark, which I feel like is a ancient times, and the way the corporate world works now.
Right.
Well, congratulations.
And you kind of this is exactly why I asked, because I'm thinking about data centers and AI, but hearing you talk about things like nuclear and thinking like man you've like must have seen some things and worked on some projects and thinking back, you know over 20 years and how technology and energy reliance and consumption must have evolved.
I don't know if this is for a question to ask, but can you kind of place our current moment in context to you know sort of what you've seen with how the world uses energy and stuff you've worked on over the years?
Yeah, yeah, that's a great question, a great way to frame it.
And it gets to why I probably have stayed here 20 years, which is that there's just so much change and a lot of different exciting things that have evolved across the sector and the industry that have sort of landed us here today.
So lots of stuff going on.
It's been interesting as you come in and I'm sitting here in Knoxville Tennessee, and behind me we actually have a big laboratory that makes up the back half of the building.
But pairing with that, over the years here at EPRI I've seen all kinds of technologies come through, whether it's solar energy, battery storage, electric vehicles, all kinds of things.
And what's interesting is you see a lot of it several years, a lot of times in advance of when it's cool and it's blown up and it's everywhere.
Right, right.
And so it's been interesting to just see all these technologies come in and evolve and the challenges that come with them, whether it's how do we handle the loads of electric vehicles or how do we position the distribution system to handle all of the solar capacity.
Right.
All these different issues and then meeting those challenges.
And so it's been an exciting place.
And I've gotten to have several lifetimes here because I've been here for 20 years.
And so working through different areas and now kind of in this AI space, which is obviously just accelerated, everything about 10x.
Right, right.
So let's get into that then.
Data center, AI.
When I say people, when I think of AI and energy consumption, data centers kind of pop to mind sort of immediately.
There's more to that, obviously.
But can you kind of set the stage a little bit for kind of explain what a data center is in this context and then maybe that can get into what this idea of a micro data center is and how it difference you know how those differ uh, from the kinds of things that you know people like me usually think of when i hear data center yeah.
So so good question.
And you know, i i think there are there, there are several flavors of data center at this point um, and i think kind of the the, the two i'm going to sort of hone in on today.
One is the, the data centers that have been really in the news a lot lately.
These multi gigawatt behemoths that are being built with the objective of providing platforms to train these really exciting AI models.
And so there's been an enormous push to build those types of capacity.
Obviously, there's been a big crunch for power in order to meet that demand.
And so lots of exciting research in that area.
But all of that's really been directed at making the models that are going to potentially do exciting things for us in the future.
Right.
But the training of the models.
Exactly.
The training of the models.
Yeah.
And you mentioned this in the plug for GTC at the beginning, inference.
I think people don't realize that, while we're so focused on training the models, there's this huge wave that's coming of once we actually get all these models and we move beyond just chatting with JADGPT and we're doing the real-time translation in our AirPods, and we're doing the smart glasses, and we're doing all the full self-driving and all these different applications.
That all those applications all falling into inference, sort of using the models, is going to sort of accelerate this second compute wave that comes along with all this in order to have the compute capacity to actually do all this stuff.
And there's actually an interesting statistic out there that if you look at the lifetime of a model so if you look at a GPT 51 or whatever only about 20 of its compute capacity and thus its power consumption is in the training side.
80% of it is in the inference side.
80%, okay.
Yeah.
And so the vast majority is actually in the inference side.
So, if you think about how much capacity we're building for training, we're going to need – a couple of times that to meet the demand for all the inference.
And so people start using these things.
Exactly.
Or close to full max.
Yeah.
Yeah.
And so that's going to create another challenge.
Thinking about energy consumption.
Is the distribution of energy consumption during inference as opposed to training?
Is that just massively different and much more spread out?
What does that look like from the perspective of energy load and consumption and figuring out how to try to balance things?
There's a lot of great questions in there.
I try to throw 12, 13 of them at you at once.
And a lot of them are things that we're looking at as part of this micro data center project.
So when the world, when we got into these gigawatt scale training data loads, nobody really realized or thought about the fact that the way the compute and things would happen is the training loads would slam hundreds of megawatts of demand nearly instantly, within milliseconds.
And they can also fall off once that job is done.
And so huge swings of power.
And that created some consternation as you had to solve sort of the technical challenge of meeting those demand peaks and spikes and things.
You compare that to inference.
And when we got into this, we started down this journey about midway through last year.
And I was initially imagining this and I'm thinking about okay, if inference is what, I'm using one of these awesome models, I'm using ChatGBT, I'm using Grok, I'm using Gemini.
And so it's being the compute tasks are being generated by me, So that's going to give it what we call more load diversity.
It's going to kind of smooth it out because it's being randomly generated.
Sort of my initial hypothesis.
Really interesting discussion last week with someone as we started bringing up just the whole agents and agentic AI that has taken over in just the last few weeks.
You know, bring open claw at the house, you know, to take over my world.
And I started realizing like, oh man, like, That's doing all this work at night now.
So when I originally thought this load was going to sort of look like a normal load curve for just people waking up during the day, putting on lights and you know, air conditioners and stuff
Now, all of a sudden, I'm like well, that completely changes the paradigm, because now it's running at night while I'm sleeping.
And is it going to do more?
And so to answer your question, that was a really long-winded way to answer it.
I want to go deeper and ask you what you're doing with OpenClaw, but maybe that's another podcast.
Yeah, that may be another podcast, because that's trying to streamline how you actually survive in the 10x corporate environment.
Right.
But all that to say that that paradigm is sort of evolving now and I'm having to change my hypothesis.
And so when we actually start monitoring these data centers and things and actually building them out and realizing and measuring them, it's going to be really interesting to see what they look like.
And I have a feeling you're going to see lots of different loads because it's something that's very consumer-centric, may look different than, yeah, there were some great stories last week um, of like some big financial institutions that were very AI forward and have, have invested a lot in models and don't have enough compute for their internal minds.
Yeah.
Which is another as a whole another, you know it fits very well into what we're looking at here, but it's a completely different probably, you know, uh Right.
Yeah.
Well, let's dive into what we can kind of grasp at the moment or, you know, is concrete, I should say at the moment.
And this idea of micro data centers.
Can you?
You kind of alluded to it in talking a moment ago, but can you talk a little bit more about what they are and why now, and what are some of the problems?
And these may be some of the examples you're mentioning.
Are you trying to solve for the power grid, as well as for AI users, with this idea of micro data centers?
Yeah, so great question.
So the real thing that we're looking at here.
I mentioned everybody's focused on these big giant training data centers.
Now we're thinking about how do we create these data centers for inference?
And when you actually look at those data centers for inference, one of the things you start to realize is that having the huge mega data centers that are centrally located don't necessarily make sense for the inference data centers, because they are more consumer-centric and user-centric.
Positioning them geographically around where the people are tends to make more sense because they can be more latency-sensitive, et cetera.
So you don't necessarily want to have them just in one place in the middle of nowhere.
Better to have it broken apart.
I may be way off here, but it reminds me of when streaming media centers started popping up, kind of in the whatever period of the aughts, I guess, right.
The first dot-com wave when multimedia used to become a thing.
And yeah, that kind of proximity, because it affects performance, as you said.
Yeah, yeah, exactly.
When you know the early years of Netflix, where it started off very centrally and then they realized hey, if we put a mirror?
Onto the local networks, it becomes a lot easier to distribute.
So yeah, it's another thing.
And incidentally, the biggest user of geographically dispersed servers, game servers.
Yeah, right, right, right.
Through this journey, learn that little tidbit.
Can you walk through a little bit what happens in a micro data center in terms of sort of you know, how do you design and build for an inference load as opposed to a training load?
And what does that mean in terms of both the energy usage but then also like the ripple effect of not housing everything in these central giant megawatt data centers that, as you said, at least for training?
You know they act differently than other big loads on the grid, they come up super quick and you know, and i imagine all kinds of other problems that are beyond my knowledge set, but just can you talk a little bit about how they sort of work on that level?
Yeah so so a couple things, um uh it that are kind of in in that that onion, uh to unwrap.
So you know, i the first one sort of on the the really underlying need underneath construction.
It's somewhat similar in the fact that it's still very sort of GPU or TPU-based compute need in order to actually run these models.
We're seeing, I think, More chips like NVIDIA has more chips, more designed for inference and training.
Now,
So there seems to be a little bit of diversification.
What was now just sort of one chip initially.
And so we're seeing that.
So there is some variability, I think, maybe in the underlying chips.
But traditionally, it's been sort of the same chip for training and inference.
And so from that perspective, it looks similar.
It's just smaller because I don't need as much.
But as sort of the result of, yeah, I don't need as much, it's sort of like, how much do I need?
And that's been one of the things that we've been looking at.
And, one of the interesting things, that we've been working with technology partners like NVIDIA to really help us understand what the compute needs of the actually technology companies that are buying these data centers, using these data centers.
Looking in, that is three megawatts enough.
Is five megawatts enough?
Do we need 20 megawatts?
And we seem to be coalescing somewhere around this idea of 20 megawatts.
But that's actually sort of.
I hadn't gotten into some of the electrical aspects of all this, but as we're looking at where to place these micro data centers, 20 megawatts can be a not insignificant ask of just dropping a load somewhere onto the grid.
And so there's not a lot of opportunities to drop something of that size.
And when EPRI was looking at.
Okay, Our partners are telling us about this coming compute wave and we want to do what we can to help our utility members be proactive and get ahead of it.
Where can we look at opportunities to find power for this type of data set?
One of the things we started looking at was well, there's substations all over the United States and indeed all over the world.
And there's a fair number of them that are actually underutilized.
So they've got excess capacity available inside them.
And so we started thinking like well, is there an opportunity there to partner with those substations that have that excess capacity and do something and, you know, put these inference data centers near it.
And you know, maybe directly adjacent is maybe ideal, but close by and make sure we've got everything that is needed in terms of fiber access.
Right, all the infrastructure.
All the underlying infrastructure.
And so look at all those things and say, does that work?
We thought that was a good idea.
But the answer is you're probably going to find three to five megawatts, maybe up to 10 megawatts, of available capacity in a single substation.
And so then we started thinking about, well, how's that going to work?
Ben, just to interrupt you real quick.
Sorry because I keep having a picture in my head of this is my own ignorance about our electrical grid, of how big one of these existing substations is and where it might be.
Is this kind of like suburban as opposed to metropolis?
Is that-
So it could be both.
Okay.
So there's a couple caveats in there.
So you're right in thinking that your suburban substation may be more likely to have some of that excess capacity.
Okay.
That said, we have found that there's interest at the metropolis level, too.
In capacity.
In capacity, because there is need.
There's people there, so...
They want to get the compute close to it.
And actually, if you see some of the metropolis environments, there's a lot of real estate that's available right now, which equates to load that's not there.
So there's opportunity to put load.
So that was another hypothesis going in.
Yeah, there wasn't going to be interest, but actually it looks like there may be interest and opportunity there at that level as well.
And so you know, as you're looking at these data centers and you start to say well, does three megawatts make sense?
And does it make sense for the person that wants to buy it?
What we realized was maybe there's an opportunity.
And this is the distributed part.
We initially kind of called this project distributed inference, truthfully.
And while distributed inference seemed to be very technically accurate, it did a really poor job of giving anybody a visual image of what it was we were talking about.
And so what we realized was if we go to an opportunity, if we go to a regional area, we go to a city and we say hey, is there other five data centers that meet this criteria?
And then each data center maybe has five megawatts of capacity.
Now we've got five data centers at five megawatts, and now we've got 25 megawatts of capacity.
And so actually looking at it as, instead of a single project that's five megawatts, looking at it as a 25 megawatt project that just happens to be distributed across five sites.
And so that helps meet the needs of what the utility grid has available and sort of meet the economics of what the data center companies need in order to actually make it realistic and viable for them.
Right, right.
How does this approach affect the way the grid functions for just people in general, the city, the region in general?
So great question.
We've really sort of seen this as a win, a general win for everyone, because the answer is if the existing substations are already kind of sunk cost, we've invested that capital, we've made the investment, we've built it.
And so if we can get, you know, extra capacity, if we can get extra usage out of existing assets, then that's sort of a win for everyone.
If a societal cost, if we're not having to put new steel on the ground, then that's helping keep rates lower and things like that.
So we really see this as a positive in terms of being able to leverage existing infrastructure.
Speed to power.
I think is also a big part of this, where there's a huge scramble for this capability and and everything.
And so it also means that you no longer have to deal with interconnection queues because you're off the transmission grid and all the things that go along with that.
So it definitely speeds up the ability to get to a finished product that's online and serving customers much faster as well.
That's great.
Are there clean energy implications?
You know, it's interesting you say that.
So there's definitely opportunities to layer all kinds of things on this.
So there's opportunities to layer this with DER and solar, wind things.
And I think there's also a lot of opportunities for energy storage.
One of the things we've been looking at getting sort of into the technical weeds.
We've been looking at flexibility.
And what you find is that you'll have a substation and it's got excess capacity, but it's actually got quite a bit more capacity, except for July 21st, when you have the hottest day of the year.
Right.
I'm making July 21st up.
That's not the hottest day of the year.
Somebody fact check me.
I was like, wait, what AI breakthrough happened on a July 21st?
Just been made up a date.
Right, super hot day.
Yeah, yeah, yeah.
So if you can engineer it so that you can have flexibility to reduce your load, you can reduce your demand during those peaks.
You actually have a lot more flexibility.
You know envelope that you could potentially use.
And so pairing it with energy storage, backup generators, just working with the technology partners.
One of the other nice things about if you have sort of a distributed network of these loads is if there is possibly like a peak demand issue, I can run down my compute and wait at the center and route the calls someplace else and smooth things out that way.
So there's lots of possibilities.
And so that's another thing that sort of makes this exciting and a really neat way, a tool that the utilities could use as well.
Yeah, yeah.
No, that's very cool.
Continuing sort of along the lines of the applications of all of this, but kind of from the other side of it.
And again you talked about this in reference to building the data centers, these smaller data centers close to where the users are, the consumers are and that performance aspect of it.
But are there other examples of real-time applications that, as this infrastructure rolls out, you think will be enabled or maybe just kind of accelerated, these applications that could directly benefit people?
I think there's all kinds of things and I am certainly not going to claim to have a view into all of those options.
You know, I mentioned some, you know, like the translation and self-driving and things.
But you know, I think, especially as agents develop, as we get, you know, smart glasses that can analyze, just here at EPRI you know other exciting things you know we're looking at.
And these are going to have applications for everybody.
But can you use smart glasses to analyze your poles and transformers and things in a substation and make your line workers smarter, more efficient and safer all at the same time?
And so there's all these applications that everyone's looking at.
Can we again grid-focused?
But can we make the control center of the future smarter and get smarter about restoration times and all these different things?
I think there's just internally at Eprate there's a few hundred use cases and things we've identified.
And that's very grid centric.
So, you know, obviously the audience is probably not all utility workers and things.
But I can only imagine that if the electric industry has identified several hundred use cases, then you know around the world there's got to be just tens of thousands.
Right.
We wouldn't be here having this talk on tape, so to speak, if there weren't, right?
I was just thinking about this as I was listening to you and you spoke to it with examples of smart glasses, workers in the field, analyzing things.
But are there ways that you've seen and whether you're using them now, or maybe things that you kind of see coming that you're excited about, ways that the energy industry has been using AI to and I don't know if it's like to design better battery storage or to explore new forms of energy, or to maybe something seemingly more mundane but still really important, like reorganizing the way that companies approach different industries?
I don't know what, but are there big examples that kind of jump out in you know your own work or what you've seen of how AI is transforming the industry from the inside?
Yeah, I mean, I think it's transforming it in all kinds of different ways.
And it's one of those things that I think has been really interesting, because things do seem to.
You know, there's lots of memes about how fast things are going.
And I already made some comments about 10xing and and things but but it's all sort of the proof is in the pudding have we seen where's that scaled demo i think there's a lot of proof of concepts that we're seeing uh pop up around uh and and really the the thing everybody is waiting for is that scaled demo of where there's this application and it's measurable and and and we've scaled it out to the entire enterprise So there's definitely a lot of work to do, but I think there's lots of applications as well.
Yeah, I'm trying to go through my head.
There's just so many different things because everything from understanding in the utility industry there's a lot of historical records and things.
And a lot of them predate sort of the digital era.
And so current models and things can make just ingesting all of that and structuring it into useful structured data sets that you can then use to create new models and create analysis and digital twins and all these things.
So I think there's some of the places the existing work is already really useful.
Right obviously, all the the, the things we do every day just to, to accelerate ourselves, you know, with understanding emails and, and you know, figuring out how to, how to, you know, have that hard conversation with the problematic coworker.
Um, and it's totally making, making these up as well.
No, no, no.
But it's, it's relatable.
It's that, well, it's that interesting sort of, there's two layers.
Well, there's many layers.
The five layer of cake is, is the, the iconic layer at the moment.
But there's kind of two layers when I'm thinking about it.
There's the layer of like the kinds of work that I don't want to call it knowledge work, but that kind of working with information you just described.
That is part and parcel of many roles in many industries, right.
And then there's kind of the and AI is helping.
You know helps me day to day in ways you were just describing or you know kind of making up, but I get you.
And then there's that layer on top which is specific to the kind of work and the industry that you're doing.
And the more people like you I get to have these conversations with, just the more in my mind I see.
Like you know, It's both right.
And one informs the other, being able to go back and ingest all that old data.
You know we've had a cardiologist or a radiologist on a while ago talking about how much hidden information there is in old analog film scans.
Oh, yeah.
That, you know, AI image analysis is able to extract now and it's useful, right?
And that kind of stuff is...
Did you see the guy with the Microfish repository?
It rings a bell, but I don't know that I did.
This was a few months ago now, which makes it ancient news.
But yeah, there was somebody that had access to this huge repository of Microfish.
I'm old enough those of us that are old enough on here will remember looking at it under the little magnifying contraptions in the library to see the news article from 1942.
But he had access to tons of this stuff and started using the models to ingest it all and just created a monster data set.
And it's so cool.
That's amazing.
I love stories like that.
Ben, as we get to kind of wrapping up here so I can let you go.
This is not to put you on the spot because, as you mentioned, these kinds of things are impossible.
It's always impossible to predict the future, but when things are moving as quickly as they are, it's harder, right?
If we look ahead to the next year or so, you know, loose time frame, what does success look like?
You know, with micro data centers and even more broadly, I guess that's what I'm thinking about putting you on the spot, both for the grid and for everyday users of AI powered services.
So great question.
So I'll start with the micro data center, you know, part since we're talking about it.
And I think you know, hopefully in a year or two years We've got a pile of these.
You know, micro inference data centers built out and we're monitoring and measuring them.
And that's helping educate us on what we need to know so that we can continue to build them out for all the wonderful things that the industry is going to create.
So I think you know, from the micro data center standpoint, you know that, I think, is what I hope what success looks like.
Yeah.
And then you know, I think just in general, you know, I have no idea that everything is so exciting.
It's, you know, you mentioned GTC at the beginning.
I learned something new, you know, from those types of conferences and stuff every year.
There's new things that come out, completely change things.
I mentioned agents, you know, which are just like weeks old, maybe a couple of months old that we've really sort of delved into that it's changing the landscape again so you know i don't know what it's going to look like but i'm hopeful and you know it's going to be exciting and just there's going to be compute needs you mentioned you know at the very beginning sort of the importance of power and stuff you know i think you know there's there's still going to be challenges to solve to make sure that we can provide all these awesome things to everybody um and really move society forward and everything.
So exciting times.
Excellent.
Yeah.
Well, I'm with you.
I'm rooting for you.
And I'm excited to see how it all unfolds.
Ben, for folks who would like to learn more about the work you're doing, about the work EPRI is doing, where's a good place for them to go online?
Websites, social media accounts?
Where should they start?
Yeah, absolutely.
So website, so you can go to epri.com, E-P-R-I.com.
It's our official website.
So lots of great information there.
Also very active on LinkedIn.
There's lots of, if you're interested into the latest news about exciting AI and data center updates, sort of in their adjacentness to the electric sector.
Lots of good stuff going over there on LinkedIn.
So those are probably the two places to find us.
Perfect.
Ben Souter.
Thank you again for joining the AI podcast and best of luck with everything you and everyone at EPRI is doing.
Appreciate it.
Great to be here.
Great talking with you.
