Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Before we get into today's episode, a quick note.
If you're interested in attending NVIDIA's GPU Technology Conference, GTC, this March in San Jose, California, and exploring the cutting edge technology shaping our future, we have a special offer.
Use the discount code AIpodcast when you register for the conference at nvidia.com slash GTC, and you'll receive 20% off your registration.
That code again is AI Podcast. Today, we're talking about the big stuff.
AI is reshaping, well, pretty much everything.
It's why we do this podcast. Climate change is also reshaping a lot of things.
It's defining our era as potentially impacting every living creature on our planet.
So today we're joined by Daniel Castro. Daniel is Vice President at the Information Technology and Innovation Foundation, ITIF, and Director of ITIF's Center for Data Innovation. and he's here to talk about the intersection of AI and climate change.
With his extensive background in IT security management and policy, Daniel has been a pivotal figure in discussions of technology's societal impacts. he'll discuss the findings of his AI Energy Use Report, focusing on the role of AI and GPU acceleration in enhancing energy efficiency and sustainability.
Daniel, welcome, and thanks so much for taking the time to join the AI podcast.
Noah, thanks for having me on. So could you start by telling us a little bit about the motivation behind the AI Energy Use Report and what you aim to uncover through your research?
Thanks for that question. I've seen, I think probably the same headlines most people have seen over the past year, which suggests that as we're seeing the deployment of AI, the increased use of AI, maybe we should be very worried about its energy consumption.
And so what's interesting to me is, we've heard this type of concern before in the digital space.
We heard this concern when Amazon was taking off and There were headlines about every time you buy a book, a lump of coal is being burned.
There are concerns about everyone streaming online and what that meant for the amount of energy that would be consumed.
And so in each of those cases in the past, it turned out the energy consumption figures early on were misleading. and they were ultimately wrong.
I wanted to know with AI, we know this is a fundamentally revolutionary transformational technology. what's actually happening.
And so that's what I set out to uncover.
And it's really a fascinating look at how conversations around energy use of digital technologies often get a little hazy when you don't follow the science.
And It's interesting how it shapes where policy debates end up.
So what did you find out? So what I found out was that there really were one or two early studies that estimated that future energy use from training AI systems could be enormous.
And that those early studies is what has shaped so much of what people think about AI's energy use today.
But it turns out those studies were wrong.
In particular, this first study that so many people tended to cite that came out in 2019, it estimated that training in AI model was the equivalent, had the equivalent carbon emissions of 300. round-trip flights from East Coast to West Coast.
And it turned out that that estimate was simply wrong.
And it was wrong for a number of reasons.
It was wrong... first of all, by a factor of about 80.
But it was wrong because they hadn't had the right numbers.
They had to make a lot of assumptions that turned out to be incorrect.
It was misleading because it was also kind of not the AI models that we're predominantly using.
So, you know, the study looked at three different AI systems.
One system very similar to the large language models we're using today that are very popular.
Another that is not at all like what we're using today.
And it was the one that is not at all like the ones we're using today that had this massive energy use.
Gotcha. And there were just a number of other factors as well.
And, you know, it turns out, of course, that what we've seen over time is that Yes, there are some AI models that use a lot of energy, but what we also see is that over time, there's continued investment in how do we make these more efficient, whether it's through improving the chips, improving the algorithms, you know, optimizing models in novel ways, or sometimes just moving to different types of, you know, algorithmic types of systems.
And so the what we see is that over time, performance improves and energy use either stays the same or sometimes even declines.
And again, the question, of course, is where are we going with this?
What will this end up looking like in a few years?
And what we don't know for sure what that looks like.
What we do know is that the early estimates are entirely misleading.
And so you said that early study you mentioned was from 2019, so roughly three to four years ago now.
I know as sort of an end user and observer of AI that we've seen tremendous acceleration in AI's capabilities, and as you alluded to, the use of LLMs and other generative AI technologies in particular have really exploded you know, in the mainstream, the public consciousness.
What's been happening on the energy consumption and energy conservation side?
Has it been the case where, as you mentioned, sometimes happens the performance has improved significantly? but the energy consumption has stayed the same or even gone down a little bit?
Or are both things sort of trending up over the past few years?
Yeah, well, there's definitely multiple factors at play here.
So the energy used to train a system. I mean, some of that has gone up as we're talking about training for weeks or months instead of days, right?
So certainly energy costs have gone up there.
But what we've also seen though, is that, you know, sometimes emissions have gone down or significantly lower for certain systems.
So for example, there's one system, Bloom, that was trained in France, primarily on energy that came from a nuclear facility.
Very different energy profile, carbon emissions profile than one that's trained on a coal-burning power plant.
And so these are the factors that I think are often left out of these conversations.
And so, for example, when you look at kind of where the trends are, well, the trends are that most of this, these large models are being trained in data centers run by some of the largest tech companies.
These are the same tech companies that have made incredibly bold commitments to clean energy use and to carbon offsets.
And so in terms of the net impact, it's actually pretty good in terms of where this is happening.
And so when you look at some of the reporting, the macro trend lines, for example, with a company like Google, Their energy use, it's gone up somewhat over time.
It continues to grow as they are a larger company that's grown over time.
But the amount of energy they're using for machine learning as a percentage of their total energy footprint it really hasn't changed significantly.
It's been in this 15 to 20% for the last three years where we have data.
That shows that even as machine learning is growing and it's eating the world, well, yes, but we're seeing optimization, of course, happen as well.
And why does that happen? Well, there's economic factors as well, right?
It can't grow so fast. And so big, because even the largest companies simply can't afford to do that.
So that's where we have to remember there are these balancing factors here.
This might be a slight tangent, but you mentioned the importance of where the energy is coming from and the difference between... training a large model using energy coming from nuclear power as opposed to coal-burning power, for instance.
I've noticed as a consumer, I've noticed over the past six months to a year, More and more on my phone in certain apps, I'll see something I think labeled a grid forecast. and kind of mentioning when the power is kind of more likely to be clean versus not clean and that kind of thing.
This feels sort of like a common sense question, but I want to ask it anyway, because you're here, you're the expert.
How big of a factor and how much consideration or how important is it that consideration is being given to... where the power's coming from and kind of shifting these things along with just those larger headline-grabbing numbers of the raw amount of energy being used to train a model.
Yeah, it's a huge factor and it's probably one of the most important things we can be thinking about. as we're talking about growing data centers and where we're training and also using the models, one of the things that I didn't know, again, before I set out and really dug into this, was how much energy use was on the inference side, not on the training side.
Based on the, again, many of the headlines, people say, oh, training the model is so big.
But really, you know, it's inference that's going to be anywhere between 60 to 80% of the energy consumption over time.
I mean, it's huge. And so, yeah, I mean, where you're training the models, but also where you're using the models has a huge impact and then the energy sources for that.
I think that's why as we think about where AI is going, there are some really interesting questions about how will it be deployed because deploying it on a battery powered endpoint device is very different than using it in a highly efficient data center.
And so there's lots of, I think, opportunities to figure out how we optimize different deployments of AI in ways that are serving the broad public interest in the best ways.
Right. GPUs, obviously, have been a huge factor in this AI explosion in recent times. and the current way that AI is used, to put it that way.
How has GPU acceleration transformed the energy efficiency of AI tech, and particularly if you could talk about the impact in weather and climate forecasting.
Yeah. Well, I mean, obviously we wouldn't have the current generation of AI systems without the advancements we've seen in GPUs, right?
It wouldn't even be here. What we've also seen is that when you look at some of these various AI models, different types of classifiers, For example, the efficiency significantly improves over time.
And that's generally because of two factors.
One is because of improvements in the hardware, and two, because of improvements in how they're optimizing these AI models based on the improvements in the hardware.
And so, you know, that's where, you know, when we look at over time, significant efficiency growth.
But there's also a question of. How are they going to use this technology to address the overall efficiency of the grid and overall efficiency of the data centers as well.
That's where, for the data centers, we've seen, I don't know that I'd say it's peaked, but it's definitely, We've seen some really solid numbers in terms of the power efficiency coming out of these data centers, in terms of cooling, in terms of everything.
They're highly optimized. In terms of the grid, that's where I think there's still a lot of opportunity to start using AI to increase grid resilience, stability. all of the factors that are going to be key towards turning this technology and using it to actually have an impact in people's daily lives.
Are there any specific case studies or examples you've come across talking about GPU technology, NVIDIA's GPU technology has really improved the energy efficiency of data centers or other AI operations?
What I was looking at in this paper was really, what's the net environmental impact?
Right. And so when we're talking about energy consumption, we're talking about energy consumption because we're concerned about, you know, obviously the carbon emissions there.
And so we're also concerned, well, what's what's all this AI being used for?
How are these uses going to have a potentially positive or negative impact in different areas.
One thing that we looked at is the substitution effect.
Are we using AI for activities that would otherwise be generating more carbon emissions.
You think about the classic things that people are using Cache-EBT for.
They're using it for writing text, and they're using systems like DALI to produce images.
So the comparison, You're not going to get rid of a human.
You might substitute a human that would be working in that, but the human's still there.
Human's still kind of living and breathing.
So the human's still doing the carbon emissions.
But you can say, well, if a human was actually completing this task, what other energy would they be using?
So if I'm writing a one-page document, I'm going to be spending an hour at my desktop or laptop typing at the keyboard.
We know what the energy profile is for that computer or we know what the energy profile is for using Adobe Illustrator and drawing something there.
So that's where we can see, okay, that's a factor of anything from 30 to 80 more carbon emissions by having a human do this manually on a computer versus having AI generate it in a couple of seconds.
Sorry, just to be clear, that the human is generating 30 to 80 times more emissions than the AI system would.
Exactly. Yeah, humans generating 38 times more.
So using AI in this way, positive substitute.
But then there's also the bigger question of, okay, well, what are all the industrial uses, right?
What are all the ways that companies are using this technology to, you know, just to optimize their operations, right?
And that's where, yeah, I mean, there's these big opportunities, for example, in California, the government figured out, they have all these fire watch stations and you used to have to have someone out there looking to see where is forest fires, where are they starting and how can we intervene sooner?
Well, using cameras and using real-time surveillance of this, you can detect it much faster and obviously you have full 24-7 monitoring.
So that's the type of opportunity. Well, that's all carbon that would be burned up and released. where there's a huge positive potential impact.
I think we see many scenarios like that, whether you're talking about optimizing transportation fleets and traffic optimization.
As I mentioned, grid efficiency, government operations.
There's a lot of opportunities like that to start using AI in ways that will pay a positive dividend in terms of carbon emissions.
And obviously, if we want to address climate change, we need to start doing more of that Sure, absolutely.
This may be too kind of abstract or I'm trying to ask too much with a single question.
So redirect me if that's the case. But when we're talking about AI and on both sides, training and data centers and everything, and then obviously inference, and comparing an AI doing a task to a human doing a task and that kind of thing.
And then we're talking about sustainability. which is kind of a big word and maybe in the public consciousness, a little bit of a buzzword that you kind of understand oh, solar power and not using fossil fuels and that kind of thing.
But actually, I don't really know what it means at the detailed level.
How do you think about sustainability and AI?
And, you know, is it a matter of kind of these barriers sort of base computations you're talking about with comparing the carbon emissions of an AI system versus a human doing the same task. are you able to factor in things like, well, if we use AI to help us figure out how to make better renewable energy sources and better battery storage technology or whatever things go into sustainable renewable energy. like can you factor in ai's you know impact and accelerating these advancements like like How do you kind of make sense of all of that in the context of sustainability?
Yeah, I mean, the simple answer is it's incredibly complex.
So we can... You can't draw a straight line from one thing to the other.
We know, for example, that when you make things easier, you do more of it.
When you make things cheaper, you do more of it, right?
So AI makes certain things easier. We're going to use it more.
That's not necessarily though a bad thing.
The question is, what's the net impact? I think in all of these cases, What we want to see is, are we putting the foot on the accelerator towards using AI for sustainability?
Are we using more of those smart thermostats?
Are we using AI more for integrating distributed energy sources?
I was just seeing something recently about recycling and how hard it is to get the human workers that need to go through these recycling centers and sort all these things. different things.
And that's a great example where AI is really good at it, right?
Using computer vision, it can identify which thing needs to go in which area and using robotics, you know, process this much faster.
So that's where I think we need to be kind of leaning then to how AI can be part of the solution to sustainability.
And then at the higher level, we always will have this question about As we are using data centers, as we're building more data centers, how do we make sure we're doing that? the most efficiently and using clean energy sources.
And again, that's where I say, I'm glad it's, you know big tech that's leading in this space because they're the ones that are making these bold commitments to you know by 2030 by 2050 you know not just carbon zero Some of them are lifetime carbon zero.
So they've been going back and retroactively erasing their carbon footprint.
I'm highly optimistic that they've made these commitments, they're going to stick with them even as AI increases.
I think the real question is, so many other sectors of the economy start using all this technology.
Let's make sure they are also making these commitments and they're using it in ways that have this positive impact.
I feel like I'd be remiss not to mention that, Dan, you mentioned both the wildfire prevention and recycling as two areas where AI is having a positive impact.
And if you're listening and you're interested, We've done podcasts on both of those subjects, so I encourage listeners to check out the archives.
They're both great episodes. and you can learn more about how AI is helping on both of these fronts.
But right now, we're talking to Daniel Castro.
Daniel is Vice President at the Information Technology and Innovation Foundation, and he's also the Director of ITIF Center for Data Innovation.
So you mentioned the role of big tech and taking the lead and committing. to renewable energy and reducing carbon emissions and carbon offsets and all of these good things.
But sort of on a... broader level or maybe kind of shifting from industry to sort of the government level What policies or frameworks do you think might be necessary to encourage the development of AI-efficient tech?
Is it, you know, obviously the private sector and the public sector don't exist in their own bubbles.
There's all kinds of work together that happens and battling over things and cooperation and all that good stuff.
But do you see anything emerging as sort of the right way to approach policies and frameworks to address energy efficiency?
Absolutely. I mean, there's a lot of talk right now about how do we regulate AI?
Should we regulate AI? What's the best way to do that.
And one thing that- If you have ideas on that, you know, go ahead.
Many ideas there, but on the energy side specifically, I mean, one of the calls is can we have more transparency in models?
You know, can we know things about the models, whether it's the type of data they've trained on to how they should be used, questions about bias, all of these things.
One thing we can also do is have energy transparency standards.
Simply put, you know, This is the estimated amount of carbon emissions in training this model, and this is the estimated amount of carbon emissions in using the ai model for a you know kind of um given uh scenario uh so that is comparable right so that when different potential users are making their decision about which AI model they want to apply for a given problem. they can consider more factors than just accuracy and speed.
They can also take in consideration energy profiles.
Making me think of those stickers when you buy like a hot water heater or something and you've got the, you know, estimated cost and estimated electricity use for a year, that kind of thing.
I think a lot of companies care about it at the end of the day because they care about, one, some of their net zero costs.
And if they're paying for offsets, they're going to care about that way.
They care about their energy costs. And it's just something that I think a lot of responsible businesses are paying attention to.
The second thing I'd say is this doesn't have to be new laws and regulations.
Some of this can be voluntary agreements between government and the private sector.
The White House recently got a number of companies that were doing... that produced foundation models to agree to certain... red teaming privacy standards.
It can get them to agree to certain energy, again, transparency around here.
One of the things that I think we have to remember, though, is that some of the other types of policies and requirements the government has thought about, government regulators have thought about imposing on AI systems, whether it be you know, privacy or, you know, certain kind of safety requirements, all of those have energy costs too.
And they aren't thinking about them. And so we have to make sure that we don't get in a world where the policies that are coming down actually have a high energy cost that nobody's thought about.
And that's where companies are kind of forced into using, doubling their energy costs. when maybe there was a better way to optimize that.
Yeah, one step forward, two steps back. What about the public's role in all of this?
When you mentioned recycling, it made me think about...
I live in California now. I used to live and work in New York.
I remember back in my first days of working in an office back in the late 1990s, hearing, and it was a rumor, but this horror story that I was like, well, which recycling bin do I put the Xerox paper in?
And somebody said, It doesn't matter. None of it gets recycled.
They just throw it all out at night anyway.
And I say that just to say that your example of recycling facilities and how it's much more complicated and difficult than people might think about to actually go through and sort the different types of recycling into the proper places.
If we're talking about energy consumption and using AI and to your point, I do a lot of writing and I've found generative AI tools to be super helpful in my writing process.
But I'm not thinking about... the energy costs and the carbon emissions of doing these things.
What should the public's role be in this?
And maybe more importantly, what kinds of initiatives might be important for educating the public and forming a public industry partnership, if you will, when it comes to the potential of AI for energy efficiency alongside of productivity?
Well, I think in many of these cases, it always comes back to costs.
Consumers are motivated by costs and they're responsive to costs.
And you want to make sure that the energy costs are not an externality. in any of these decisions.
The reason somebody isn't gonna leave their computer generating thousands upon thousands of new images is going to be because of the cost, not because they're kind of turning the lights off. to be responsible, it's because they have to pay for it and they should have to pay for it.
That should drive efficiency and responsibility.
And I think the only area where we've ever seen digital technology and the energy use kind of deviate from maybe a responsible or desirable outcome has been in cryptocurrencies.
And that was driven by speculation, right?
So it was rational actors kind of acting irrationally because of speculation. we're not going to have that same thing in AI because there's no speculation around the randomly generated text or randomly generated images.
People are going to be motivated by the economic factors.
And I think here is where part of digital literacy is going to need to include AI literacy.
We have to have people using the technology responsibly, understanding how it works, understanding how everything's interconnected, but also hopefully being optimistic and willing to use this technology when it does have a positive impact.
I ideally want to see more people saying, why don't I have a smart thermostat?
Why don't I have... Why aren't I using AI to improve navigation and reduce the amount of traffic on the roads?
We want to see... people embracing this technology.
So I'd say to the extent that education can also help just lower those barriers, the resistance to adopting new technology. that's probably going to do the most towards having a positive long-term impact, AI's overall impact on our climate future.
So you kind of touched on this just now, and so in some ways this questions, how do you envision the future in terms of the interplay between advancing AI technology, energy sustainability, and climate change.
So I think AI is going to be a core part of how we address climate change and how we build a more sustainable future.
AI is central to integrating distributed energy technologies.
It's central towards... using technologies like autonomous vehicles.
And it's key towards just having more digitalization of the economy in ways that we start substituting bits for atoms, right?
And that can be much more efficient. One of the unrecognized opportunities I think with AI is how it's going to improve the quality of so many of the digital services we use right now.
So you think about something like video conferencing, which works really well. but it is high bandwidth and you want to have low latency.
That doesn't work across always for everyone.
With AI, of course, you can reduce the jitter or the lagging feeds.
By using AI to substitute for that, using computing instead of having to transmit as much.
So those are the types of improvements that when you have real-time video that works flawlessly every single time.
Well, that means you can take fewer flights, you can have more video meetings because the quality has been improved.
And I think that's, again, we just don't recognize that, yeah, that's going to be AI that gets us to that you know, level of quality that people say, okay, I can substitute a face-to-face meeting for a video meeting.
Those are the types of changes that I'm looking for. in the future where AI is what's going to get us there.
Is it more of a sum of many incremental gains like being able to even have a shorter video conference because the AI helps optimize the stream.
We don't have to pause every couple of minutes because of lags and that kind of thing, as opposed to the kinds of things that I like to fantasize about.
AI has discovered a novel energy source that will, in one fell swoop, revolutionize the whole world.
Is it based on your expertise? Are we looking kind of more towards the former?
I think it's a snowball effect. You know, this can get bigger and bigger and grow over time.
We are going to have, you know, the, breakthroughs the scientific breakthroughs where ai is you know discovering new compounds that can be used in batteries and that's going to unlock some new things But I don't think it's going to be the kind of flipping the switch and suddenly we're in this new perfectly sustainable world.
Yeah. I think AI is going to help us get there much faster if we work with the technology.
And so as we've been talking, we haven't said this out loud, if you will, but this has been kind of a United States-focused conversation relative to thinking about you know, some of the big tech companies being headquartered in the US and a lot of the innovations we're talking about, not entirely, but being kind of US centric.
But clearly, technology and AI aren't US only, but climate change, more importantly, is a global issue. issue and it's something that the whole world really needs to cooperate on if we're going to make progress.
I wouldn't imagine that's no easy feat.
And when we're talking about, you know, tech companies leading the way and public-private cooperation and that kind of thing, From your experience and your view, how does that translate from thinking about the United States out to the rest of the world?
Well, we know when we solve climate change, it can't be something that any country solves on its own.
It's something that we're all working together on.
And when we think about AI, though, we recognize, you have to recognize that The United States is competing with many other parts of the world, in particular, is competing with China.
Right now, we have restrictions on exports of chips and semiconductors to China, And that's going to have an impact in terms of what China is able to do with its own energy efficiency of AI.
It's not hitting pause on its AI ambitions.
So we have to think about there are trade-offs as we try to handicap China in this space. by limiting the chips it has access to, well, that's going to increase its energy profile and its energy use as it pursues AI.
Again, there's multiple goals here, but as we're thinking about where does this go?
Where does AI go? How does it relate to sustainability?
At the end of the day, we need all countries that are at the forefront of AI innovation. to be embracing energy efficiency, to be embracing clean energy and trying to address climate change.
And in part, that has to be through having access to the most efficient chips on the market for doing AI models.
Well said. Daniel, for listeners who want to find out more about the paper, just more broadly about the role of... of energy efficiency and AI development and AI use and how all of these things work together.
Where would you direct people to go online to learn more?
Yeah, please visit our website, datainnovation.org.
We have a weekly newsletter. And of course, we're very active on all the social media channels.
Excellent. Well, again, Daniel, thanks so much for taking the time to come on and educate me, educate all of us. about this.
It's almost like you can't talk about any one part of this without talking about all of it. which seems like it's kind of the point, because if we're going to solve climate change while continuing to enjoy the fruits of tech like AI, it's got to be a global effort together.
Appreciate you having me on. Thank you.
