Thank you. Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Agentic AI is on the tips of everyone's tongues right now, it seems.
But what is it? What makes agentic AI so exciting?
And what should AI leaders like CIOs and IT execs be thinking about when designing an agentic AI system for an enterprise?
Here to break it down for us live from GTC 2025 is Bartley Richardson.
Bartley is Senior Director of Engineering and AI Infrastructure here at NVIDIA. where he leads a Gentech AI and cybersecurity AI.
Previously, Bartley was a technical lead on multiple DARPA projects.
He holds a PhD in computer science and engineering with a focus on AI.
And he's here right now. Bartley, thanks for taking time out of a busy GTC week to join the podcast.
Thanks, Noah. I really appreciate it. That's a great intro, by the way.
I'm going to bring you along with me everywhere I go, and you can do that intro every time.
Anything I can for the cause. Happy to do it.
So maybe we can start with, to put you on the spot, you don't have to define things, but talk about... agentic AI, what it is, why is it so exciting in this context of enterprise leaders? yeah you know i i feel like what we're really good at the in the tech industry and i'll include us at Nvidia and that and everybody, right, is we're really good at making things very complicated.
It's one of our primary concentrations, right?
And when I talk with people about agents and agentic AI, what I really want to say is automation.
That's the word I really want to use. But automation, you just get half of it out of your mouth and people fall asleep.
Right, so we call it Agentsic and Agentsic AI.
And really what it is, it is that next level of automation.
If you think about the evolution of automation and how we were doing things manually.
Even if you go back to factories and these types of things, these, I don't want to say mundane, but I obviously just said it, but these kind of like everyday repeatable tasks, right?
We've, come up with better ways to do it.
For example, I grew up on a farm in the middle of Ohio and we would dig post holes by ourselves with these terrible tools. we have technology that does this now for us right agents are very similar they're just instead of working in the dirt in the middle of ohio they're working on massive petabyte scale data silos or information repositories to take that data, turn on a little bit, do this like mundane task.
And then instead of returning data that the human has to go back through and look at, actually take that, give it context, synthesize it together with other types of data and make it a little bit more actionable, a little bit more easy for the human to digest.
Right. Can we talk about reasoning? Yeah.
Is there agentic reasoning? Is it a thing now?
It's a thing, right? Like, I think we're going to make it a thing, right?
You know, it's definitely a thing. reasoning is another one of these terms that gets, you know, kind of bandied about quite a bit, right?
And really what you have is this kind of different type of model, almost, where you have a I'll say a traditional LLM or an old school LLM, I don't know what that means, like two years ago, five years ago.
You have these old-school LLMs, and they're really good at this token prediction.
They can do the thing where... oh, okay, I have this sentence, complete the sentence, and all of these types of things.
They're going to image task and that. Reasoning models are kind of like this next level, and they've been trained and tuned in a very specific way to almost think out loud if you look at their text and look at their structure.
Where they're going through and you've given them the task, they're like, oh, I could do this, I could do this, I could do this, I could do this.
And it's kind of like when you're brainstorming with maybe your colleagues or even your family, right?
You're like, oh, we could do this, we could do this other thing.
Reasoning models have that type of feel to them, right?
And so putting them in a place of a system that's like, hey, I don't really know what I want to do, but I kind of want to make a plan.
Here's my loose guidelines. Yeah. Reasoning models can very easily explore that space for you and get it to like, here's some options.
Maybe we can dive into the technology a little bit and you can go through some of the components involved.
There's different NVIDIA technologies here.
And just talk a little bit about each one, what they are, what they do, rather, why they're important.
Yeah. And I think the way you set that up is really great because it is this collection of technologies, right?
When we talk about agentic system, it's not just like, I have this completely 100% brand new thing that we're trying to do.
Exactly. Right. Exactly. It's and you know, you tell people it takes years to be an overnight success.
And so it takes years to do what we're doing in agents and agentic systems right now.
And it starts with something like data ingest. and data ingest and being able to retrieve that information.
So if you look at things like Nemo Retriever, that's a great place to start. we have a capability that now Nemo Retriever doesn't just ingest text, like unstructured text. but we can ingest multimodal documents.
So everyone's favorite document type, the PDF.
Everyone loves it. Great document, easy to work with.
If you would have told me 10 years ago that I'd still be working in PDFs, right?
Like, I don't know, right? But anyway, here we are.
And it can have all of these different modalities in it.
It can have pictures. It can have structured images, which are like tables and graphs and charts.
It can have unstructured images and all of these types of things.
And extracting that information and keeping the context, I'll give you a really simple example where let's say you have a graph, a bar graph, a simple bar graph.
And if you were to just give that to a really kind of basic computer vision parsing model, it would give you back and be like, yep, I understand that I have five rectangles.
Okay. It's like, no, you don't. Because, right, just having five rectangles doesn't make that interesting.
So it's the relationship with the five rectangles to them.
It's the caption that goes underneath it.
Is there text somewhere else in this document?
It's not even on that page that explains that chart.
So you have all of these different, when we talk about the retriever and ingestion process, We have all these different models, these NIMs that go into the pipeline, and then we stitch them together to make this type of parsing possible.
So we have models. that detect bounding boxes of images.
We have models that will work specifically on text.
We have models that look at captioning and put all this together. three, four NIMs that just go into this one process.
And what I love about Retriever is and you're gonna see the theme, right?
It's built for enterprise quality. And so if you look at PDFs, if you're the average person, You have like a couple, you're doing research, something like that.
You have 10 PDFs, you upload them. You don't care how long it takes.
It takes five minutes, you know, whatever.
Come back and do it. If you're an enterprise, right?
Like, you know how many PDFs we generated in video?
Like, I think we could run the world on these PDFs that companies generate.
And you need speed, you need consistency, and you need accuracy, right?
And so if you look at something like Retriever, We're 15 times the amount of throughput that you can get in other leading systems with a really complicated PDF. we can go 10 pages per second through on a single GPU instance of this.
I extract this. It's really cool that 50 percent fewer errors and accuracy compared to other systems.
So it all starts if I think about agents, it all starts with that part, getting the data from human source into an embedded kind of source right right right uh should we talk about asian ops tools Yeah.
So I like the progression that you're going right here.
So I've got my data. I've got it in there.
We're pretty good at that. Agent op tools are another way of saying this is kind of like a flywheel.
Yeah. Right. So you've got your your system.
We'll talk a little more about the system a little bit.
And you might have your models. And what agent obstacles are really good at is fine tuning, honing and making that system even more efficient.
Okay. Right. So, you know, for example, you can, through successive iterations of fine-tuning through Nemo Asian Ops tools, you can have like a 10x reduction in your model size.
Right. By fine tuning that down, distilling it right a little bit, which obviously translates to either speed or translates to energy or translates to money.
Right. These are kind of currencies. you can exchange, right?
Depending on what you care about. Almost 4% increased accuracy by doing this fine-tuning model.
And the real cool thing about AgentOpsTool is is it just kind of sets there, right?
And you hook it in, you've got your inputs, you've got your outputs.
You hook it into your outputs, it feeds back into your inputs. and it'll prompt the human like every once in a while.
It's like, hey, what do you think about this?
And you know those, you go to any tool and you give it the thumbs up, you give it the thumbs down, right?
There's that, but what's even better is you can give it freeform text.
Not just, did I like this or did I not like this?
Oh, you can get feedback, yeah. Yeah, this wasn't quite like this, whatever.
And it keeps that context and we'll use that to like steer in a different direction, right?
Oh, very cool. And so it's useful in a lot of ways, model reduction size, all that.
But then you've ingested all this data, you're using this model.
It's a way to kind of push or steer this model a little bit towards your particular use or suite of models.
If I'm jumping ahead to something that you're going to cover, tell me and I'll back off.
But you mentioned accuracy a couple of times.
Accuracy, obviously, a big thing. Do the reasoning models themselves improve accuracy?
And I know it's not like there's one kind of reasoning model that fits everything.
But I'm just wondering, as you've mentioned it, like, is that part of what the reasoning model is meant to do?
Or is it just kind of a happy side effect or?
You know, it can in a system, right? And certainly if you look at reasoning models, there's, you know, if we talk about NVIDIA's model, right?
Okay. Mama Nematron reason super. I might have messed up.
Mama Nematron super reason. Nematron super with reasoning.
Super with reasoning. Yes, exactly. If you look at that model specifically, it does have a higher accuracy than these other reasoning models, right?
Like in its kind of system. Where I think or where we see this being advantageous is as part of a larger agentic system. where I've got this reasoning model and there are good tasks for reasoning models.
And I'm just going to say it. They're bad tasks for reasoning models, right?
Like just like there's good uses for a fork and there's really bad uses for a fork.
Right. Like it's the same for this reasoning model.
And with our reasoning model, with Lama Nivashram reason, one of the great things about it is you can turn reasoning on or off.
So you've got a single model that now operates as a reasoning model or it can operate as just a regular like non-reasoning model.
Right. And so when we talk about accuracy, one of the really nice things about reasoning models is the ability to iterate with them.
I'll give you an example. You're in a deep researcher type environment and you're like, hey, this is the topic I want.
I might upload a few PDFs, right? Like I give it the structure.
I want to have an intro, you know, a conclusion, right?
Keep it pretty light. You know, you give it all this thing.
A model can go, a reasoning model can go and say, okay, I'm going to really quickly generate a lot of ideas, a lot of questions that I want to do.
I'm going to present it to you, human. It's like, this is what I'm going to do.
This is like seven tasks or something like that.
Something the human can, is sticky for the human in their brain, right?
Like not a hundred things. And a human can go through and they can be like, oh, that one looks good.
Can you slightly, that one, I want you to retry. tweak it, right?
I want it about this topic or not that topic.
That one looks like it's gonna be too high level.
Can you take that more towards a 12th grade level?
Right. And it's really in my mind, it's the improved accuracy that we do see a little bit in the reasoning models. but it's the human model connection that lets you really, it becomes a coworker.
These agentic systems become the coworker. where you can guide them and you can fine tune them and you can talk with them and interact with them.
So you have the report that you want, and then you do all of the deep researching.
Now you're in minutes of like, I'm going to do this stuff. and what if i take us i'm gonna date myself if i take us way back uh if you remember the internet when we used to talk about things like uh load times and you know what i mean like what what's the what's the user's perceived load time of this and we would We would do things to do that.
We now have this ability with the reasoning model, you can interact with it early.
And so when you get to the final report or a draft report, you haven't spent five minutes and it's something you totally different want.
Didn't work. So not only is the actual measured accuracy up, but the perceived accuracy from the human, the perceived efficiency, because you were involved in the process early on.
Yeah, fantastic. I feel like that's a great segue into talking about enterprise, the enterprise industry, IT leaders, whoever's designing, developing the AI applications, what are some of the things that should be considered by a tech leader or CIO? when they're designing an agentic AI system in particular?
We have a couple hours, right? Yeah. I think the biggest one is that when I talk with anybody in the enterprise, I remind them, I'll be like, Okay, so what is the agentic system we should use?
And I'm like, I'm gonna answer your question, but let me ask you a parallel question.
I'm like, tell me what's the one piece of software you use in your IT?
And they're like, what do you mean one piece of software?
We have 15 different vendors and we do all this kind of stuff.
I'm like, correct. right you're going to get some from your vendors right uh whoever that is you're going to have an application you're going to have your crm you're going to have all these different things for your developers They're gonna put their own agents in that, right?
And you're gonna work with those. Some are going to be homegrown.
By you, right? Because everyone's moving really fast in this space.
You have your own enterprise data with your own sources.
If I can talk about a very specific NVIDIA example that I tell everyone.
Yeah. We have this thing called NVBugs Pro AI Search.
And I like it for two reasons. One, it uses AI agents.
And two, the name just rolls right off the tongue.
Totally. Right, at least when you say it.
In the Bugs Pro AI search, it's just like, that one sticks in my brain.
Not the order of super or reason, right?
But not that one. But the thing I like about this is our IT, our great IT department, who are much more engineers than the average IT department. created this, and they created it very early on in LaneChain's existence.
And then they modified it a little bit, and now we're here, fast forward six months.
And we want to not only use it with IndieBugs Pro, but we want to hook it up to coding repos and forums and CRM systems and all of that.
And what that means is that now you're in this similar situation where I might have various different vendors and stuff that I've grown myself.
So when I talk with them and say, what you're really looking for is you need to look at it in that same context.
It's not, I'm going to buy one piece of software.
I'm going to write one piece of software.
You're going to have all these agents working together.
And the trick is, how do you let them all come together, mesh together in a a somewhat seamless way for your employees.
So when I log into our systems or if you log into our systems, It's context aware.
It gives me the information that I need.
It helps me do my job. I look at it like that.
It's a Look at it in the traditional IT deployment sense.
Right, right. How does that differ from... what enterprise IT departments and developers were doing, you know, sort of pre-LLM, pre-agentic AI, that kind of thing.
Is it To me, being a little bit on the outside thinking about it, it sounds sort of like, oh, you've got different vendors and it's like an app store and CRM systems have their own apps.
Everything's got their own like you know, place where you can go grab apps.
Are we moving towards that for agents? You know, I mean, I think so.
Yeah, I mean, there definitely will be. I know they're out there already.
Yeah. Right. You can, everyone has their own and, and that's fine.
I think the difference is I say, you know, when we had all these apps in the before times, I guess.
When we had all these apps, we were in data silos.
Yeah. And now it's very, it could be possible.
We're in these agentic silos, right? Like a little bit where I have data here and I interact with that agent to talk with us.
The big difference though, is we're in a situation now where we're not always going to have API to API access, right?
I don't have to necessarily always have a developer script and code that.
I can have CRM agent over here talking to Confluence Wiki agent over here.
And they are communicating not via an API that a developer set up.
They're communicating via human language with each other.
Right. And that kind of makes connection, it kind of eases the connection points.
It raises some things you have to look at.
Right. Like and how are we monitoring these systems?
How are we observing these systems? Right.
But that is one of the key differences is you're going to not just have API to API to API specs.
You're gonna have some of that. And then you're gonna have these agents that are just communicating with themselves on your behalf.
Right. What kind of complexities does that raise or perhaps doesn't raise when it comes to data security, and obviously this all runs on data, like you said at the beginning, right?
Ingested data. So if you've got different agents and they're maybe from different vendors and they're doing... how do they talk to each other in a secure manner?
It's not 100 percent solved yet, but where it's headed is this idea that we call context-based security.
And so if you go back in the history and you look at, you know, security really started with, let's say, firewalls.
Right. And what was the motion? I'm going to put everything in this circle and I'm going to put my hands around it.
And then I secured it. And then we did this thing where we, I don't know if you remember it, we moved to the cloud. or I think we did, right?
Like we moved to the cloud and then it was like, oh, oh, I can't put my hands around it anymore.
Now I'm application-based security, right?
Like, cause I have pieces everywhere. And now we're in this motion where there's cloud, there's on-prem, we still have all of that.
But when I am accessing a system, like I said, versus maybe with like our CFO is accessing that same system, We are looking at it in a different context.
Right. And the information we want out of it.
So we're moving to this context based type of security. where you not only have to understand the person and the credentials and do all that stuff that you are already doing or supposedly doing, right?
Like our back should be there, right? Like if it's not there, we have a more But you have to look at the context in which the question is being asked.
Like look at the things around it, look at what pieces of information are coming with it. do pieces of security before that question is, and then do an analysis before you return the information to the user.
And you can look at that context. So it does, I don't want to say complicates it, but it does add about maybe 10% new stuff that we weren't doing before to roughly 90% of just like regular security applications i was speaking with bartley richardson bartley is the senior director of engineering and ai infrastructure at nvidia And we've been talking agents, agentic AI, a little bit of cybersecurity, got snuck in there.
But that's Bradley's domain. He leads agentic AI and cybersecurity AI. here at NVIDIA, and we're talking about agentic AI and the enterprise.
To shift gears a little bit and kind of talk about business needs, designing one of these systems Are there best practices or things that you've seen that work well when it comes to making sure that you're designing with the business care abouts in mind and then going back and kind of rechecking and keeping that that North Star.
Well, I think, yeah, there are. Some of it is kind of what we've already talked about, right?
Like, you know, building on... these strong foundations of not just tooling and ingest but you know models and so there's all of that right like obviously One thing that we haven't talked about explicitly yet is this notion of traceability and observability and profiling.
Right. You mentioned briefly of it. Yeah, and you have to imagine if we go back and we have this kind of distributed system, there's all these agents, they're connected with other things, they're talking in various modalities faster than a human can understand from different vendors and different even like agent tech framework providers, right?
Like you might have Someone in your business wrote this thing in lane chain, and someone wrote this thing in crew AI, and someone wrote this, there's all these different things.
So how do you have a holistic kind of traceability and observability platform like across that?
And that becomes a little challenging. And it's why we made this new thing.
It's called Agent IQ. And it's not an agentic framework.
There's plenty of those. I haven't checked.
We've been talking for a few minutes. There probably are 10 more that popped up while we were talking, right?
We don't need another one of those. Are frameworks the new prompt engineer?
Yes, they're the new, yeah, exactly. But it's not one of those, but rather it's a really simple, as much as necessary, as little as possible way to get all of these frameworks and tools and everything to work together. and be observable on the same point.
Being a CS person and coming from an engineering background, the way I tell people is I'm like, again, we're really good at complicating things.
We've got agents and tools and all these things.
I'm like, no, everything's a function call.
And so what agent IQ does is it it lets you use the frameworks that you were using and you still develop in those frameworks.
Like I was saying, you probably already have stuff.
Don't rewrite that code. Let's just hook it up to other things that you have, but let's do that in a way where we devolve everything to a function call.
And that lets you say, oh, I've got this agent pipeline here, and now I want to add a capability to it.
Well, I can add it on. Or I can wrap it in something else and I have an agent inside of an agent.
It allows us nesting kind of capability.
And what it gets you is this really cool traceability.
Yeah. So you can go into every tool call, every LLM call, every chain of tool calls.
You can look at the input tokens, the output tokens, the time it took to do the tool call, the sequence of actions of the tool call.
We have customers that are already seeing, they look at the timing charts. and they'll optimize their tool calling chains and they get the 15 X, right?
Like speed up through their through their pipeline for it.
Right. Or they'll get a, you know, a five X improvement in accuracy by moving things around.
And it's that type of information that is one of the reasons that something like Agent IQ exists.
Yeah, that's fantastic. We've got a few minutes left.
Is there an area that we haven't covered that you want to go into in particular?
Yeah, I mean, the thing I think I would end on, right, if we're talking about agents is, I feel like as an industry, we've been talking about them for a really long time.
But we're pretty new to them and their actual usability. and how we're kind of getting them into, like we said, enterprise scenarios.
I think what's What's incredibly interesting is these use cases that we have and some of them we have here in booth demos right at GTC this year.
Right. So I'm sure afterwards we'll have those right where people can consume as well.
But we have this one that is automating a lot of what happens in the kind of feature requests, like what issues are a customer having with my product? all the way through writing a PRD for that, a product requirements doc, all the way to assigning who might be the best engineers for doing this.
I think the coolest thing that I've seen as part of that is we use a reasoning model to say like, oh, here's some issues your customers are talking about in forums, or here's people... tickets they filed, right?
Here's a set of brainstorming questions that I reasoned through.
Go have a meeting. And then the humans get in the room, they have the meeting about, they talk about, they might draw some diagrams and that kind of stuff.
That goes back into the system. So think about this as a big human in the loop. that goes back into the system, what comes out the other side?
A fully formed PRD where we've taken the language and the action items from the actual We've taken pictures of the whiteboard, it produces the diagrams, and then it gives you a PRD that then, you know, if it gets you 75, 80% of the way there.
That's fantastic. That's great. Cause what's, you know, I'm sure, you know, you do your fair share of writing, right?
Like the hardest part for me about writing is that blank page.
Totally. And if I can get something that's 80% of the way there, it's great.
And that's the point I would leave on is agentic systems and models will make mistakes.
They will never be 100% accurate. I would challenge you to find anything that's 100% accurate.
But the way to think about them is like, look, the human will be in the loop.
And if it gets you 60, 70, 80% of the way there, that's amazing.
Yeah. I mean, that's 70% of work you don't have to do.
It's there. Exactly. And that's the part that I think is incredibly compelling and and we should focus on accuracy and we should always try to make things better.
But I never want to lose sight of like, look at the 70% that we did, and how are we gonna continue to make it better, of course.
Absolutely. All right, Bartley, before we let you go, a little change of pace, but in your own daily routine, your own daily work, Are there AI powered tools that you're using every day that, you know, you can recommend them.
It can just be a category specific tool.
What are you using that you like? Yeah, I'll preface this by saying I have no official affiliation with any of these.
Some of them I pay a lot for. But other than that, no official affiliation.
Of course, I think a lot of people, AI powered search engines, that's something like a perplexity.
I use that a lot. I use ChatGPT, the reasoning model in ChatGPT quite a bit.
That one's fantastic if I'm trying to research things or if I
I just have something that I wanted to think about just a little bit more.
Of course, that was before our reasoning model came out and before AIQ is out there, which is our deep researcher.
So those are kind of the obvious ones. I don't get to code.
My teams don't let me code as much as they used to, and they shouldn't.
That's smart. That's smart of them. But I love Cursor. as a coding tool and even just making diagrams, like it's really good at that.
The amount of times that I just hit tab and I'm like, wow, get out of my brain, right?
Like I just hit tab. The other one that I use a lot is called Napkin, napkin.ai.
And that's the one that I don't know if a lot of people have heard about.
I haven't heard of that one. Yeah, it's in beta right now.
It's free. Again, I have no affiliation with them other than I love their tool.
What you can do is you can either just like type freeform text or you can do like a perplexity style and it'll research things for you.
And it will take, let's say you have a process diagram or a complex tree or whatever, and it turns that into the associated diagram and the infographic for you.
And then what I love about it is it gives you kind of pre formatted options to start from.
Do you want this to be a funnel is this a process tree is it the cycle and all this in different color options you're like yeah i want that And then you can go in and you can change anything.
You can move the text, you can change the text, you can move the shapes and all that.
So I talk about the 70% there, right? I would toil away with these in PowerPoint for so long, right?
But now, and it's getting better, I can just describe it to napkin, like back of the napkin.
And it produces this great SVG or PNG diagram.
And so it's fantastic. Bartley, for folks listening who want to find out more about all the things you've been talking about, all the work you and NVIDIA is doing with the enterprise, Where's a good place to go online or places to get started?
Yeah. place to go is build.nvidia.com, right?
Like that's a great place to get started.
You can see all of our models, all of our blueprints, which is just kind of like intent examples.
So if you go to build.nvidia.com, AI.NVIDIA.COM takes you to a similar place, right?
If you're interested in this new thing that we were just talking about, which is called Agent IQ, We'll play on agentic, right?
It's agent IQ. Oh, yeah, okay. Yeah, it was clever.
But if you're interested in that, That is open source software on GitHub.
So github.com slash NVIDIA slash AgentIQ.
But we'll also be all linked off of build.nv.com.
Yeah, great. Bartley Richardson, thank you so much for taking the time out.
Best of luck this week. Have a great show.
And in all you're doing, I mean, we can catch up and do it again down the line and, you know, see if we're still talking about PDFs in a year or two.
Oh, my goodness. Yes, we'd love to. We'd love to. you Thank you.
