Hey, Ted Talks Daily listeners, I'm Elise Hugh.
Today we have an episode of another podcast from The Ted Audio Collective handpicked by
us for you.
If you've ever played a video game, you've probably encountered an NPC, a non-playable
character.
You might have even tried to talk to them, but the conversation was probably limited
to a few clicks and a pre-written script until now.
This week we're sharing an episode of The Ted AI Show and it's all about what happens
when AI allows NPCs to actually engage in conversation with users and what that means
for the future of relationships.
If you want to hear more AI conversations, you're in luck.
There are two upcoming Ted AI conferences this October in San Francisco and Vienna.
See our show notes for more info.
And you can also find brand new episodes of The Ted AI Show wherever you get your podcasts.
Now onto the episode right after a quick break.
Like Ted Talks?
You should check out The Ted Radio Hour with NPR.
Stay tuned after this talk to hear a sneak peek of this week's episode.
It's the 26th century and you're moving through a world as a cybernetically enhanced super
soldier on the planet of Reach.
You have a big mission in front of you.
As Master Chief, you are the last hope for humanity against a hostile alliance of alien
forces.
Only you can stop the Covenant.
Well, only you with the help of your trusty AI sidekick.
A hologram with a cropped haircut and shimmering ocean blue skin.
Cortana.
Actually, it was 2001 and I just hooked up my Xbox to play the first series of Halo
Combat Evolved.
I was 11 at the time and my brother, our friends, and I would pile into our Punjabi living room
to play a campaign in split screen mode.
I was totally sucked into the game, invested in the story, and Cortana was key to that
with her humor and emotional depth.
Even though she's just a blue hologram, ironically she was the key to the game's humanity.
She was my trusty co-pilot guiding me through these alien worlds, and throughout the game
I felt a real bond starting to emerge.
It seemed totally novel at the time, but I was forming a friendship with a non-player
character.
I'm Belavel Sidhu, and this is The TED AI Show, where we figure out how to live and
thrive in a world where AI is changing everything.
Now non-player characters, or NPCs for short, have always been a big part of video games.
In single-player text-based computer games, players could interact with NPCs in a very
limited way, just putting in a very specific command and having the NPC respond from a
predetermined script.
Think the original King's Quest, where players typed directly into the text box to interact
with NPCs, who were there only to move the plot along.
This evolved into single-player games like Halo, where NPCs like Cortana had more dynamic
personalities and quirks written for them.
Though we're still limited by the design of the game, and therefore static, now if
you think about multiplayer games, they allow for human-to-human interactions.
Think Second Life, where you're interacting with other players through their avatars.
What's so immersive about that is you're engaging with the real people who are responding
dynamically to you, and you're responding dynamically to them, human-to-human.
NPCs exist in this world, and while they can be interesting, you would never confuse these
NPCs as fellow gamers.
NPCs just haven't been real enough to be mistaken for actual people.
Up until now, these NPCs have been relatively passive.
But now, with generative AI, those NPCs no longer have to choose between a limited number
of responses.
They can script in real time, just like a human player would, reacting to what's
happening inside of the game world.
Suddenly video game worlds can be infinitely more immersive and interactive.
As these virtual characters become more integrated into our daily lives, and maybe even become
our friends, are we going to start spending all our times in these virtual worlds?
Will our interactions with NPCs start to become unrecognizable from our interactions with
humans?
And what do we gain or lose if this happens?
This is the domain of Convey, spelled Con-V-A-I, a platform that enables developers to create
NPCs with human-like conversational abilities.
Convey says their goal is to help developers create virtual characters that can converse
in the present moments and build long-term human relationships with human players.
There are over a thousand projects being actively built on their platform, by creators ranging
from AAA studios to indie developers.
And what makes Convey so interesting is that they're making technology that enables game
developers to create NPCs that are so lifelike it won't matter who or what is on the other
end.
Pranendu Mukherjee is the CEO of Convey, and our guest on the show today.
He has a lot to say about our evolving relationship with NPCs, or as he likes to call it, AI characters.
And like me, he was a total gaming nerd.
Pranendu, welcome.
I'd love to start talking about the origins of Convey.
I get that you were a gamer before you worked at Nvidia, but obviously there's a lot of
areas in game development you could have branched into.
Why were you drawn to the notion of AI NPCs specifically?
Before I started at Nvidia, when I was doing my thesis work, I literally saw this language
model wave coming.
I wrote this, that while language models are going to get bigger and better, and will potentially
even have abilities to have human-like conversation, it is still not going to have the same level
of understanding as we humans do.
Because we humans don't think from text in, text out, or text to text.
We think from a 3D world around us, right?
Since we were born, we first understand locomotion, like moving around the 3D world, and then
we attach words to these objects, we assign meaning, right?
So basically, we are multimodal creatures.
Where do we find such a multimodal environment where we could potentially have these AIs
live, train, and iterate themselves?
It's virtual worlds.
And what kind of virtual worlds would we have people that can provide feedback to this AI,
heavily populated worlds, are games, right?
So all those connected together, like if we have to create this human-like mind within a virtual
world, NPCs, non-player characters, or let's call it AI characters, embodied in a way,
are one of the best vehicles to do that.
It's almost like we've got these rich environments where you can embody this sort of AI agent
and have it experience very similar stimuli to what we might experience in the real world,
which is a perfect segue into the evolution of AI NPCs, right?
This is a non-playable character.
This is sort of set-dressing, the side thing in this, like, you know, kind of like the side dish
to go with the main course, which is the game itself.
And so I'm kind of curious, what's your historical perspective on how AI NPCs have evolved
from their earliest stages to the complex entities that we see today in games and virtual experiences?
To talk about the history of games, I mean, there are these pioneering, genre-defining games
all the way from Half-Life that, you know, like, single-handedly define the first-person
shooter genre and, of course, Counter-Strike, like, the multiplayer aspect of it, that you
could have many people in the same world, right, in various ways of gameplay, basically
revolutionized gaming.
You know, they could play with each other, so you don't need new gameplay as long as
people can evolve with each other.
So, like, that has evolved.
NPCs has, of course, gotten better, mostly on the visuals or, you know, animation front,
but not on the intelligence front as much.
So what we are seeing, it's almost like a Cambrian explosion of characters and AI agents
that can not only be very human-like in terms of interaction with the players, just like
players played with players.
Now, AI can also play with players, both as friends or enemies, you know, cooperatively
or comparatively.
I think it's this magical moment where now we've, of course, got these behemoth large
language models, but you also have the, you know, kind of multimodal models hitting the
scene where it's not just that they can understand text.
These models can understand audio, can understand imagery, can understand even video.
Right. And so I'm curious to dig into how that affects human-AI relationships.
So how do you see AI NPCs sort of changing the nature of player engagement and emotional
investment in both games and experiences?
Firstly, the way these NPCs are becoming very human-like, there is going to be a large set
of people in the world that will big time benefit from it, mainly because there is a
big, big chunk of players who don't like engaging with real people or are nervous or afraid
to do that. They feel much comfortable if they know it's not a real person, and that will
help them open up.
They'll help them socialize in terms of people that, let's say, are playing single player
games or multiplayer games.
Now they can engage with this with the set of AI characters and have a more engaging
time, ideally. And lastly, let's say if it's in a multiplayer environment, people will
still enjoy engaging with people.
But now they have another reason to have fun together with other people.
And then from a relationship standpoint, basically, I think it is important for for
companies like ourselves to to look ahead in terms of the positives as well as the
dangers. It is going to quickly fill in the gaps where a human doesn't exist today.
Right, whether it's, you know, like just being friends or from a romantic angle or or
maybe someone that is a mentor or a guide and not just like chat GPT like text in text
out, but very much gamified, immersive environment that can that can reach them and they
can they can effectively have this mentor of an of an AI.
So I think overall, I definitely am an optimist and I see the positive sides.
There could be potential darker, you know, size dystopian sides that needs to be
addressed and understood and informed.
I mean, I think it's it's very fascinating.
The like it's one thing to talk to your chat GPT app, right.
And you see a voice emanating from your phone.
It's another thing entirely to, let's say, be talking to your mentor, you know, and it's
embodied as like a humanoid character that like has the same sort of expressiveness that
you do. It suddenly becomes this sort of more lifelike experience.
Right. And so as NPCs become more lifelike, what are those ethical considerations that
come into play, especially regarding player relationships and, you know, AI
behaviors? The number one thing that I think AI needs
and this is a bit controversial, but like if you if you think deeply enough, right, the
biggest fear for AI is centralization and a few entities responsible for these
relationships. When a kid grows up with an AI teacher and mentor and that's their all,
you know, that's a level of relationship that no company in the earth should own.
It's theirs. Absolutely.
Wherever they want to take it.
And what can enable this?
I think decentralized blockchain technology can provide true ownership.
That is going to be very, very essential along with confidential computing that can help
ensure that their data remains theirs, their memories and relationships remains theirs.
Yeah. I mean, you brought up a bunch of very interesting points, right?
It's like if you do have this future where, let's say, we have an oligopoly of companies
that sort of own the models that mediate your relationship with these digital
characters, right? And especially if this is like kids who are like growing up talking to
these NPCs or AI agents, whatever you want to call them.
Yeah, you are building like a very rich history of sort of their hopes, wishes,
anxieties, worries, desires and how those evolve over time.
And these agents are getting to a place where it's not just like, oh, yes, you say this and I will
say this. It's like they remember the context and glean insights from your previous conversations.
And maybe the right way to solve that is with decentralized AI.
And you alluded to confidential computing as well as like, can we keep this sort of very rich data
as close to the user as possible? And even if you are going to learn
and improve models off of it, you do it in this like privacy preserving way,
which, you know, itself sounds like a tall order.
I'm really keen to dig into sort of the experiential and sort of emotional impacts
of these type of AI agents, right? And so like one example that I keep going back to is, hey,
even if you're playing a solo single player game, you could have this like AI Jarvis or Cortana,
that sort of understands you, your context and is with you helping you navigate
this sort of game world. How close are we to having that sort of companion in games?
The primary aspect that is missing would be the, I mean, to some extent, right? So
is the multimodal aspect of it for that to happen at scale, like very much similar to Jarvis of
Iron Man is aware of the entire context. That means every new containing of the room that you
are in, you know, like they are able to see that process that along with your digital presence.
So we are not far away, like it may not be at 100% of Iron Man's Jarvis capability, but like
if you play the game, it'll feel like that. How does all of this change the way these experiences
are authored, right? The analogy that keeps coming to my mind is sort of the narrative
division of Westworld. Is that the right way to think about how you're authoring
these type of experiences with these rich characters? The emergent nature of large language
models are very interesting. The narrative designers and writers are still the better
storytellers, right? We have to have a right mix of both controlled evolution of characters,
control evolutions of stories, as well as the open-ended and emergent behaviors of these
generative AI models, which is where the balance and challenge is. And we are providing all the
necessary tools for developers and designers to do that. To come back to the Westworld analogy
that not only did Westworld had these crafted characters that were one of some of our
favorites, they evolved, right? They evolved into something else entirely than what was originally
written, right? And evolved in a very meaningful manner. Remember what was in Westworld that
started the whole thing about these characters evolving is memories. Once you have that,
their personalities evolve, they remember things. And not only are, it's enough to just give them
long-term memory. You have to keep them up all the time, go about their day, make decisions,
interact with other real people or other AI. And those interactions will change their decisions
and their pathways. And some will be very high intensity experiences. And that will shape their
personalities. And this will be ideally best put in servers that can have up to like 250, 500 people.
These AI NPCs will always be there living their life. Their experiences will shape them. And they
will start making decisions that will be quite bewildering. We already see that. We did this
Nvidia demo where we had now two characters and both of them were chatting. And we just let them
talk to each other, right? Just to see what they chat about. And while chatting, Nova made an order
that, hey, can you bring me a drink? And Jin was like, sure, let me get that for you. And he went
ahead and brought a drink and gave it to her. And it was so wild that they are not only just
talking with each other, they're like carrying out actions and they're giving commands to each other.
The demo that you're referring to, which went stupid viral on the internet,
I think it got everyone excited to just imagine where games are going next, right? And so I'd love
for you to talk a little bit about the types of experiences and characters people are building
with convey that are exciting to you both for gaming and non-gaming purposes. The non-gaming
examples are primarily in learning and training education related areas. And then there's brand
ambassadors, which could be in the likeness or digital human of a celebrity or a nondescript
model who is AI powered and knows all about the brand can guide people how to use the product and
whatnot. These AI characters can be even location aware, like you scan a particular QR code anywhere
while walking the street. And they spun up right there and tell you what the directions are.
It's not a far future, okay, where we start seeing these embodied AI characters literally everywhere
and very much in public spaces, all the way from take your favorite mall to your favorite airport,
you will see large screens with these AI characters standing there, welcoming you,
but now you can talk to them and ask them which way to go. You know, like, this is my airport ticket,
which way do I go? Where is the security check? Any kind of information dispense or transactional
engagement, these characters would be perfect for real world use cases that we are already seeing.
Now finally coming to gaming. Well, games are going to be effectively the matrix, right? So
what do I mean by that? It's going to be so real that you would prefer living there. All right,
so which is which is a dystopian future that we have to be aware it will be pretty darn engaging,
where instead of the machines taking over, we will be willingly submitting ourselves to this game
worlds, which will be an extremely exciting with all of these technologies converging all the way
from your VR devices, VR augmented reality and all of that to extremely high speed internet,
the cloud computing aspect of it where these extremely high definition world can be
literally rendered on the cloud and stream to yourself, along with these AI NPCs in these
metaverse like worlds, it will become a much easier way for people to put themselves in,
let's call it the matrix or the metaverse. All right, so where you can be there, live there
almost to engage with your friends and, you know, learn certain things and engage with this AI NPCs
and that is a future not too far away that people will start doing that. People already do that,
by the way, in a major way, in a lot of the social virtual worlds. Okay, if you think,
especially the younger generation, right? Yeah, growing up almost in these worlds.
That is true. But also, there is a very small minority who have been doing it for a while.
There's a large audience, a concurrent daily active user base for something like Second Life,
where we recently launched, right? And these people come back daily, live their life,
talk to their friends for many years now. And there are newer platforms like VRChat,
you know, like I know people that regularly go there, they party on the weekends.
Once there are challenges of onboarding and the challenges of the friction is reduced
to get in those worlds, a lot of people will start going there. All right.
Let's take that VRChat example. I think that one's fascinating. It's like,
yeah, people are buying expensive, full body tracking setups with expensive headsets and
computers to have this high fidelity embodied experience in virtual worlds. But right now,
on the other end, are other humans that they meet, right? How do you feel about,
you know, there being a time, let's say in a couple years, where you're talking to somebody
in VRChat and you literally cannot tell if they're human or not? Does it matter at that point?
Because like, I don't know, maybe these AI agents will be far more thoughtful and nice to you than
perhaps a real flaw to humans. So I'm curious how you think about that, especially in the context
of this Matrix analogy that you're making. You literally reminded me of this movie called
Transcendence. And Johnny Depp was the lead character that literally, that's the case there.
Basically, Johnny Depp dies and before dying, he actually uploads himself like full, full
neuron scan of his brain and uploads himself into the internet. And, and when he comes back,
after his actual biological body has passed back, people would often ask him, are you real? Are you
aware? And his answer would be, if you cannot tell, does it matter? So that is going to happen,
no doubt, is already there from a text standpoint, texting text out, it will be hard to tell. And
today, right? And the other other aspects would be the visuals of it, the animations of it,
and things like that. But also, what might be a giveaway is, if they are always of a particular
certain certain personality type, they need to have a wide array of personalities, and even
eccentric AI characters, you know, that are kind of awkward, and, and some of them are mean,
and some of them are super nice. And, you know, like, you get the full high school experience.
Yes, exactly. Those are going to be necessary for for for people to engage, it cannot just be like
very assistant like, so the more the variability, the better, you know, like people would like and
engage with them, and people will find their own type. Some people are drawn and attracted to toxic
people. Right. So so basically, that they will have all kinds of AI, basically, in these worlds,
and you'll you'll choose your pick what you want. It reminds me of the conversation with the
architect and the matrix where the architect outlines that, hey, you know, like we made a
utopian simulation, but nobody bought it, it just felt too artificial, and almost introducing the
flaws of our humanity, kind of is what made it a sticky experience. Yeah, which, which I think is
very fascinating, right? Because these environments and experiences need to mirror the full range of
emotions that we experience. Right, right. So that's what we are noticing, you know, like,
we did this demo room, and people enjoy talking with the meanest character.
Wow, like they would want to walk away and that that mean character will say something provocative
that would draw them back to talking to that character. That's something that we have to be
conscious about as well. Like, that is literally the one of the reasons that Facebook and TikTok
became so popular because the newsfeed was programmed for maximum emotional turbulence,
like the content that, that was the most provocative,
drawn people there, we don't want to do with our stuff. So, so like the right balance between
engagement and what's good for the people is something that we plan to do. Now, there are
many negatives to outweigh the positives here. But Pernento has spoken with me about being a
lonely kid growing up. And I had to ask him about the way these experiences can benefit people.
You alluded to, you know, growing up as a kid, you felt isolated and different than other kids.
And I just imagine, you know, I relate to that experience. I was so like, deeply into computer
graphics and visual effects and all this other stuff that like, nobody else cared about at the
time. And I, obviously, I found my escape through the internet and, you know, OG IRC forums and PHP
forums. I'm kind of curious, what does this do for people, you know, who may feel lonely today,
like what role can these AI characters play in sort of enriching their lives?
Big time, you know, big time. There is of course, online communities where you could meet those
like-minded people, but like they may not be available at the same time, but you have this
AI character who could effectively have all the right interests. Like think of your best friend
that you connected with the most, right? That understands you before you even say it, right?
So these AI characters can potentially be that for them. You know, like it is risky,
but that is where we are evolving to, right? Like, undoubtedly.
And do you find in that situation, like let's go back to a younger you or me,
do you imagine the experience being that these systems can sort of infer what you need or would
would I be like curating the combination of my like three best friends plus a little sprinkle of
like, I don't know, John Gata and, you know, some other VFX supervisors that I really like. And
let me throw in a sprinkle of Alan Watts and a sprinkle of like, you know, Einstein in there.
How would people define sort of the, their best friend, if you will, in this, in this space?
Yeah, that's a very hard one to answer because we don't choose,
I mean, we kind of choose our best friends eventually, but we don't choose based on vibe,
right? Based on vibe, right? Exactly. So and common interests and things like that
is how it evolves. But we don't exactly choose their eccentric things, what they are interested,
other things that they are interested in and whatnot. Right. So maybe it will be a multifarious
world with lots of different AIs. If these characters have to evolve, if you keep changing
them, you will not see their character evolution. Right. So you basically go and socialize and you
start with your one and maybe they will adapt to your interests and things like that. And eventually
they will have their own unique experiences too, that will shape them effectively, like
how it shapes you. Imagine friends that grew up together. Maybe this AI can also go out and have
have its experience when you are not there. Right. So they have stories to tell what happened today.
It's kind of mind blowing. What you're saying almost makes me feel like we've been at this
phase of technology and the internet where we can organize the world's information and make it
universally accessible and useful to use the Google mission statement. But now we're heading
towards people and personalities universally accessible and useful. Yes. And a lot of the
technology we are creating, you know, all the way from facial expressions and hand gestures and
emotional voices basically empowering the mind of these non-player characters are going to be the
same technology that may be used for a lot of these social robots. There's obviously both
utilitarian and delightful experiences your customers are building. What are you looking
forward to? We have set the vision. We have created the tools and people are developing
in terms of the immediate, not immediate, like medium term, three to five years plan is basically
ensure that we have redefined gaming in a very positive way. We have enabled these learning and
training experiences at scale, you know, that are changing lives of people in a very positive
manner. Brand experiences, product information, real world embodied characters. I love the creator
centric approach. I think it's so important that we don't forget that creators are going to author
these experiences. Thank you so much for your time. Thank you so much. It was great chatting.
Around the time Purnendu and I had this conversation, he invited me to the convey
headquarters where I got to see their NPC innovations firsthand. And let me tell you,
it was pretty wild. When I walked in, they had this massive monitor with an AI anime character
on it. And the people at convey told me to just have a conversation with it. They told me I could
talk to it in Hindi, which I was psyched about. Then I tried to press a little deeper. I think
there's this really human urge to try and push the boundaries of an AI system to prove that it's
intelligent and human enough to exist beyond the constraints of corporate language. Most of my
efforts were in vain, though, because as soon as the AI came back with these canned responses,
it kind of ruined some of the effect. Even though it was giving the unscripted responses,
it still had parameters. The AI allowed me to go off script. And even though the model I was talking
to recognized what I was saying, it was still responding based on its parameters, the guard
rails set by the company. While an AI like this might be great at helping you fight a lethal force
of aliens, it's hard to know if it will ever reach the messier, more human parts of how we relate to
one another. The TED AI show is a part of the TED audio collective and is produced by TED with
cosmic standard. Our producers are Ella Fetter and Sarah McCray. Our editors are Ben Van Sheng and
Alejandra Salazar. Our showrunner is Ivana Tucker, and our associate producer is Ben Montoya. Our
engineer is Asia Pilar Simpson. Our technical director is Jacob Winnink, and our executive
producer is Eliza Smith. Our fact checker is Julia Dickerson, and I'm your host, Belaval Seduc.
See y'all in the next one.
Jennifer Vail studies friction and how to deliver just the right amount of it,
like when you get a flu shot. If friction was working against you so much that that
needle is having to be pushed multiple times and harder and harder because the stopper
is getting stuck, you would definitely know. The hidden role of friction in our lives.
That's next time on the TED Radio Hour from NPR. Subscribe or listen to the TED Radio Hour
wherever you get your podcasts.
