Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Since it launched in 2006, Roblox has grown into one of the biggest gaming and digital experiences platforms in the world by basically any metric you want to look at.
According to the company's fourth quarter 2023 financial reporting, Roblox boasts more than 71 million average daily users, accounting for 15.5 billion hours of engagement last year.
Those numbers include both players and creators, as user-created content has always been central to the Roblox experience.
Providing both cutting edge tools for creators and quality experiences for users at scale is no easy feat.
And with so many of Roblox users being under the age of 16, including my own son, who can often be found drifting virtual supercars through various of the Roblox games.
Keeping things civil and safe for everyone has to be a high priority.
How do you run a platform like this at such massive scale?
And how does AI play a part in the process?
Here to take us under the hood of Roblox is Anupam Singh, Vice President of AI and Growth Engineering.
Anupam, welcome, and thank you so much for joining the NVIDIA AI Podcast.
Hey, Noah. Thank you very much. The son I mentioned in the intro made sure I did my homework on this one.
He's very excited we're having this conversation.
Maybe very briefly, because there's a lot to dig into in terms of the infrastructure and the things that are under your purview. again, for both the creator and end-user side of things.
But for folks who might not be familiar with Roblox, and I think there are at least a few out there.
Maybe give us an overview of what Roblox is all about.
At a vision level, as a founder talks about it, for the last 10 plus years.
It's about reimagining the way people come together.
You and I want to hang out somewhere. Obviously, we have many, many different ways to hang out, but Our vision is to bring a billion people together with optimism and civility.
Now, especially the civility part is very important because that gets us to deep technical innovation to keep everything safe, everything civil, And from day one, we've invested a lot in that area.
And optimism is about, if you go into one of our experiences, you'll be amazed on how encouraging and interesting they could be.
There's the latest one is something called Dusty Trip.
There's another one called Pet Simulator, where you hang with your pet, but there's a lot of other things going on.
Valentine's Day, St. Patrick's Day. So it's all about connecting a billion people with optimism and civility.
And it's a group experience, right? As you put it, it's as much, it seems, about hanging out with your friends as it is about this incredibly diverse array of games and experiences.
Yes, absolutely. And so how do people hang out?
It's text chat, voice chat. How does the interaction happen?
Yeah, so it starts with you open the Roblox app or the Roblox webpage, and there's hundreds of experience that are there for you to select.
Think of a movie that you have to select to watch, but after that, you're inside the movie.
That's the difference. And so you press play like you would do on any other app where you consume content.
But then you get into the universe. And my favorite ones are where you can actually immediately create an avatar.
You know, you get feathers if you're willing to.
And if you're interested, you get like a lovely feather dress and you, you know, There's a lot of audio going on.
And so another favorite ones of mine are You know, where you can dance with people.
So I managed to say ballroom dance is one of my favorites right now.
Excellent. And I just keep learning dancing, something that honestly, I wouldn't be able to in my normal weekday, but I can do a ballroom dance in the evening.
So let's talk about your role, what's under your purview and how that sort of informs the experience for the millions and millions of Roblox users.
One of the things, as I understand it, that you're responsible for is managing and optimizing infrastructure across the platform.
And there's a lot happening now with AI and generative AI tools in particular.
So maybe you can take us a little bit into that.
Yeah, so two-part answer. When your son presses play, he doesn't realize that we have to connect him to a data center, which is hopefully close by.
So that his experience is amazing. Remember, your son is of a generation who believes that breast clay does not have buffers.
And I'm sure looking at you, you remember the time when you would press play and some buffering would happen.
So that's the first magical experience. Oh, I remember that.
The AOL dial-up sound, that's what I'm thinking about.
There you go. But this generation of users on the internet that expect to press play, Then we have to place inside the experience.
Behind the scenes, what's happening is he's connecting with one of our edge data centers And then you have to decide who are the best people your son could play with You know, it has to be age appropriate.
It has to be interesting. Maybe he already has friends on the platform.
So I have to quickly figure out Where are his friends?
Are they in the same data center or they're in a different data center?
Placing them in the right instance of the experience.
Our biggest experience sometimes has 40 to 50,000 copies running worldwide. on 20 plus data centers.
We have to place him in the right instance.
After that, let's say he wants to chat with people.
So we have to enable that chat. Let's say he wants to do a voice call with one of them.
We have to enable that. Let's say he actually wants to talk to somebody in India, which means he has to talk in Hindi, but Assuming, let's say, your son doesn't know Hindi.
We have real-time translation. to facilitate chatting with somebody in India, so in Hindi.
So that's the kind of stuff that we in the infrastructure team build.
Now, we like to be invisible. You should never think about the data center that you're connecting to.
Right. How does machine learning play a role?
Broad question. Very broad question. So we think about machine learning typically from, to start with, either for the creator or for the user.
So you have to create these magical experiences.
We talked about Ballroom dance, we talked about dress interest.
There's a lot of storytelling in there. But most of our creators have the right code to build that world.
Anything that facilitates their building of the world, it's their idea.
We don't build our own experiences. It's millions of developers who build experiences.
We need to facilitate that really, really quickly.
And I'm sure we'll go into those examples.
On the other side are the users. The users just join an experience.
And if we want a safe environment, if we want a civil environment, a very simple example You and I are chatting on Roblox.
Every letter that you type, every word that you type, goes through a text filter for a civility check.
And just that portion of our world is i have to get this right it's it's 90.7 billion messages are translated and 2.5 billion chat messages are sent every day.
And every one of these have to be checked whether you're being civil to me, whether I'm being civil to you.
And this is in real time across... All the different languages, all the different data centers and geographic locations and the private chats and the public chats and just the scale.
Everything. Yeah. Now, of course, people choose to sample this in other settings, but from day one, We've not sampled.
Every one of these goes through a text filter, which means you need a machine learning solution to look at 2.5 billion chat messages And sometimes, as you said, in a private setting, we might be more colorful with each other But in a public setting, we're not going to be colorful.
We should not be colorful. So making all those decisions in real time could have only been possible with machine learning.
How do you approach a problem sort of at that scale?
Has that been from day one? Has the importance of real-time civility tools, to put it that way, has that always been baked into practice?
Roblox's mission or is it something that kind of evolved out of necessity?
Yeah, so one thing for us is it's always been baked into it.
So our engineers never start building something saying, maybe we should sample, maybe we should just do it for NOAA, but not for Anupam. which is arbitrary, right?
It's for every user. So first, your design principle should be clear that I want to do it for every user, every platform.
Once you've established that, every few years, we have had to change our machine learning model.
So a few years ago, there was a model that was... you know AI people love to have acronyms by the way Noah so one of them is SOTA
State-of-the-art. The problem is the SOTA changes every year.
So we had a state-of-the-art model which could do a billion messages.
So we took the model, we trained it with our own data, and then we had to make it smaller to run faster right right it could be billions of parameters and then you're sort of Put it on a diet now so that it can run faster.
And so we go from, let's say, 70 plus billion to 7 billion per hour.
But what's happened in the last, I would say 24 months, which is exciting, is the entire model architecture is the same.
So it's not just about, hey, let's take one more model and that does natural language processing you now have all these generative ai models that can do text summarization, that can do text moderation, et cetera.
So we take those bigger models And to run it on scale, we take every technique possible.
We have to distill it, which will make it smaller.
We have to fine tune it, which take our data and fix it.
And then we run it in production. And honestly, once you start running things in production, you have many facepalm moments where you say, wow, We could have done this.
We could put it in memory. We could back the thing.
So yeah, it's our design thinking and then running it in production.
I know that, broadly speaking, and also in Roblox, and as I understand it, even within the purview of scaling safety and stability, multimodal LLMs models are, uh, I don't want to say the new thing, but you know, they're becoming more and more important and talked about in, in,
All phases of technology. How do multimodal LMs play a part, multimodal models play a part in your work And this may or may not be relevant, but it's something that popped into my mind talking about civility.
When you were talking about running the chats through text filters, it made me think about slang and lingo and how you sort of keep up with, you know, words that I might not understand, but might be extremely colorful, to use that word, to people in the So how do you just keep up with all of that?
And then second question is about non-text things.
So if an avatar... has something questionable, or I don't know actually if, you know, you're allowed to share images and memes and that sort of thing.
But yeah, you can talk about that. Yeah.
So let's talk about multimodality. And then we'll talk about, you know, what I would call internet wisdom.
Perfect. Things that you and I might not know are offensive. but they're offensive.
But multimodality, you're 100% right that in the last two years, We just assume text models are there.
Any of us can access PPP, any of us can access, you know, any model that you want to pickup, right?
What's exciting about the last one year is the use cases are also changing.
So more and more on our platform, people would love to use voice.
What we did around two years ago, we decided that we will treat voice the same way as we have treated text, which means You know, you want to check for profanity and other offensive words in a speech.
And so we used a very large model. Thankfully, large models are available both open source and closed source. to convert firstly voice to tech and then we use all our tech knowledge from 10 plus years to moderate Why?
The next question that you ask is even more interesting, which is images.
We always had image moderation. in our platform since 10 plus years.
Our founder likes to tell the story of He used to do the moderation cue maybe 15 years ago.
He does not anymore. But we can make billions of thumbnails every day.
And the exciting part is each one of these large models has the wisdom of the internet packed into them.
Earlier, we might have to train it from set.
Maybe five years ago, Roblox would have to train it from set.
So now there is a new plan. or a new offensive in there, I don't get the benefit of somebody else playing a large state-of-the-art commodity, an open source in there.
But now we start many of our initiatives with a large open source model that has the wisdom of the internet. and is able to update to newer things.
Then we fine tune it when we distill it to Roblox as we would like.
So image, voice, and text are all in production for us.
Then if you're going to take multimodality further out, and I know you talked about this in some of your other podcasts, is 3D.
That's not just a picture that is, you know, I could make something really interesting.
And I don't want to just focus on civility here.
It's just the beauty of what you can create on the platform uh and how quickly can you create about it so we are very focused on 3d and 4d as the next modality.
And there, honestly, we don't see a lot of models that have already been built in open source.
There is no easy wisdom of the internet as far as 3D and 4D.
And so you're using kind of across all these things, I'm generalizing, but you're using a mix of available open source, available models that you find train and otherwise, but then you're also building models in-house.
Yeah. For example, for recommendations, our model is internally built.
For certain safety use cases, our model is internally built.
We have a coding assistant. but we have our own language.
So we take an open source model and then we combine with our language and build a lot model.
I'm speaking with Anupam Singh. Anupam is Vice President of AI and Growth Engineering at Roblox, Roblox is one of the leading online platforms in the world, bar none.
But certainly when it comes to creating and using all kinds of digital experiences, From games to roleplay simulators, ballroom dancing, as you said, on a palm.
It's just amazing all the things that are out there and being built literally every day.
You touched on this briefly, and I want to sort of flip over to the creation experience and how...
You know, you mentioned Roblox has its own code, that Roblox Studio I believe is the platform for creating.
But actually first I wanted to start with talking about avatars. um because my experience has been you know as a user as a parent of a user but then as a user myself i'll go on and play games um i love the driving games they're some fun and Big paintball is one of my favorites.
But the whole idea of an avatar, is just incredibly essential and important.
And it's vital to so many people's online identities broadly, but particularly on your platform.
You know, every time my son will show me like, hey, check out my new avatars outfit.
And it'll be, you know, his favorite footballer's uniform or it'll be, you know, some some just wild costume or this or that.
So can you talk a little bit about the technology sort of behind being able to create and customize avatars?
And, you know, how that relates to, I mean, basically anything that is important to you, but talking about civility, but then, you know, more talking about just like giving... the tools to express themselves.
Yeah. In terms of such a rich theme, we could spend hours talking about it.
Sure. Let's start with a very simple application.
There's a likelihood that you want the author to reflect some of your personality, right?
Including... You have this lovely backdrop behind you, which people can't see on the audio podcast of these lovely electronic devices.
And so let's start with taking your photo and creating an avatar out of it.
It would be simple if it was only 2D. are on the phone.
You can do them today. Great technology by all the great phone manufacturers.
But you want it to moan, you want it to moan.
That's when, you know, creation of these photos of dark Generation of 2D and 3D are major innovations, and it is ongoing.
There, our technology has to... both be built from scratch, but also use some of the existing 2D to 3D innovation.
Then, okay, we took your photo, we will turn our car, but we want to set it up.
You know, we want to give it a form, which is beyond just your maybe face.
That's how thought ought to set it. Next, you want to track your upper body because your upper body movements, I love to gesture a lot.
Maybe you gesture less. How do we capture that?
And after all of that, we need to put it in a world in a cohesive way.
So each one of these, think of these as almost separate modalities.
Movement is a different modality from texture.
It's a different modality from what you're wearing, which is material.
And none of that to you or your son is interesting.
It is our job as an AI platform that the right models get called, the right models get orchestrated.
Of course, in an ideal world, and I'm looking very, very into the future, All of this should be answered by one model.
It can do photo to octa. It can set up your upper body.
It can... set up your face and body coordination.
It can set up your gestures and then put you into a world where you're dancing to impress or dressing to impress.
But in reality, I'll tell you, each one of these are different models being called for very specific, special things to do.
Let's talk about creators. You mentioned you've got your own language that creators can code in.
There's obviously, as we've alluded to during the conversation, all kinds of visual 3D physics engines, all kinds of things happening in these experiences.
How are you incorporating generative AI into the creator tools?
And what are some of the, not just concerns, but some of the exciting things that are happening whether they're at the level of an individual creator or if we're talking about orchestrating all of this at this massive scale.
Yeah, so the creator journey is very fascinating because many of our creators are amazing storytellers.
Right. And that's, and forgive me for interrupting, but just to sort of emphasize, it's been like a wake-up call to sort of how, you know, this generation, and for me, it's the generation of my son, but obviously your users span ages. but just how important storytelling and just sort of the like personal narrative, emotional aspects of these experiences really are, right?
I'll watch my son role play buying a house and decorating it.
And just doing what to me seem like mundane things, but he's hanging out and living this experience.
And it's amazing. Oh, yes. Building a house on roadblocks is so much fun.
We could go on. But let's go to the creator.
So there are creators. One of my favorite experiences right now is being able to land on any airport in the world. which is, you know, airport simulators has been around for a long while, but combining it with roadblocks and storytelling,
But think about the creator. What they have to do is they have to start building the airport.
And all of a sudden, it vibrates. We want to facilitate that really, really quickly.
It all starts with the most popular use case today in the industry is coding assistant.
Just let me go back. But after I've coded the word, now I want to give it texture.
Now I want to give it material. You know, you're giving your creator, you're giving your users lovely aviator glasses.
They could be pilots, right? Top gun reference.
Sure. And these aviators should have textures.
They should have shine. The material should be maybe metal.
All of this takes some time for our creators.
And they're in a hurry to tell a beautiful story.
But this takes time. So for every step of their way, generative AI starts helping them.
The coding assistant is a pure text model, but the material and texture models are much more richer and more interesting.
And now every one of our creators would love to have their players worldwide.
Right, of course. Now, should they spend time translating?
And that's where Roblox comes in. Our translation is automated.
You, if Noah, is creating a brilliant experience, but...
Now you want to translate it everywhere.
We just make it happen. Now, once people are into your experience, we do all the chat translation too.
So from the time of coding to the time of release, we are helping you everywhere, virtually everywhere.
That reminds me of something I wanted to ask you earlier.
When we talk about real time, in the context of say translation or even civility filters to put it that way.
Do you have a benchmark for what real-time means?
So real-time for us is literally as you're typing, the translation happens.
So as long as the user doesn't see the thing, you know, we talked about when you press play, you shouldn't see buffering.
Yep. That's what we aspire to. Yeah. So we shouldn't even exit.
A great example of that is, and our CEO, our founder loves to show this example is, Noah and Anupam get on a one-on-one on Roblox.
We have Roblox office. You and I get on a one-on-one.
Noah says something colorful. Yeah. In real time.
As I want to do. Yeah. In real time, you will get a nudge from our voice moderation system.
Right. Then you say something more colorful. you will now be suspended maybe for a few minutes.
So that is what real-time means to us. It is not a warning that will come to your inbox a day later.
As we start to wrap up here, I'm going to ask you the question we always ask at the end of these conversations.
What's next? What are you either... working on now or kind of looking ahead to?
You mentioned, of course, way out in the future, the one model that can handle all these different modalities and things.
Are there either problems that you're trying to tackle right now or something in the near horizon that you're really excited about that you might give us a glimpse into?
Huge excitement is 4D. I know you've talked about 3D and 2D in many of your podcasts.
The fourth dimension, and I sound science fictiony, but the fourth dimension is you want a car in your experience.
The car doors have hinges. It has to open the right way.
My favorite example is I only create, I live in San Francisco.
I want to recreate all of San Francisco in its glory.
Alcatraz Iron Ferry Building, the Painted Ladies, Victorian Houses. on Roblox.
I can create that, but it will take me some time.
But once I create a Victorian house, I want to enter it.
I want to open the door I want to hear the echo because the sound is different when I enter a Victorian house versus a glass house.
All of that takes effort. on our current platform?
Of course. What if you could just talk to it and then you start building the story?
You've seen all these movies set in San Francisco, right? from romantic comedies to action thriller.
What if we could do that storytelling much faster because the base tantrums were created so quickly for you.
So that's what I'm excited about, where they don't think about models.
They don't think about coding assistance, voice translation, et cetera.
Think about storytelling and the models in the background just assisting them continuously.
It's not even called an assistant because it is so much in the background.
Anupam, we could talk for hours at least about any of these topics, as you alluded to. throughout.
And perhaps we'll get the opportunity to do this again in the near future.
I don't think. roblox is going anywhere so it would be great to check back in as as technology advances and the experiences advances But we do have quite a few technical minded listeners in the audience.
So for folks who want more, want to dig into how all of these things, the models and the filters and the real-time translation are happening or happening at such scale.
Are there places online, obviously Roblox has a website and a blog, but are there specific places that you would direct folks to learn more?
Oh, yes. We have a Tech Talks podcast. Oh, fantastic.
That is hosted by our CEO and founder, Dave Bazooki.
Every one of the topics that we talked about, whether it is South Ours, growth, engineering, safety, civility, We have 40 to 45 minutes of deep dive by Dave himself.
Great. And that's Roblox Tech Talks? Yes.
Excellent. There's a developer blog as well?
Yes, there's a blog.roblox.com where you can go and look at what we're launching.
Great. Well, Anupam, thank you again. This podcast is not about me, but you've made me a star in my own household, at least for a day, getting to talk to... talk to you.
So I appreciate that. But more importantly, appreciate your coming on the podcast just to give people. a little bit of an insight into everything that has to go on behind the scenes so that whether you're chatting with a friend, being kept safe and civil, or, you know, experiencing these amazing 3D experiences, you don't have to wait for things to buffer.
They just happen. Yes, thank you very much.
I'm an average listener of your podcast, so this has been great.
I appreciate that. Look forward to chatting again and all the best to you and your teams.
Thank you. ¶¶ Thank you.
