Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Whether you're wandering through the sprawling virtual worlds of Grand Theft Auto or picking out a new piece of furniture on IKEA's website, virtual worlds are here and they're everywhere. but we've got a problem.
There just aren't enough artists to build all these virtual worlds and populate them with friends, foes, and furniture.
Well, that's where Art-O-Matics comes in.
Artomatix is using AI to bridge that gap and fill our virtual worlds with awesome digital art.
Today, we're joined by Artimatic's co-founder, Eric Risser, to talk about how deep learning can merge with art to fill these virtual worlds.
Eric, thanks for joining us. Thanks for having me, Noah.
I'm really excited about this podcast. Us too.
And we're excited to be joined by you from halfway around the world in Dublin.
I hear you're in an old Guinness factory. yeah yeah it was uh yeah about about 200 years ago it was an operating uh basically storehouse for grain and sugar and barley.
Now it is tech offices. Let's start at the beginning.
What does Artomatix do? So we solve the problem that art costs too much and it takes too long for the entertainment and design industries, and we do it through better technology.
So we've built an AI that can mimic human-like artistic creativity for very kind of low-level, tedious kinds of artistic creativity.
So we think of it kind of like we're doing for the for the creative space.
What what spreadsheets did for the accounting space where, you know, A lot of what you had to do in accounting was just like spreadsheets of simple arithmetic.
And computers are really good at doing simple arithmetic, so that's one of the first things to automate away a lot of the tedium out of an industry.
And then that let accountants actually be... be more creative, be more high level and strategic with their accounting, less grunt.
And we're aiming to do the same thing for the creative space.
So by having a machine almost be like your team of digital assistants, it can automate away those tasks that you really don't want to do.
You know, the stuff that you kind of That, you know, you that like entry level artists kind of have to do to pay their dues or you have an intern do, you know.
The kind of stuff like removing seams from a texture or like making a bunch of different simple concepts on a theme and stuff like that.
So you mentioned spreadsheets. It's a good segue.
You've got something called the zombie graph, which is kind of a cool example. of what you're talking about and how your technology works.
Can you explain? Sure. So the zombie graph was just kind of a layman's way of explaining, you know, some sort of concepts from information theory basically we're just trying to to frame the idea of machine creativity in terms of data redundancy or entropy so The way I like to explain it, imagine you're seeing your first ever zombie.
And imagine you're like a newborn baby and for some reason you can still parse the world and understand things, but you've never seen a zombie for whatever reason.
And you're seeing your first ever one, and presumably you've never seen a human face before.
There's so much new information in that image.
There's the two eyes, the nose, the mouth, the ears, the shape of the head.
So it's mostly new information. Now you see your second ever zombie.
Now there's a lot of redundant information.
There's still, you know, the two eyes, the nose, the mouth, the ears, but you know, now there's some new information as well. such as blood splatter, scars, maybe one of the ears is missing, maybe, you know, part of the, the scalp is ripped off.
Then you see your third ever zombie and, and, You know, there's still some new information.
There's different blood scars. Maybe the other is ripped off.
But the reality is a lot of the information is the same.
It's still two eyes, a nose, a mouth, and the general shape of the head.
And the idea is it's about... you know, certainty over, over how much data you're looking at.
So as you get more data, you become more certain as to what this thing or is less uncertain.
So. The zombie graph is basically just us plotting unique information over total information.
As we add in new information, elements of a set, a set being, you know, any category of art such as zombies or cars or dogs or whatever.
So from the standpoint you were talking about, you know, sort of a higher level artist managing sort of a virtual team of entry level artists doing the grunt work.
How does that work? Do they define the parameters of what makes a zombie and then what they might like in their individual unique zombies and then? automatic spits out an army of zombies?
How does that happen? So we call our workflow, the air quotes, example-based workflow. as opposed to procedural.
So in procedural, you'd write some bit of code or put together some kind of a node graph that is essentially a piece of code. that says how to put together, how to programmatically create a bit of art.
We go a different direction, so an example-based, Basically, the artist, the way they program Artomatix is by just making some examples.
They just... They just either find them on the web or make them themselves.
They provide Artomatix with that example of a few zombies.
And then Artomatix basically defines that probability space of what a zombie is.
And the zombie graph more or less just says, this is kind of, what your confidence looks like as you add elements to a set.
And basically, you need to add enough elements that you kind of give the computer something of a well-defined probability space, and then it can take it from there and add in a way where it kind of conforms to a very specific curve.
So the way an artist kind of injects their opinion on that is very much in a genetic algorithm approach where you give a few examples from the set, so you kind of divine the variance of your probability space,
A computer kind of imagines some new things. within or maybe even a little bit without that probability space.
Sometimes you can get cool new things if you let the computer go a little crazy, and then you give feedback.
So then you say, oh, I like... I like these features.
You know, I like this ear. I like this nose.
You know, you highlight the stuff you like.
You're basically. telling it kind of what function you like and lowering the variance around that area of the function.
And then you say, okay, now let's try again.
Take the features I liked, throw out the rest, basically home in on the function I want and now generate another population more variants and kind of rinse and repeat until you kind of walk in on the exact thing you want.
So it's very much the way, you know, artists are currently working with the concepting phase of things.
So in looking at the Artomatix website, you offer several services related to textures and creating textures that clients can use in their own... artwork.
And I know that there's a somewhat obscure field called texture synthesis that had an impact on your own work and and the development of Artomatix as well.
Can you speak a little bit about what texture synthesis is and how it relates to deep learning and neural networks and everything that Artomatix is doing?
Sure. So yeah, texture synthesis has been very influential for myself personally, as well as the company.
So texture synthesis is actually a relatively old field.
It dates back at least 20 years. and it's a subfield of the computer graphics area.
It had its heyday In the late 90s, early 2000s, with a few influential researchers such as Lee Eway and Aaron Hertzman.
And then for a while, people actually thought texture synthesis was kind of a solved problem. to the point where it could pretty reliably do a decent plethora of different stochastic textures. you know, things like grass, concrete, asphalt, you know, that sort of thing.
It could even do some slightly structured things like brick walls with some basic regular patterns.
But texture synthesis never really had any kind of real-world application because the algorithms were all very slow, very low-resolution.
There was some work on doing them in parallel on GPUs and kind of pre-processing certain data to make it faster, but at the end of the day, it just wasn't really... practical and in a real world use setting.
Until 2009, when another researcher named Connolly Barnes came up with an idea called PatchMatch. which essentially took the really slow aspect of texture synthesis and made it near instant.
And that was actually what led to the first real commercial feature of texture synthesis.
So that would be Photoshop's content-aware fill. which actually went viral back in around, I want to say 2010.
So that was actually an implementation of a subfield of texture synthesis called image and painting, where the idea is if you have a grassy field with somebody walking through it, you you put a mask over the person and then it deletes them and then it fills it in with new grass. merging it in with the surrounding area.
So it's texture synthesis to do whole filling essentially.
So you spoke about having an academic background.
But how did you get into this? Did you know you wanted to be a computer scientist as a kid?
Were you drawn to art? Were you building zombies in your backyard?
How did this all get started? Well, so when I was a kid, I always wanted to be an artist and I had absolutely no talent for that.
I was clearly, it was pretty obvious to everyone I was going to be a computer scientist or some kind of engineer.
So many of us wind up where we are because we have no art talent.
I have all the personality of an artist and none of the talent.
So I figured the next best thing was to get into video games.
I thought I'd work somewhere in the video game industry.
And I even did some indie stuff in high school and early college and then realized I had absolutely no talent for that either.
Luckily, I got offered a job working in my university's graphics group just as an undergrad researcher.
This was at the University of Central Florida, working in the real-time rendering lab.
So I first did some real-time graphics techniques, so GPU programming.
So I was... I think it was one of the only 20 people to program a GPU back when Shader Model 3 was brand new.
Actually, my first involvement with NVIDIA was I published a chapter in GPU Gems 3, I believe, back when I was an undergrad.
I did a bit of real-time rendering, but I didn't really feel like that was for me.
I decided that I was always really into AI, and I thought to myself, is there some way that I can use AI to do computer graphics?
Is there some way I can make an artificial imagination?
And what's cool about being a researcher is you can really just work on any crazy cool thing you want.
There's very little expectation to deliver.
And I guess what really inspired me was I took this graduate course on neural networks at UCF, where I.
And this was back before we had buzzwords like deep learning, like it was just neural networks because they weren't really cool yet.
And as my final project, everyone in the class basically had to implement a neural network from scratch and then apply it to some new untried application domain.
So, so like. this neural network could basically encode, like it could learn non-linear functions.
So I thought, okay, what if I could give a 3D model and train it to try and basically just mimic that function, you know, just trying to encode that function in the form of like nodes of a neural network, nodes and weights of a neural network.
So from your undergraduate work, you went on and did a master's at Columbia and then went overseas to Dublin.
And what happened then? So I tried it with neural networks at the time.
Neural networks were just too slow, too low resolution to really be practical, but I fell in love with the topic.
I went to Columbia University, learned about texture synthesis, worked on a graph paper. with Charles Hahn, Eitan Grinspen, and Ravi Ramamorthy, got really into texture synthesis, realized this is something that could actually achieve that goal of AI art creation.
My doctorate was really about taking this field that was very much how could we expand out textures of grass and make it work on really highly structured things like zombie heads.
And then how we can take all of these image-based techniques and make them work as 3D models.
So that was really what my contribution to the field was. highly structured, sophisticated things and how to do that as shape as well as color.
And then when I finished my PhD, you know, I knew I had at a very high level, a broad solution to what a very high level I knew was a big problem.
And I thought, well, this really needs to exist in industry.
I love this field. I've been working on it pretty much all of my adult life.
I should really... see if this could be a thing.
And that's why I started Artomatix and it's been going great ever since.
When was that? How long has Artomatix been up and running?
I turned in my thesis in 2012 and opened up a fresh visual studio project and started coding, but.
I'd have to say that it really got off the ground in early 2014.
That's when I incorporated the company, got some funds, started building up the team.
And today, yeah, we're up to 20. Excellent.
Okay. So how is Artomatix taking that idea of patch match?
I like that. I'm going to latch on to that one. patch match and texture synthesis and using neural networks in the work that you do now.
So we do it in a number of ways. So texture synthesis historically hasn't used neural networks.
That's actually a relatively recent development.
And that was in a, I want to say the summer of 2015, when a researcher from the neuroscience area named Leon Gattis wrote a pretty influential paper on the field.
He basically treats parametric texture synthesis as a neural network training problem.
And this is super cool for two reasons. One, it's the first attempt to meaningfully use a neural network to do texture synthesis.
And two, it just completely shattered the state of the art in parametric texture synthesis.
So Parametric texture synthesis was kind of a direction that people tried in the late 90s.
What does it mean exactly, parametric texture synthesis?
So let's say you put in an input texture, so a picture of grass.
Parametric texture synthesis would be you analyze that input and you turn it into just a set of parameters. know, a set of numbers that somehow statistically, meaningfully represent that image.
But then you can throw away the image and you don't need that anymore.
You just need that kind of summary statistic.
Right. As opposed to non non-parametric where it's more of a machine that tries to create a new texture. that statistically matches your original texture.
Using that original texture kind of as, I guess, the ground truth that it's always comparing itself against.
So it's more trying to minimize. some descriptor against what reference it can find in there.
And that was really very much like the kind of I mean, that was kind of the branch of texture synthesis that really won out past like the mid to late 90s.
And in many ways, that is still the leading approach.
So what was the innovation that parametric texture synthesis brought and why it's still being used today?
Oh, well, it really wasn't, it was kind of competitive with nonparametric in the, in the nineties and then nonparametric just became vastly better.
What kind of gave it its resurgence was neural networks.
Like in the 90s, researchers were trying to put together kind of simple, I mean, they were taking the examples and trying to extract simple kind of statistics out of it using, you know, hand-tuned descriptors. and they were just kind of trying various things.
And I think as the whole industry is saying, you know, if there's anything related to human perception, Learning, I guess, your descriptor using a neural network is just the way to go.
So a lot of kind of what made parametric synthesis, I guess, become a phoenix from the ashes is...
The idea to use a neural network to basically summarize the parameters of an image.
So basically run an image feed forward through a pre-trained VGG on ImageNet.
Now all of a sudden, all those activation functions are really good parametric representation for what that texture is.
Gotcha. Okay. So let's, let's shift gears a little bit and talk about your company Artomatix and what you're doing now, what the service is, and it's applicable, as we said at the beginning, to more and more industries all the time as more and more industries delve into virtual worlds and using 3D art for marketing, sales, entertainment, all kinds of purposes.
So what does Artomatix do and how are you different from what else is out there?
So I'd say what makes us different is that we're really the first company to use you know, data driven art creation to really commercialize it.
There are other companies out there that are really more tools for artists to manually make art.
So either paintbrushes such as in Photoshop.
Or a coding platform to kind of build art from the ground up using, say, noise functions and various programmatical techniques such as, you know, like algorithmic substance sweep.
And those are, those are really awesome tools, but we saw a hole in the market where. those tools are really for very talented and trained, technically savvy artists.
And There aren't really tools that are that appropriate for people kind of just getting into art, say, from like an indie game standpoint or people that just need to do. some certain tasks on, on 3d art that, uh, are relatively simple and they don't necessarily want to have to go to art school and learn a very high learning curve tool for.
I'm building a virtual world and for instance, I need grass to fill parts of the world.
I need it to look good and I need maybe background characters, whether it's a zombie army or aliens or what have you basically and and you know maybe you don't have a huge you know art team and a massive art budget right uh so Yeah, so we kind of fill that niche.
And also for AAA game studios, this isn't just a tool for indies.
It's for them, too, because... At the end of the day, the budget for projects have been going up exponentially over the last decade.
And it's led to the reliance on overseas low-cost outsourcers, which isn't necessarily a sustainable practice because the desire for content is going up exponentially and artists aren't going up exponentially at some point that can't continue how big is the market do you think how big is the opportunity for artematics Oh, it's massive.
I mean, it's billions because there's several verticals.
You know, you have You have entertainment, which is movies, games.
You have advertising, which again has the same problems.
You have industrial design, which is furniture, fashion. architecture previs.
And what's cool is there are all these different verticals that have traditionally tackled 3D in different ways.
And they're all starting to consolidate and combine around some communal standards.
So actually a lot of the pipeline is starting to become unified across different industries and as more new industries start to come into the 3D space.
And so how does a company... whether it's an indie game maker or a big studio or an Ikea, so to speak.
How do they work with you? Do you offer, is it a service model?
Do you offer APIs? How does that happen?
At our core, we've built a platform. So we built an API that does example-based art creation.
On top of that API, we offer it in different ways.
So We built a kind of simple GUI for artists to come in and play with.
We built that within Chrome as kind of a web browser, WebGL 3D thing.
And we offered that as a software as a service tool.
It did relatively well, but we got consistent feedback that people didn't really want to work, like artists didn't really want to work within Chrome.
It was awkward sometimes. They always had to drag and drop their art into the browser, wait ages for it to upload.
Then they had to download it, grab it from the download folder to put it into Photoshop.
It was kind of a pain. We've actually been working on basically the, I guess really you could call that the web prototype and now the version 1.0, you know, air quotes.
So basically the professional studio version.
So we've got a tool that you can install on your computer.
It'll leverage as much of your hardware as possible to do as much locally as possible.
And it'll leverage the cloud for whatever your hardware needs. our tool for artists or our tool for people that want to use a tool themselves to do this sort of thing.
But at its core, you know, our product is really the API. which we also licensed to people directly.
And there's a number of clients in the works for that.
We also kind of came across certain clients that they don't have programmers, they don't have artists.
So verticals that are just now getting into 3D for the first time, such as furniture companies, fashion companies, architecture companies, companies that don't necessarily have artists or programmers.
They don't necessarily want to use a tool or an API.
They're used to outsourcing all this work.
So we figured, hey, Our tool sets make us very competitive against cheap overseas labor because we essentially have very cheap AI labor.
So we're also happy to do service work as well.
So let's extrapolate from that five years down the line, five years of Artomatix providing. art to the world.
What's the world going to look like? Where's Art-O-Matics heading?
And where's the role of AI in creating artistic content?
Where's that all going? I hate saying what five years from now will look like because I'd say...
Three years ago, yeah, I wouldn't have imagined today would... would look like today.
So it's really hard to say, but I can tell you what I hope it'll look like.
Speaking of somebody who's been an academic in this obscure field of texture synthesis and neural networks, which was, you know, maybe a little more famous in science fiction, but equally obscure in terms of research.
I've kind of been in these obscure fields that were largely overlooked and not really even valued in industry at all.
And now, I mean, they're exploding. I mean, neural networks are currently the really groundbreaking new technology.
Machine Creativity, which was... I think I had to explain to people what it was five years ago when I started Artomatix.
And, you know, I started Artomatix because there was no industry for this.
And I really loved the field and I knew it was important. and I knew it needed to exist.
Now people get it. People are aware of Prisma.
People are aware of Google Deep Dream, all of these kind of fun little appy things that are showing up that do some kind of generative adversarial network, you know, generation of something.
Things that you turn doodles into cat drawings or whatever.
I mean, people get it now. People know what it is.
Companies like NVIDIA, you guys were kind enough to host us at your booth at SIGGRAPH.
It was amazing to see your tagline being AI for computer graphics.
That's something that, you know, I never would have dreamed during my PhD.
Any company would be championing that. I mean, if anything, it was almost controversial to call any of this. stuff AI.
So what I hope the future will look like is I'd like to see this trend keep going. you know, talking to artists and sitting down with them and looking at their workflows and having them explain that, you know, oh yeah, I spent a week basically generating you know, trash textures for this dystopian future wasteland, you know, getting, getting like the bump of this like old, generic Coca-Cola bottle and glass shards everywhere.
Your AI can take care of that grinding. Yeah, yeah.
It's like just give an example and just let it do it for you.
Just this building of huge worlds. I'd really like to see, like, artists in five years look back and like, can you believe we used to remove seams on textures manually?
Like, that's crazy. I would... How is that our job?
We're better than that. We're way too talented to waste our time on stuff like that.
And five years hence, they'll be more freed up to dream up even greater things. grander dystopian worlds because they don't have to spend their time rendering trash.
Exactly. And Eric, you mentioned before that you're preparing, it sounds like, a new tool for artists to use that'll be downloadable, installable.
Can you give us any sneak peek on when that's going to be released?
It's currently in closed alpha. We're about to open that alpha up at the end of the month.
And we're planning for a public beta launch at GDC 2018.
Oh, we like the sound of that. For a lot of people, I think, you know, a lot of indie game artists out there, a lot of just people that want to create something cool I think that the current learning curve and talent curves are prohibitive.
And I think with virtual and augmented reality just around the corner, I'd like to see it five years from now, VR and AR really being not just a new screen for a video game or a slightly more immersive video game.
I'd really like to see it become personalized spaces that we create.
I'd like to see it be more of almost social networking thing, you know, almost like a Minecrafty thing in a way where like you build your own virtual world, you customize it, you make it your own.
So it's a little bit like the holodeck from Star Trek.
And then everyone has their own world, and anybody can create anything they want.
I'd really like to see everyone become an artist and everyone be able to put that into more of whatever it is they do, you know, either if it's for recreational entertainment use only, or if it's their actual career, if it's their actual profession, I like to just see 3D be more ubiquitously prevalent for everyone?
And I think that AI is the thing that's going to do that.
As we've seen, computers have kind of enabled so many people to do so many things.
I see this as, again, just an enabling technology.
I'll tell you what, there's no better place to land our chat than on that nice pivot from a dystopian future to a...
Entirely utopian one. I love it. If people want to see what Artomatix is doing and examples of your work, where should they go?
Go to www.artomatix.com. Terrific. Well, again, thank you for your time calling in from lovely Dublin, Ireland.
Eric Risser, CTO of Artomatix. Thank you for being on the podcast.
Cool. Thanks, Noah.
