Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Yes, it's a neural network, and it colors old black and white shots to make them look good, to de-oldify them.
What is amazing about this particular effort is not just that it seems to work well, but it's an amateur, in the best sense of the word, effort.
You don't need to be an academic or have the support of a big company to get into AI.
All you need is know-how and the application of a lot of effort.
Jason Antic, who describes himself as software guy, currently digging deep into GANs to do some cool photo colorization and restoration. has been doing the sort of work that usually needs a team of people.
His only help seems to be a 1080 Ti GPU, and even then it takes two or three days to train a new model.
That's David Conrad writing for a website called i-programmer.info about Deoldify, Jason Antic's system for photo retouching.
Usually, I try to come up with my own intros for our shows, but I really like David's take there.
It's just a guy, a 1080 Ti, and a self-attention generative adversarial network.
It's like the olden days. It's so quaint.
Joining us to talk about Deoldify is Jason Antic himself.
Jason, welcome to the NVIDIA AI podcast.
Hey, thanks for having me. Thank you for joining us.
So Deoltify is super cool. I've been reading about it.
It got written up on Boing Boing, which is always a good sign.
A bunch of other machine learning and computer science blogs, including the one I cribbed the intro from there.
And then your own GitHub and Twitter feed are great.
You have a lot of really great examples of of your work and action and lots of, you know, technical detail, but also explanation for the lay people.
So congratulations. and everything you've done to date and all of this sudden attention you're getting.
And maybe let's start, just tell us where the idea came from and how Deoldify came to be.
Well, I mean, I don't have too much of a story on the idea.
I just, you know, thought that colorizing black and white photos was just a really cool thing to do, honestly.
And, you know, I just basically what happens, I just finished my fast.ai courses like just two months ago.
And my plan was to dig deep into this neural network stuff and just hammer out a whole bunch of different projects.
So this is just going to be like the first of many projects I was going to do, very different projects.
So I just had this particular idea I wanted to try.
You know, there's a few different variations on this.
I was thinking about and I was going to try and I tried them.
They didn't work all that well. well, but then there's this other idea where it's going to use again.
And I was like, you know, it's, this is really ambitious, but you know, it'd be really cool if it actually worked.
And so I wound up just obsessing over that and it wasn't working for like six weeks.
And then all of a sudden I, figured out a way to make it work and it works really well, like way better than I thought it would honestly.
What's the phrase? Obsession is the mother of invention, something like that.
So prior to taking these courses then, did you have any experience with AI and deep learning?
I was a computer science major. in college and that was like a good, like 10 years ago.
Uh, actually more than that now. And they did offer.
I had a computer vision course and an artificial intelligence course there.
So two semesters. But you know, honestly, artificial intelligence back then just didn't really work. you know like like the computer vision stuff like uh they had like this facial recognition thing that you'd have to face camera exactly right and you know and it could maybe distinguish between two different faces some of the time you know right so it's nothing like what we had have now, but I've kept my eye on this field over the years and I've, you know, I read up on it and you know, like back in 2012 is when everything really changed, you know, with learning stuff.
So I started paying more attention and it kept my interest and you know, eventually it was like, you know, I really need to get into this field because it's just mind blowing.
It's going to, you know, revolutionize the world, I think.
And so I've had a few false starts over the past few years.
I tried taking a few different courses that a lot of people have heard of and I started and stopped because it just takes a lot of time and you had to find the right course and I just didn't find the right course.
But then finally, I found fast.ai and it stuck and I loved it.
And I even went as far as going part-time at work, just to pursue this.
Oh, very cool. Okay. Yeah. And you work as a software developer?
Yeah, I'm just an ordinary website developer.
I do the full stack, but very, very normal software engineer.
Which, you know, way back 15 years ago, like you were talking about, you were the king of the world as an ordinary website developer, how fast things change.
Cool. So the fast AI stuff, it did it for you for whatever reasons.
And you got through the courses and then how'd you, how'd you get in?
I, I was going to do one of those horrible faux journalism things where I ask you a question I know the answer to to set it up.
I'm not going to do that. I read that you have very little in the way of photo retouching experience yourself.
You've messed around with Photoshop a little bit.
I think there was something I read you likened it to like a kid mixing all the flavors of soda at the soda machine, something like that.
So without that, how'd you get to messing around with old black and white photos?
Well, yeah, I'm really impressed you did your homework.
You know, I think it's often the case with these deep learning breakthroughs is you got people that don't necessarily have the domain expertise, you know, coming in and just, doing the whole thing where you get the data and then you get the machine to learn from that data, right? you don't necessarily have to know the domain in order to be successful.
And that's the real power of this deep learning thing, this whole machine learning thing in general.
And that's exactly what I did. I didn't have any prior expertise on this.
I do know. that it's a very tedious process and it takes a long time it's very labor intensive so people who do this manually it's a very skilled thing to do.
It's painstaking work. Yeah. Yeah. Yeah. just to kind of set it up for folks, you'll take a, an old black and white image photo and, And it's not like you've got a colorized version to compare with, or it's not like you're taking a photo that was originally color and then stripping the color out. and trying to recreate it.
You're just going from an original thing that doesn't have color in it.
So there's no right or wrong. to your model's output, right?
You don't have a 100% answer you're going for.
So what is your criteria? What are you, when you think about, you know, yeah, I'm onto something here.
Like what's your, what's your North star there?
That's a really good question. You're right.
It's what is called, I believe, an unconstrained problem, which means, yeah, there's no for a lot of things.
There are right answers for a few things, right?
Like, you know, the grass is green. The sky is blue, skin is skin color and so on.
But you know, when it comes to things like, you know, what color is a car? or what color is somebody's clothing?
It's like, there's no right answer. Now, you've got to be really careful about how you train a neural network on that because if there's no right answer, you can easily – fall into the trap of it just throwing up its arms and saying, oh, everything's brown, you know, because that's an easy answer.
Brown's technically like closest to all those colors, because it's got all those colors mixed in it.
Right. What you really want, though, is something that is pleasing to the eye, I think.
And that's really the criteria I think I'm operating under more than anything else.
There's no... real objective criteria i'm using uh when i'm like posting these on twitter and stuff you know i I'm basically making an artistic decision, right?
Like I'm saying, yeah, it looks really cool.
I don't know if it's historically accurate and there's no way for me to figure out that this is exactly what it looked like back then, but it looks really cool.
And I think I'd make it as a wallpaper on my desktop or something.
As we're talking, I'm looking through your GitHub, which we'll go over these links again at the end of the show, like we always do.
But it's github.com slash jantic, J-A-N-T-I-C slash deoldify, D-E-O-L-D-I-F-Y. for folks following along at home.
And I mean, some of the images on here are just amazing.
You know, it's interesting. Some of them...
Some of them to me, and I do not have a very well-trained visual eye, but some of them to me look a little like those... those original efforts on whatever it was, Turner Network Television a while back, where they were colorizing old movies.
And you're like, ooh, heavy hand, a little too much.
And there are others that Paris in the 1880s under the Eiffel Tower.
And there's one Texas woman in 1938 where-
There's so much detail in the photo, small objects or shadows, different light gradients and skin tones, obviously.
And then but it really looks natural and you wouldn't to me, you know, I would look at the colorized version and I wouldn't think, oh, somebody or a machine learning network painted that in, I would just think, wow, what a cool photo.
So it's really something. It seems like you're doing a lot to document the work on GitHub and, like you said, on your Twitter feed. what kind of a response are you getting from, you know, from peers or just folks out there in cyberspace on what you're doing?
Oh, wow. Yeah, it's phenomenal. It's a little bit overwhelming, honestly.
Because, you know, it's a combination of things, right?
You know, a lot of people, you know, I get a lot of people from the machine learning world or people interested in that that are impressed with the technical aspects. aspects of it.
And in fact, I emailed one of the co-authors on the self-attention GAN paper because I thought maybe they would like to know that their tech that they made actually works really well with image modification.
That's not what they did originally. And they said, oh, yeah, we we totally love your work.
We were actually talking about it last Friday.
That's awesome. And hey, by the way, would you like to do a paper about it?
Yeah. And again, this is just like so...
Weird to me, you know, because like two weeks ago, I was like a nobody in this field.
I mean, literally a nobody. I just finished the courses, you know.
So that's pretty cool, yet a little overwhelming.
Right, right. And then you get – the thing I really like, though, is there's people that aren't in machine learning that just – you know, like the aspect of, you know, the art.
And then I keep getting... requests from people just went off request like from like five or six people now. where they're like hey can you colorize this old photo of my grandfather whatever And I kind of figured that would be the case.
That's really why I made this is because I think – you know, a lot of people are going to find it very personally meaningful to do something like this.
You know, there's a lot of history in the images, whatever, whatever they are.
Yeah, I mean, it's really cool to... make an impact like that.
And I just it wasn't the plan at all. I mean, I
It's supposed to be one project of many, and it just turned out to be a really good project.
Suddenly you're dean of machine-assisted art restoration at – yeah, no, it's –
It's crazy what the internet can do. You mentioned the self-attention GAN paper, and I'm looking at your GitHub page right now.
And it says, but I'm in love with self-attention Gans.
So, so. For me, but for the folks listening out there who aren't, you know, maybe they came to this episode of the podcast through the art.
What is a self-attention gap? Should I break down what a GAN is first?
Do it. Yeah, school's in session. Let's hear it.
So the best definition I've heard of a GAN or general... adversarial network is you got two deep learning networks.
You got one that's called a critic or discriminator, depending.
And another that's called a generator. The generator is the one that's making the images in this case, right?
And then the other one, is the critic that looks at those images and looks at real ones as well and tries its hardest to distinguish between real ones and fake ones.
And so it's trying to not be fooled, whereas the generator is always trying to fool that critic with more and more realistic images.
So the whole idea there is that this competition is supposed to lead to really good results, and it does.
For the most part, so far you've seen it produce really good results with random images generated from noise, like celebrity faces and stuff like that.
So, you know, I, what I did is I just took that and made it work for image modification.
Now, the thing is, uh, with the self-attention again, they, uh, added one more. in my mind, crucial component.
Well, two things that they made a difference in.
The first thing, which actually I think made it possible to train this to begin with, was that they added this thing called spectral normalization, which just –
The TLDR version of that is it just makes the training a lot more stable.
You know, before I just wasn't having any success and it would just, the images would just explode into these weird kind of artistic looking, but not what I was going for kind of thing.
Right, right. It just wasn't working. Believe me, I was trying really hard.
It just didn't work no matter what knobs I turned.
And so I added this, the spectral normalization.
And then the other component that they added, which is really crucial too, is this thing called attention.
And in attention, I mean, it kind of literally is what it, says it is just in this case, they were trying to solve the problem. of you know these gans tend to make like 10-legged dogs and stuff like that like it just wouldn't It wouldn't keep things consistent across the image.
It would kind of lose track of what it's doing at one end versus the other.
And if you think about it, that's quite relevant, actually, to the problem of colorization, because you want, for example, consistency. across like the colorization of the ceiling of something or, you know, like a... And indeed, if you look at other... colorization efforts in the past, you know, even if it's like a big gigantic, like, you know, thing in the middle of the, in the middle of the picture, like a pool or something like that.
I remember seeing this where. half the pool would be filled with blue and the other one would just be black and white.
It's like, how'd you miss that? Well, it's because you didn't have attention.
Right. Right. As far as I can tell. I mean, now I'm going to add these disclaimers here because I'm like.
Honestly, when I implemented this, you know, I was really frustrated with WiserTeen's GANs after like six weeks. which I really thought would work, but they just didn't.
And I just skimmed this self-attention paper, honestly.
And I was like, okay, I know what attention is.
And, you know, I just read the abstract and was like, oh, that sounds promising.
And I just threw it in, let it go overnight and it worked.
But I can't, I mean, you know, if you're going to like drill me on the details on like the exact theory of it and the math, no, forget about it.
Well, and then, I mean, I think that really speaks though to, you know, like you were saying before, the fact that like- you can just do this now, right?
You can... Or, you know, as the intro that... that David Conrad wrote, you know, you're not a professor at a research university with a whole team of grad students.
You're not, you know, heading up a team at some big company, like you're a person with some know-how and some skills and a GPU.
And you're able to actually like, OK, yeah, I'm going to skim the paper.
I understand what it means. Let's give it a shot.
Let's see what it does. No, that's exactly it.
That's actually really the message I want to get across more than anything is Uh, it's actually the thing I really want to try to pursue here is like, you know, I think this field up to this point, it's kind of been dominated by academics, right?
You even hear, I think Andrew Ng has said this before, where what we really need is to get this deep learning stuff out in the hands of like, you know, normal software engineers and, you know, get this, you know, actually working and real world situations and make it, usable, you know?
So, you know, that's really my aim here.
I, you know, I, I've been a software engineer for like 10 years and I think, you know, that's really. what I can bring to the table here with deep learning is just, you know, trying to take this cool stuff and trying to make it work and make it work well, not just be a tech demo.
We're talking to Jason Antic. Jason is the author of a deep learning system called DeOldify. which is a photo retoucher and colorization system.
It's super cool. You got to go check it out for yourself.
Best place to go is his GitHub, although you can also just Google Jason Antic, Deoldify, and you'll find a bunch of articles there. that we mentioned at the top.
But on his GitHub, he's got a really, really well laid out and written explanation of what's going on. technically, non-technically, and a bunch of examples of the colorization and process.
And Jason, since this is something you got into, you were just talking about Maybe I could ask you, how do you work?
You're one guy, and like you said, you're a software engineer and have a bunch of non-
AI-related experience and you start poking around for classes.
And so maybe, you know, I'm putting you on the spot, but if you can, in kind of a very high-level, brief way, Walk us through, what was it like getting started with the OpenAI classes?
Did you have to go out and buy more equipment?
Did the whole thing take a lot longer than you thought it would?
I don't know. For somebody listening who's thinking like, you know, yeah, I'm like this guy.
I'm a, you know... I'm good at what I do and I, you know, software wise and tech wise, but it's not AI, but yeah, I'm going to dive into it.
I don't know, what advice would you give or what's kind of, you know, how did it work for you?
Okay. Yeah. So, okay. It's fast.ai. You said open.ai.
Oh, I apologize. Thank you. That's fine.
Yeah. Because I want to make sure people know it's fast.ai because it's a wonderful course.
They're doing wonderful things. And I would actually say that's my first piece of advice is go directly to Fast.ai, they will set you on the right track, I think.
Because their whole philosophy, their whole message is trying to break down the barriers to deep learning.
And that's not just in terms of the material, learning it.
And they do a great job of that. But it's also like, you know, the hardware, you know, how you get started because, you know, the fact is, is you really don't need. to have a whole server farm of GPUs to do this stuff.
And the first lesson you take in fast.ai is cats and dogs.
Mm-hmm. They have you train a computer vision system that can distinguish between cats and dogs with like 99% accuracy or something like that.
Right. And it takes like two or three minutes to train it on your computer.
It's amazing. I mean, granted, these GANs that I train on my computer – uh, take three or four days, but I can still do it, you know, and it's just, a single GeForce 1080 Ti.
It's not the cheapest card, but it's something that I know for a fact a lot of people have because they have gaming PCs.
Anybody that has a gaming PC can do this for sure.
Stop mining Bitcoin and start training Gantz.
That's a really good message. Yeah, so I'd say if you're trying to get into it, don't view the hardware as as a barrier.
Don't view the material itself, you know, in terms of like the, you know, whether or not, you know, you have the intellectual capacity or something to learn it.
Because honestly, I kind of suck at math.
You know, you might think that math... is a prerequisite.
And if you look at all these papers, they are definitely heavy with math, I mean, I can read them.
Don't get me wrong. I'm not math illiterate.
No, we hear you. But you shouldn't. It's not as daunting as maybe people people think it might might be before they get their hands dirty.
Yeah, honestly, I get by with a lot of good intuitions.
And Jeremy Howard is really good at teaching those.
If I have to look at the math, I can read it really slowly, like under my breath out loud kind of thing.
Right, right. So none of that. Don't view that stuff as a barrier.
I think the thing that's the primary barrier is is you know making making the the proper amount of time to do it in the first place you know like it's it's And this is where I really hit a few snags over the years.
Like I tried several times to get into this, like going back, like all the way back to like 2014 or 2015.
Like I was really interested in this stuff as soon as it became popular.
It was finally in like 2017, the summer of 2017 that, I took Jeremy's fast.ai course.
And it's the first iteration and I really liked it.
But, you know, I wound up taking vacation days in order to do it.
And I was kind of getting burned out after a while.
So I was like, OK, I definitely want to do this, but it just it takes a lot of time.
So, you know, what I did is Uh, I sat down, I was like, okay, if I realistically want to get into this field, I, I just. need to make some hard choices here.
And what I did was I waited for a year. I saved up some extra money and then I negotiated a part-time situation at work.
So I could use the rest of the time to really dive deep into this stuff.
Now, I know that's not realistic for everybody.
No, but it makes the point that – Carving out the time is perhaps, you know, the biggest commitment to make or say it's commitment, but That was the big barrier for you.
It wasn't the intellectual stuff. It wasn't, do I have the right equipment or I don't have access to whatever.
It was like, all right, I'm going to do this.
I need to carve out the time to really get into it.
Yeah. And I really think that is an important message, honestly.
Cool. Jason, before we let you go, and you may not be able to answer this, we were saying before we hit record that...
Or no, you said it on the podcast here just a couple of weeks ago.
You were just some guy toiling away at his fast.ai courses.
And now you're all over the media. You're on the NVIDIA AI podcast, for God's sake.
But what's next? Are you going to ride the Deoldify train, so to speak, and keep working on these models?
Or are you teeing up one of those other projects that... that you had thought of.
Oh, yeah. I'm totally going to try to milk the whole deodified for all it's worth.
Excellent. Good, good. I mean, I'd be a fool not to.
No, actually, honestly, I don't expect to make a hit like that all the time.
And so, yeah, I'm going to. You know, rather than just be satisfied with a tech demo that only works on Linux, And up until yesterday, because I pushed a really awesome update yesterday, it was only working on like $500. pixel images, you know, 500 by 500 pixel images, as good of an image as you get.
But now it's, you know, that's not a problem anymore.
But anyway. you know, stuff like that, you know, it's just, you know, it was a tech demo.
Right. And also you had to like, play around with the size parameter to get it right.
And you know, quote-unquote artistically select amongst like 100 different versions.
You know, so I don't, you know, I'm not satisfied with that.
Like I want it to just be something that anybody can use on their phone or something like that. and just say, hey, I got old pictures.
I want to de-oldify them. It'd be really cool if people use it as a verb, by the way.
But... It's every young programmer's dream.
So yeah, that's what I want. And I think that's where I can actually really make an impact.
I kind of accidentally, the whole thing in terms of running into this colorization algorithm that actually worked well, but I do know I can engineer things well because I've been doing it for quite a while.
So that's what I'm going to do. I'm going to try to make it work more efficiently, more reliably, maybe on video eventually, and make it so that you don't have to install 100 different packages and only on Linux.
You know, just stuff like that. You know, just make it useful.
Very cool. Well, if people want to track your progress, I mentioned a couple of these things before, but why don't you tell the folks where the best places are for them to follow your work? book.
Yeah. So you mentioned the GitHub repo and pretty much the way I've been communicating to my quote fans, I guess, is on Twitter.
I just, you know, every, every day I've been posting, you know, two or three result images.
And, you know, there's two reasons I'm doing that, you know, one to, uh, keep people up to date on the progress because the images keep getting better and better.
And, you know, the other thing, too, is just I really like posting these old images.
It's pretty cool stuff. Yeah. And the screen name is Sidnaj.
C-I-T-N-A-J. Just my name backwards. That's your Twitter handle.
Cool. All right. Well, Jason, fantastic.
Thank you for taking the time. It was a pleasure to talk to you.
It was a great conversation and your approach and kind of your, you know, look, man, this is who I am.
And this is why I got into it. And I think other people can benefit from the work.
And also, I think from your story about like, You don't have to have XYZ to get in this.
Just make the time and you can get going.
It's a great takeaway for folks. So thank you for coming on the show and all the best of luck to you with Deoldify and whatever comes next.
Thank you. Thank you.
