Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. We're coming to you from GTC24 at the San Jose Convention Center, and we're here to talk about hyperspectral data.
What does hyperspectral mean? What is hyperspectral imaging and why should it be more accessible to more people?
Well, I've got just the person to answer these questions and more.
Here to help us understand all things hyperspectral is Robin Wang.
Robin is co-founder and CEO at Living Optics, whose recently released spectral imaging camera is bringing advanced imaging to the mass market. including a host of new applications across multiple industries.
Robin's here. Let's get right into it. Thank you so much for taking time out of your GTC week to join the podcast.
Welcome. Thank you, Noah. Great to be here.
How's your trip been so far, first things first?
It has been utter carnage, but hopefully in the best way possible.
Everyone's coming out of the woodwork and we're still here and it's still alive.
I kind of want to end the recording right now because what more is there to say?
But I'm going to ask you about living optics and about what hyperspectral imaging is.
So let's get into that. Right. So living optics, hyperspectral imaging.
Where to start? So normal human eyes, normal cameras, your iPhone camera, your webcam, they see in three colors.
That is a bit limiting. Right. Hyperspectral camera, like ours, ours one specifically sees in 96 colors. which means you can see tiny little hidden details that are floating around in light, but not detected by us. normal cameras so things like blood oxygenation plant health plant stress these things are completely visible to us we can just see them This is somewhat of a trivial question, given the nature of the whole conversation.
But when you say, my eyes can see in three colors, your camera can see in 96 colors, The other 93, are these actually like novel colors, so to speak?
It's not just blends of red, green, and blue?
It's kind of hard to visualize because it's literally invisible.
So out there in the world, like... But let's do it on the radio.
I don't make it easier. So I think a good way to think of How three colors is limiting, but also the blend of three colors.
Think of a computer screen or a camera. They only have red, green, and blue colors. receptors or LEDs.
And every color you see out there, all the kind of salmon, mauve, cyan, they're all blends of these three colors.
But what we're missing is the finer details between these colors.
So a good example is blood oxygenation. This, in theory, is in the visible range, but the spectral features or the color features are so fine that human eyes, no matter how hard you try, cannot detect them.
If you train a computer vision model with a trillion images, you still can't see them.
So these are the fine little features that you need hyperspectral imaging to see, that you just can't do that.
Unfortunately, Although it will make the world a better place, can't see what's a normal red, green, and blue camera.
Right. Got it. I like to pull things off of people's websites and then ask them about them and make them speak to them.
Your website says, at Living Optics we are dedicated to a future where hyperspectral imaging is not just a tool for the few but an asset for the many.
I remember telling the marketing people to write something similar to that.
So that was all good. Good marketing people.
So not obviously knowing too much about spectral imaging myself, When I was poking around, I was kind of surprised to see how many different use cases there are already across so many different kinds of industries.
My question is, how are you going about making that technology more accessible to more people?
But maybe before that. Some of these use cases, where is hyperspectral imaging being used and kind of why is it so important?
Right, so there's a few questions there.
Where's it being used? Why are we making it better?
I warned you ahead of time. Before we started, I told Robert.
I claim to ask one question and it's really got 12 parts and I apologize.
No worries. So where's it being used? Why are we different?
Yeah. What's been, yeah, anyway, those are the two I'll answer first because I forgot the third one, sorry.
So where's it being used? So hyperspectral has been around for, say, 40, 50 years.
So think about Newton's prism, the Pink Floyd.
Wait. Yeah, those guys. Yeah, it's been around.
So it's been around. Starting was military and satellite imaging.
The first hyperspectral imager was actually built in Eastern Germany.
It cost $200 million. And they had to make a special machine tool to make it.
So it started off seeing as a defense tool, seeing camouflage, seeing You can see further underwater, for example.
I should stop there before I get shot. And then slowly it moved into mineral prospecting.
Where do I dig up things? So... A lot of ores you can't see with a normal human eye or normal camera, but you can see the chemical signatures inside with hyperspectral imaging.
They're like, ooh. Let's go look at some good stuff here.
And slowly over the years, this technology became popular.
A bit cheaper. And by a bit cheaper, I mean it went from a million bucks to about $100,000, which meant people, big national labs, started using it.
So that's where I came from, a big national background.
So people started using it for plant health, like medical imaging.
But the problem was these hyperspectral images were still 100 grand.
It wasn't a million bucks anymore. And they were slow.
They were really hard to use. They were big, chunky bricks with no prettiness or industrial design.
They scan things line by line. So like we said... Sorry.
Like I said... There's 96 channels or colors in this image that we take.
So the hyperspectrals, the 100 grand guys, by taking these colours slice by slice.
It was like building up a ream of paper one sheet of paper at a time.
So to get one hyperspectral image, that was a few minutes.
And then if you're moving or your target moving, then get lost.
You're done. There's nothing there. What we've done here is we've cut that 100 grand price down, we've taken zero off, and we've also My finance director is not going to like that.
But anyway, and then we've also made it video rate.
So rather than waiting two minutes for one image...
We can go 60 frames a second. Wow. Okay.
Use it like a normal camera. Yeah. There's just better because you see more colors.
You see better detail. You can do more. There are so many advantages now that we're in this age of computer vision.
One of the biggest problems is gathering your joint data set.
Say if you wanted to see something really niche or specific in your field, say one of our customers wanted to do see the difference between different apple varieties.
No matter how many images they gathered, they couldn't see the difference between a Braeburn or a Golden Delicious.
Turns out you only need 10 hyperspectral images of each apple variety to tell the difference to basically 90-something percent accuracy.
Wow. The other weird thing is, I shouldn't say this out loud since Jensen said how big a bigger server is, is that we trained that model in about 0.3 seconds.
We'll keep that between us, but that's fantastic.
So I think I know the answer to this, but I'm going to ask anyway.
When you talk about taking a hyperspectral image and then seeing, what is that actually like for the end user?
Is it data represented? in a chart, on a graph?
Is it a long text string? Or are we somehow magically making me able to see the other 93 colors I couldn't see previously?
Oh, this is a really spicy good topic is that you can't visualize this because it's literally invisible to the human eye.
We realize that. partially through the competition, it's been around a few years, we can still call ourselves a startup, but anyway, is that you need to give people the tools to work with hyperspectral data.
If you just try to dump hyperspectral on someone, they're going to look at you and say, what the hell do I do with this, Robin?
You're hurting me. We came up with the SDK.
So that's completely open source. I think one of the goals of this company is that We don't want hyperspectral imaging to be behind closed-door big national labs.
We want you to access it. We want this to be everywhere.
A few ways to massage this 96 channel data.
So think about it as a book. But human data can only be seen in three colors.
So one way is to analyze this data to, say, bring out one of the applications as chlorophyll content.
You calculate an overlay and you paint it on top of your image.
It's like, hey, here's where your plant is super healthy.
Here's where... Plant diseases, cut that off, otherwise you'll lose your whole harvest.
Right. Or spray here because there's fungus here.
So you overlay that on top as a big red... Think about a highlighter.
Yeah, we just highlight the scene and then that's the part of the user experience we really want to make easy is that you don't need a PhD in spectroscopy to use our camera.
Right. So... I'll answer your question in a really roundabout way.
I hope you got what you needed. No, it was a good answer.
I like it. And the highlighter analogy, you're painting on top of the image, right?
A 2D image. with very valuable additional data sort of annotated on top.
Exactly. You have to store 96. or channels into a highlight or a sharpie, whatever you like.
Yeah. So you mentioned a few, taking care of plant health and that kind of thing, chlorophyll.
What are some of the other uses, both sort of industrial and maybe some of the outliers, some of the weird, wacky uses of hyperspectral that you've seen?
This is the weird part, and I can't tell you too much, and you'll see why, is that the military found us before we spun out the company.
They had their own mega spectral camera and they saw you when you were invisible.
I think they saw us as something that's an upgrade.
It's faster. It's cheaper. You don't have to.
Parliament won't come spank you when you break it. because this is faster, cheaper, better.
And over the medical side... We've just started.
Just today, it got confirmed, so I can talk about it.
Working with the Bill and Melinda Gates Foundation for anemia detection and for blood oxygenation and for... because Bill Gates and I, we had dinner once together and we agreed that this has got to be everywhere.
Yeah. Yeah. And I think I just discovered really weird things along the way.
Cause it's like, I was a PhD student I was a shut down lab I was doing math I didn't know the world out there I was like 1.8 billion people have anemia There's no good way to see it.
And they just locked onto this. Like, hey, come see this.
And then other things are, I think we mentioned plant hells.
There's a lot of industrial inspection, wafer imaging, color matching.
Turns out, When you want to quality assure, you want to see if something got printed correctly or you're clothes got dyed correctly, you hire someone, a human that is qualified to see color correctly.
It's really subjective. And then, yeah, this is the solution.
This is a literal number you can get out.
This is... blue this is navy you can quantify that rather than just if someone tell you right this is blue And then, of course, cannabis farms have come out of the woodwork, fish farms.
All these industries I never really knew about.
More will come to mind, I think. Yeah. One thing I will say is that most of the customers, most of the inquiries are unexpected because sometimes when I take a step back, which I don't do very often, is this is just a camera, but better.
Right. And cameras are used everywhere.
And camera has problems everywhere. So people have been coming out of the woodwork and saying, hey, what you got?
Can you see my thing? And my answer a lot of times, I don't bloody know.
So when people use the camera for the first time, how do they react?
It's like a kid in a toy store. They're like, oh my God.
Yeah, I remember there was a really surprising application.
A civil engineer came to us and said, Hey, we need help here with the camera, because we need help in identifying cracks and bridges, and the serious cracks.
I'm paraphrasing here, of course, have algae or lichen growing in them.
And it's just a thin, thin, like a microscopic layer that's not visible to human eye.
I remember she said, can you do that? I said, I have no idea, but let's try this algorithm we built for vertical farms.
So we went to the car park and picked up a few rocks and we put it on the table and turns out one of them had lichen on it and lit up like a Christmas tree.
Yeah. And then so recently we took it to a bridge We pointed it out the part of a bridge and we see the lines where the cracks are forming and the lichen was growing.
And that was just something that just came out of the blue.
I remember when they messaged us, emailed us, saying, hey, can we talk?
I almost said no, so... I said, civil engineers, what have we got to do with you?
So I think I've learned that lesson. I got to take some of these calls.
I'm speaking with Robin Wang. Robin is the co-founder and CEO at Living Optics. whose new spectral imaging camera is, as we've been talking about, as Robin's been talking about, faster, cheaper, more accessible than spectral cameras of the past.
Let's talk about the camera itself for a second. as has been kind of a theme of this conversation so far, I'm imagining you'll say, well, I can't really talk about that, but I will anyway.
So that's what I'm aiming for. Is your camera, aside from being faster and cheaper, and I'm guessing smaller than cameras of the past...
Is it built differently? Are there technological innovations?
Are there things that you develop from the ground up that are patented or should be patented going on? in the camera itself?
Or is there something else happening in the software stack and the algorithms?
Or is it just simply a matter of, well, nobody's updated the hyperspectral camera in a long time, and technology and optics have advanced so much we were able to do it?
I'm going to give you a terrible answer, and that is all of the above, and I'll go into the detail now.
How about we start from the beginning of the beginning?
This was actually my PhD project. It was a side project.
I started off as a fresh-faced stupid theorist.
I'd never... Where? In Oxford University.
Okay. Yeah, so I'm originally from New Zealand. which is why I sound like this.
And yeah, I came out of applied math and a pure math degree and went straight into the world of physics.
Okay. My supervisor, being a diehard experimentalist, they go and solve this problem. which was every time we turned on the laser, I used to work at an inertial confinement fusion, either cost $70,000 or cost $2 million to fire a shot.
So we needed to get as much information out of that shot as possible.
So he said, hey, Robin, I shouldn't try to do a British accent, but Hey, Robin, go build a camera that sees at 100 trillion frames a second.
That's a world record. That camera didn't exist, of course, because that's a world record.
So we built one. So I haven't published a result yet because, you know, startups are busy.
Yeah, but we got a video where light moves at about three microns per second. per frame.
That video was a hundred frames per second.
But turns out Only about six people cared for a video about 100 trillion frames a second.
So I applied for a patent. It's like, hey, cool.
No, I don't apply for that. My supervisor made me.
And then out of the woodwork comes a bunch of VCs and say, hey, what else can we use this technology for?
The technology was to take, ingest a 2D image and spit back out a 3D cube.
So the 100 trillion frames a second is, we did a one shot.
We ingested all the super fast events onto a 2D image.
They reconstructed it into a video. That's useless.
So we said, okay, wait, we can reconstruct Spectra too.
So the main funky part is, OK, we can reconstruct and the rest follows.
It's like, oh, Jesus, here's all those applications.
Let's form a company. Here we are. But the funky part of how this technology works is in order to make hyperspectral imaging, I realized that You can't just improve the hardware or the software.
You're getting blood out of stone. People are smart.
People have gone to the end of that path.
You have to do both together. That's where you get the orders of magnitude. improvement.
So we kind of went back to first principles.
We rewrote ray tracing software. That was horrible.
If I knew how hard it was going to be, I would never do it again.
We redesigned the software and the hardware to work perfectly with each other.
And since I had limited resources at hand, I couldn't buy expensive components I was scraping by.
So I inadvertently, by accident, stroke of luck, borrowed a lot from the mobile phone industry, how they design lenses.
So from the ground up, this camera was to be manufactured affordably and cheaply, not because I thought I was going to start a company, because I was cheap.
So I bought a lot of components from AliExpress and scraped together.
And by a lot of luck, it worked. So to summarize, designing hardware and software together in an affordable way, borrowing a lot from the billions of dollars people have sunk. into mobile phone manufacturing.
So we have a beautiful, wonderful manufacturing team in Taiwan. that we poached from Lagan, who designed the iPhone lenses, and Yang, who designed, okay, I can't tell you that.
Yang Optics, who designed something that is coming out that is very exciting.
And we stole some of the iPad team at Foxconn.
Gotcha. And the goal by stealing those people was to... scale this product right the thing was to be affordable scalable mass manufacturer consistent Soon, you heard it here first, self-calibrating, because that's one of the biggest problems in all instruments.
And so currently it needs to be calibrated before it can... It takes about three seconds, but we're trying to make that zero seconds.
Nice. Good. That's why I love doing this podcast, because you'll get that question.
It's like, well, no, it just takes a couple seconds.
That's too long. We're down to zero. Yeah, this is your experience. the more things the more barriers more friction we can take away from sex recipe the more it can change the world and that's what very and british about us well i'm not british what's is try to aim high and slather this over the globe.
Aside from the apparent trauma of designing your own ray tracing software, what was a particularly difficult, surprising...
Maybe you look back and say, like, that was one of those out of nowhere aha moments during the development. of the camera, the hardware, the software together that you kind of look back on that really stands out.
I think the hardest part, though I'm going to completely hijack the question, was the people. is that the technology always happens.
If you put enough smart people in the room together, that will happen whether you like it or not.
But I think one thing I realized is that how much was rely on the people. is that if you're one of the best manufacturers, you can't go to the best manufacturer, optics manufacturers, and say, hey, I've got the best technology.
You have to know them. You have to tell them the story.
You have to get the right people to sign up for the best products to be made for you.
I think that's something that I never really realized is that the best technology doesn't win. the best combination of supply chain, people, user experience.
That wins. And I think I learned that the hard way.
But I think after those lessons are learned, you can hopefully see the fruits of it as, hey, here's a camera.
Bloody well works. Yeah. There's a friendly SDK that helps you do what you do rather than fights you along the way.
So we're talking at GTC. NVIDIA has been around for a while.
They're around before this current AI explosion.
I think we were talking before you used the the phrase AI wazoo is happening everywhere.
How does Living Optics fit in? There's... robots, there's SDKs, there's new hardware, there's pipelines for enterprise applications, all this stuff happening all under the AI umbrella.
Hyperspectral imaging, living optics, your camera.
How does that slide in? This is about going back to first principles, which is garbage in, garbage out.
Yep. Is that... We've done computer vision to death.
RGB, three colors. People are building billion parameter models.
People are training. Each model takes, good big model, it takes millions of dollars to train.
Our argument is Why don't you get better data to begin with?
Why don't you get a richer, more informative data set? rather than just adding more layers to your neural network, or better transform it, LSTMs.
More data. Get better data. Use smaller models.
Not because you want to. No, not because, use smaller models, because you can, because that beats state of the art and RGB.
Use less data. Spend less expensive engineering time labeling.
This is the argument. Get better data. Get hyperspectral data.
Whenever there's a vision problem to solve, don't use RGB.
Just because evolution evolved us to see RGB doesn't... mean machines you see in red, green, and blue.
If you want to make a really informed decision, see beyond just red, green, and blue.
See hyperspectrally. I think I'm going to borrow that line and use it randomly with my kids.
Thank you. Just keep saying that. Don't freak people.
Just pick it up. You know what? You want to figure this out, see beyond RGP.
I'm going to shift gears for a second. Living Optics is partnered, part of the NVIDIA Inception Program for Startups.
You can talk about your startup experience.
You talk about it a little bit. You can talk about it in general if you want.
But what is the Inception program like for folks out there listening who— maybe have an idea, maybe they have a company already and they're wondering, you know, yeah, I keep hearing about this inception thing.
It is. What's it been like for you? I think inside all the startup program wazoo, NVIDIA Ascension has been really bloody reasonable. they have not taken my time when they didn't need to they didn't make me come to events Well, I wanted to come to this one.
They didn't make me come. So that's why I'm here.
And then they've just tried to be actively helpful rather than trying to push their own agenda.
So can't recommend it enough. Yeah. Send them an email.
Why have you got to lose? Right. Well said.
So what's next? Hyperspectral imaging, living optics.
Is the camera shipping? Yeah, it's chipping out.
It's on the shelf, I think. So somebody listening right now could go to the website and order one?
Yeah, probably. If you have fast, it probably comes in three days.
But if you are between the manufacturing cycles, maximum three weeks.
So this is another part we're trying to change is that, yeah, this guy is mass manufacturer.
The yield is high through the wazoo. I should have made I should have made the optical design more risky.
The yield is too high. Yeah. Yeah, you don't really hear that.
Could have done more. Yeah, could have made it smaller, quicker.
So yeah, go ahead and buy it. Comes in the whole dev kit.
Like we on purpose need it. come with support we want to help you right we want to make it work you want them everywhere you want spectral spectral data everywhere everybody using it and what's next yeah Smaller, better, faster, cheaper.
We're selling a development kit. The products.
Look at us. It's going to come next year.
Is there more that can be captured and extracted from hyperspectral imaging, or is—
I don't know what a good, I'm not a photographer per se, I don't know what a good analogy is, You know, if we've been living in a world of three color images for so long without knowing there are actually 96 colors, you just can't see most of them.
Is there something beyond that? There is, and it will come, but I will not tell you what it is.
Oh, I could not have asked for a better answer.
That's fantastic. Robin, for listeners who want to get a camera, who want to learn more about the company, who want to see a picture of the camera first, because we're doing a podcast, they can't see it.
Or for folks who just want to learn more about hyperspectral imaging, what you can do with it, how does it actually work in more detail, all that kind of stuff.
Should they go to the website or should they go to find out more?
Go to livingoptics.com. On the top right thing, there's a shopping bag button.
Click on that. And I'm sure this is not the first time you've heard this, but do you do voice acting?
Do you read bedtime stories somewhere people can go?
And granted, I've been at the show podcasting for two days straight here.
So, you know, but your voice is so soothing.
Thank you. This is just me talking for so long.
This is my hoarse voice that happened after six hours of shouting at people straight.
Well, if you ever do voiceover tutorials for the camera, shout for six hours and then record because it's—
We're talking about the future that it's hard to visualize, literally invisible, and yet I feel so calm and soothed.
Thank you. Yeah, it's fantastic. Robin Wang, thank you so much for taking the time to come on the podcast.
I need to go find a way to experience. hyperspectral imaging myself and see what 96 channels of data look like.
You've inspired me. So come to the business.
No, I'll shout at you. Fantastic. All right.
We're going to sign off. We're going to go to the booth.
Enjoy the rest of your time at GTC. Best of luck with everything.
Very excited to chart the progress of Lung Optics and see where the next generation of imaging takes us.
Thank you so much. Thank you.
