Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Ever see a photo of an amazing looking meal? maybe in a food magazine or on an Instagram feed, and wish you had the recipe to make it yourself, Thanks to a project born out of the MIT Computer Science and Artificial Intelligence Lab, we're a step closer to being able to do just that.
Joining us today is Nick Pines, one of the minds and stomachs behind Pick to Recipe.
Nick, thanks for being here just in time for Thanksgiving and the holiday meal season.
Yes, of course. My pleasure. Thanks. So, Nick, what is Pick2Recipe?
What does it do? Pick2Recipe is an application that goes from image of a food item to the recipe that most likely created it.
It's a little bit different from, for instance, reverse image search, where you go from image to image and then back to uh the content in this case we go directly from image to recipe Okay, so it's a little more than just a reverse image lookup.
How does it work? I've got a picture of sweet potato casserole, and I upload it into Picture Recipe, what happens?
So it takes a state of the art visual model and turns the image an internal representation that it can then use to identify similar recipes that have been transformed into that same representation.
So for instance, our example of sweet potato casserole, it could be finding sweet potato casserole.
It could be finding something else with sweet potatoes, a different type of casserole.
Is it breaking things down to the individual ingredient element?
So implicitly, yes, it has to do that. So when it looks at a sweet potato, I can't really tell you right now exactly what it is doing because that's just the state of deep learning these days.
But under the hood, it's basically saying like, okay, this is like an orange thing over here.
This is maybe like a marshmallow over here.
And then as the image is processed by the model, it develops a conception of bait. like casserole-ness, like it sees the pan.
It's like, okay, this is a casserole. Then it sees like the orange, right?
This is like a sweet potato. And then it merges those into like sweet potato casserole.
Of course, if it can't see something, for instance, like it might think it's like a squash casserole or something like that, but usually it knows what it's doing.
Does it run into problems with ingredients that are maybe hidden under the surface?
So in the case of... Clearly, it's almost lunchtime, and this is a great topic for me right now.
In the case of casserole, it made me think of something like a lasagna. lasagna where maybe you've got a layer of baked cheese on top and that's the visual element, but there are other things underneath.
Does that kind of thing, the system work on that or is it an issue?
So that is definitely an issue and that's actually one of the main challenges I think facing food recognition in general.
But the model can infer things about the image and the rest of it generated it.
So maybe it can't tell you that it was like a... vegetarian versus like spinach versus meat lasagna, but it can tell you that it's definitely a lasagna type thing.
How many recipes are we talking about? We've got to have more than just a couple dozen since this is a live demo and not something canned.
But how many recipes are in the system right now?
So right now we have exactly like around 1.2 million recipes and several million.
Well, now several million images but the dataset that's been initially released only contains 800,000 images. which is a fairly reasonable number.
Sure. And so people are uploading new images via the web demo?
Yes, and these can be used to train the system on real data because the things that people upload to for instance, a recipe website, are going to be there.
It's going to look nice. They're like showcasing their results.
So a demo lets us get real-world data that lets us train a really good model for when you're in a restaurant. for instance.
Got it. So this was born, Picture Recipe was born out of your time at the MIT Computer Science Artificial Lab, Artificial Intelligence Lab, excuse me, like we said. but now you're a PhD student at Cal Berkeley.
Maybe tell us a little bit about How you got into this as a kid?
Were you studying pictures of food and wishing you could teach a computer to break the recipes down.
How'd you get into the field and what did you do at MIT and what are you up to now at Cal?
So I'll actually start with how I got into the field.
So I actually came from a software engineering human computer interaction background.
And then I realized that what I really enjoyed was the sort of the mathematics and fun applications that went into computer vision specifically.
So I was like, all right, I'm going to learn computer vision and turn out the state of VR is deep learning.
So I will end up learning deep learning.
So I studied that over the summer and ended up in Professor Taraba's lab working on a project generally designed to create a structured, unstructured, some representation of food that could be used for things like predicting health and just in general, like basically improving people's health.
So like the whole idea of pick to recipe is simply a side effect or like rather a nifty application that comes with developing these representations of recipes that can be used more generally.
The goal was always to create a representation of a recipe that could be used for things like improving health and It just happens to be the case that going from image to recipe that was likely associated to that image even if the application that you can do using these representations, right?
So is and I'm inferring a little here, but is a next step something like? a calorie counter or more sophisticated snap a picture of what you just had for lunch or maybe what you're considering to have for lunch and and you'll get back sort of a breakdown of the nutritional content, something like that?
That's exactly the goal. So when I said that, perhaps you'd use the application in a more realistic context someday.
That's exactly what I meant. So like right now there's a, um, Another demo I prepared that goes from list of ingredients plus image to macronutrient break. which I think is a super useful thing because I actually made it for when I was at a company over the summer. doing an internship and they didn't have any nutrition stats for their menus, but they had all the ingredients.
So I was like, all right, I have the ingredients, I have the image, perhaps I can turn it into a rough nitrogen breakdown.
So that worked out pretty well. it wasn't quite the level where like a real person could use it because no one's going to sit down in a restaurant and enter like broccoli or breadcrumbs, all that kind of stuff.
So the next step is to go from image directly to ingredients, feed it into this other system, and then get out And a complete breakdown of what you're actually about to eat.
What are some of the complexities involved with that?
A few come to mind, but I imagine I'm just scratching the surface.
Oh yeah, it's incredibly difficult. You already mentioned one of the main challenges, which is obscured ingredients that you just can't see.
But then there's also what I call the French cuisine problem, in which if you're looking at an image, there's no way of telling whether it actually just contains two cups of butter.
Right. Usually you find those things out after the fact, not just looking at the image.
Right, exactly. And then, of course, there's also the basic problems of lighting condition, camera quality, and all those kinds of things.
I think that the problem is difficult, but I'm hopeful that there's a solution.
I was going to ask, is there... the potential for a deep learning driven solution to the French cuisine problem.
Well, perhaps. Maybe if you were looking at a croissant, for instance, you can see, oh, maybe the exterior of this croissant is quite... shiny, perhaps contains a larger than normal amount of butter, or maybe had like an egg white wash which makes it shiny.
That's one way to do it, but a human obviously would just go up to it like take a piece out, smell it, taste it, and be like, wow, this is a really rich croissant.
So another avenue of research is actually to you – well, actually, one thing that we were exploring was using infrared – spectroscopy, I think it was called, but it basically uses an infrared beam to analyze the molecular content of the food item.
It's not quite clear how you get that on your phone, but I think maybe someday phones might actually have that.
For now, I think images are a great place to start.
What sort of accuracy are you achieving And I'm wondering both with the system as it works now, but getting into some of the higher end techniques you just mentioned, not enough. but just in the lab, how close are we to accurately being able to turn any photo into a recipe or break down the molecular structure of a croissant?
I can probably tell you right now that that latter part is definitely a big issue.
That's a huge challenge. and really only possible in the context of a lab, or unless you have one of those fancy handheld devices.
How far are we from... being able to do that using an image So right now the system evaluated on the test that basically you give it images that it's never seen before and the idea is to figure out what recipes it like predict the correct recipe so like these are all paired right so it can get the exact recipe about a quarter of the time and then if you allow it to say like give me like a 10 most probable recipes that are associated with this image The correct recipe will be in the top 10 65% of the time.
Now that we have more images, we're actually seeing even better results But we still have yet to publish those.
Sure. What if anything has surprised you? in your work on food recognition, whether from the food side or perhaps more relevantly from the deep learning and AI side.
So I guess let's start with the what's the prize on the food side.
A lot of it is like how much variety there is in just food recipes and images of like thereof.
Like you might not think that there are hundreds of ways to repair lasagna, like lasagna is lasagna, but no, like there are hundreds, like hundreds of variations.
And it's another funny thing is like, you're looking through the, you're trying to like examine the data.
It's like, oh man, why isn't the model working?
And then you just see like, a recipe for like soap I'm like what is this and like then you're looking through the ingredients and someone meant to put like a quarter pound, but it turns into like 14 pounds and it's like, all right,
This is ridiculous. But nonetheless, the model is able to learn through all this and do something reasonable, which is actually surprising in itself.
So in terms of what the model is actually learning, I think is more interesting.
So recall that the A whole idea of this was to generate representations of recipes and images.
Not necessarily to go from image to recipe, which is still interesting, I guess.
But these representations... If you look at them, basically what the model has learned basically the preparation associated with an image, like how it can look at an image and figure out how it's prepared.
So like if you take these representations and try visualizing them what you'll see is like along one axis it has like cooked foods on the other axis it has like things like salads, like uncooked things.
And you have like a little cluster over here.
It says like baked. And then over here you have like And then inside of that you have like cookies, muffins, pies.
So it's like taking these abstract concepts like baked, like cookie, casserole, salad, sandwich, and has... learned to like put them like organize them into like this map and that it can then use to like look into and then pull out another, like the nearest like recipe or image.
So given what the model is able to do or starting to show signs of being able to do in terms of what you just said, taking these. these disparate elements that make up an image and then form different conceptual models for what's a plate of five separate cookies versus a casserole that's all baked together.
What's next for the work that you're doing?
Are you going to further the art of food recognition or are there applications for what you're learning to apply to other types of data processing?
So I'm definitely interested in actually making that application that goes from image straight to nutritional information which is definitely an area of research in itself there's no human expert saying like this recipe is prepared this way but you can use the data itself to train a model to understand the data.
So everyone's all about training supervised models, but in this way you can use the data to self-supervise.
Honestly, I think that unsupervised and semi-supervised learning is the way of the future, and I think that a lot of people agree with that. just because getting supervised labels is just so expensive.
I think that this is probably the right approach going forward, or at least a reasonable approach.
So I understand that on your own GitHub repository, you've got something of a customized meal that you've concocted for yourself.
Can you tell the audience a little bit about that?
Yeah, so that's just another one of the ways that sort of my interest in...
I guess both food and optimization has expressed itself Basically, there's this trend towards improving your efficiency by drinking these meal beverages.
I don't agree with some of the more popular ones on the market because they're not made with real food, real ingredients.
So quite related to this project, I went and used the available data sets, particularly the USDA nutrient database, In combination with linear programs to develop a meal beverage for myself that consists of four ingredients, soybeans, chia seeds, cocoa powder, and peanut butter that turns out if you drink three of these per day and eat some veggies, it satisfies all of your nutrition.
This is why we have these conversations to uncover the gem late in the podcast.
So tell me a little bit more about this.
You found you took the nutritional data from the USDA.
And you ran it through, did you write a custom algorithm for this?
No custom algorithm, but I use what's known as linear programming, which basically says given a bunch of constraints like greater than, less than, or equal to, and optimization objective, maximize the objective subject to the constraints.
And you wound up with, and I... I don't mean to belittle it, but I keep wanting to call it a super smoothie in my head.
You wound up with this drink, chia seeds, peanut butter, soybeans, and cocoa powder.
Did you optimize for taste? So there's a little bit of human in the loop twiddling that had to go into the cost function. but i think it's actually kind of interesting from like an optimization perspective when we like think more uh generally in terms of like robo chefs but Yeah, so I had to like tweak the constraints a little bit.
For instance, you have to add like a cost constraint.
Otherwise, you end up with basically cocoa powder, peanut butter water. which isn't too tasty.
I can only imagine. But the formal objective is technically fat because fat is tasty.
I mean, is this your meal? Do you have three of them a day?
You bet I do. I'm going to have one in about an hour and a half.
So three of these, and then you said you supplement with some veggies.
Yeah, the veggies are essential. And how long have you been doing this?
Well, this is what helped me power through my work at CSAIL in developing this application.
So I was actually really just living vicariously through all of the people's recipes. and all the delicious food that they're making.
So this has been a couple of years. Yeah, it has actually.
The project took a while, but I think just in general, it's important to take a step back every once in a while and then you end up figuring out new techniques, new concepts, and you come back, loop around, and you have new insight on the problem.
I've found that that has actually been the case numerous times in this project.
And I think it'll continue happening as we move forward towards the final iterations of food recognition.
Fantastic. I, for one, cannot wait to see where Pizza Recipe goes.
And I think – I don't know if I can commit to three a day. but i'm definitely going to check out uh your optimized beverage recipe that may fascinate my own brain more than my own stomach but you know i'm trying to find a balance Nick, if you would spell out the URL for people so they can see Pick to Recipe in action for themselves.
So it is im2recipe.com. CSAIL.MIT.EDU slash demo great and uh is your github with with the beverage recipe is that available also Yeah, actually, so that's github.com slash nhines, N-H-Y-N-E-S, slash pbean, P-B-E-A-N.
It's a kind of play on peanut butter and beans.
And in fact, the original recipe called for navy beans which I can substitute for soy, navy beans, or pea beans, so it's kind of funny, I guess.
Gotcha. Well, Nick Hines, this has been a fascinating conversation and again, a timely one as we lead up to.
Oh, the season of eating, let's call it.
At least up here, we're all in the Bay Area and Lots of foodies around here for sure.
But this has been great. I can't wait to play some more with the system myself.
And best of luck to you as you take Pick to Recipe. to its no doubt bright future.
Thanks so much. You know, it's really been a pleasure talking with you.
