Hello and welcome to the NVIDIA AI podcast.
I'm your host Noah Kravitz. Plastics. Convenient, cheap, and tough.
Plastics ranging from sandwich bags to water bottles are popular the world over.
But some 50% of all plastics worldwide are used once and then thrown away, according to PlasticOceans.org.
That means that large amounts of plastic waste get swept by wind and rains into our urban waterways, where they flow into creeks and rivers and the like, eventually polluting our oceans.
It's a big problem. But our guest today is using, yes sir, convolutional neural networks to help clean up our act.
Lorenzo Flores is a web application developer in the Environmental Informatics program at the San Francisco Estuary Institute and the Aquatic Science Center. which is actually located in Richmond, California, shout out to the East Bay.
His GTC Digital Talk, Accelerating Trash Detection for the SF Estuary Institute, With GPU-enabled convolutional neural networks by Oracle Cloud Infrastructure and Kinetica, presents some of the work he's doing to get trash out of our water.
And he's here today to tell us about it.
Lorenzo, thanks for making the time to join the Nvidia AI podcast.
Thanks for having me, Noah. So tell us about trash detection in urban waterways.
Kind of set the stage for us what the problem is and how you're going about trying to solve it.
Yeah, so we have a lot of trash making their way into the Bay Area area. our oceans, as you mentioned, rightfully so, in the introduction there.
Essentially, we at the moment haven't had a very good of assessing the amount of trash going into our wetlands and waterways.
And that's sort of the first pass towards getting into the oceans. here at the San Francisco Estuary Institute, we kind of attack contamination in the bay for multiple levels.
So we actually have a lot of research about microplastics that's gotten a lot of notoriety recently and gotten a lot of press coverage.
What's a microplastic? Microplastics are very, very small-scale particles of plastic that you just can't see.
Unfortunately, we are still trying to understand the impact that these small particles of plastic have on the cellular level.
So like animals eat these and then they get into to their systems and we're still not exactly sure what kind of issues will come about from that.
And so we are finding more and more microplastics everywhere.
And in fact, actually recently we found that tires are a big contributor to microplastics.
In the environment, plastics start to break down and turn into these little molecules and Next thing you know, we're able to, we have to use microscopes to find them.
The image of turtles with six pack rings around their neck, getting their heads caught.
I remember that was, Kind of the first thing that raised awareness for me and my peers around plastics in the ocean being a problem.
But it's microplastics. It's all kinds of stuff.
Yeah, actually, that's a really good point.
For the longest time, especially we can see that trash.
You can see how that plastic is impacting the environment.
And I think that's what's been fantastic about the microplastics research we've been doing.
You can't see that. And it's harder to get a sense of how it's impacting. trying to get a sense of, Things we can't see is a big part of what we do at SFEI.
So microplastics, can't see them, but they're everywhere.
What's next? Yeah, so we can't see them.
They're everywhere. We're noticing it in our labs.
And so there's one... big sort of glaring blind spot that we've had, which is trash transport.
So, you know, we have our landfills, we have pathways for trash where ideally we throw it in the bin, it gets picked up, it gets taken to a site where it's able to sit or maybe we have recycling where it gets sorted, but there's a lot of opportunity for trash to escape those encatchment routes. and slowly make their way into our waterways, into our environment.
And sort of the way that the transport mechanisms work, a lot of times these trash will find its way into rivers and creek beds and marshlands.
And that's sort of like the next step towards getting into the bay or the ocean.
Okay. And so once it's out there, then it even starts to decompose even further and becomes really enmeshed within the environment.
And so that's where this work comes in. This might be an area where I might have to go back and look it up.
There has been sort of a state decision to like attack this problem full hand and basically compare survey methodologies because we just – haven't really been going out to these sites and assessing how much trash there is.
The first step for the grant that this work is based on is really about comparing survey methodologies.
And these surveys are incredibly labor intensive. and costly.
You have to get multiple people out to a site on the day.
You have to coordinate with local officials.
We do for drone imagery. We'll get into that in a bit.
Either way, you have to really coordinate a lot of people and go out to a site and you basically count trash.
I've gone on a couple of these where we go out to a creek bed And part of this work was also digitizing the survey, but filling out a form that says like, I'm noticing X amount of plastic bottles.
X amount of, there's the suitcase, there's all sorts of things we find in these waterways.
And we gather all those and we basically measure the volume of how much trash we found And that's sort of the first step towards just getting a sense, like a bit of a signal for how much trash is just kind of escaping our normal and catchment methods and going into these waterways.
And so that's where part of this grant, we decided, hey, let's see if we can come up with something acknowledging that this is very labor-intensive there's sort of this sense of, well, this isn't going to scale very well.
There's going to come a point where You just can't afford to keep bringing people out to the field and gathering trash, looking at it, assessing it.
Frequency is going to be low. The geographic scale is going to be low.
In order to scale it, We really need to leverage technology, you know?
So, and that's actually one of the things that SFI, we pride ourselves on, which is marrying technology and science.
We have a lot of mechanistic models that scientists use that have been developed over the last like 20, 30 years.
Some of them are written in Fortran. and these are models that they just don't scale well.
One of the things we hope to eventually do is to augment or replace Some of these with neural networks or other machine learning algorithms that can scale well because there's so much potential there.
We try to use technology to do good science.
And so a task within this grant was let's just see if we can to take some drone imagery and just even try to identify trash within the drone imagery using machine learning of some sort.
We haven't even settled on what kind of algorithm we would want to use.
Now, is drone imagery something that you were previously doing in other parts of SFEI's work or brand new idea?
So I have a coworker, Pete Kahanan, who has been, basically he got his drone pilot license a few years ago.
It was an anticipation of the fact that for the geospatial aspect of the work we do, sometimes we just need to go get data and taking a drone out is the number one way to do it.
Sure. I think this work in particular, though, has sort of been the biggest challenge phase of drone image capture that we've done at SFEI.
So I think this was like our first big project to leverage drone imagery.
Actually, that was a big part of the first, I think we started this work, in 2018 and a lot of that first year was really just capturing the images And basically identifying how we want to approach the algorithm and generating a data set.
We hadn't really considered that part of it.
It's just, hey, you want to train an algorithm or train a machine learning algorithm?
You need data, you need annotated data. I know there's a lot of ways to tackle that, but being a non-profit, we have very limited resources.
Oftentimes, you have deliverables, you have things we have to get done, and then there's a little bit of room for risk in trying new things.
So that's sort of how we got started with this sort of like machine learning component of the trash monitoring stuff.
You mentioned that when you started, you didn't have a dataset, but even so, you're like, we're not sure what algorithm or approach we'll use, but we know that machine learning is a tool that we can use to try to make sense of this.
So how'd you get there and what'd you wind up trying?
Yeah, so I think right at the beginning, it felt like a bit of a daunting task because trash is almost anything and everything.
The big challenge with trash is we're trying to detect something that shouldn't be there.
Yeah, right. And it could take a variety of forms, like basically anything can be trash.
So in a way- it's more like filtering out the environment and sort of trying to identify the things that just aren't part of the environment.
So I think with that in mind, It was pretty quick that I identified, you know, hey, I think we're going to want to use an old network.
On top of the fact that we have this huge problem of trash being almost anything, we also, dealing with limited resources, quickly realized like, We're going to be hard-pressed to identify some sort of pre-processing step or way of taking images and doing some sort of like gradient analysis.
Or I know that there's literature for ways of like, identifying plants and on top of that, if we have different modalities, for example, we're using just colored RGB-based images from drones.
It'd be nice if we had infrared imagery or other modalities But that doesn't scale as well, and it's a lot more costly.
One of the incentives here was to make the cost low.
So... As a result, the idea was it can't be too labor intensive as far as the pre-processing.
And we know that neural networks over the last few years have just demonstrating the ability to find those latent patterns that sometimes our brains can capture very easily, but other times we don't see as easily.
I think we decided, okay, it's probably going to be easiest to focus on creating our dataset and then leveraging a neural network just to see if it's able to identify trash, you know, just, just even if it's, are we, are we just this pie in the sky?
Is this possible? Like. Like if I just do a little bit of training, can I get a little bit of return?
So that's kind of what the impetus was for using neural networks from the start.
And so how far along are you in the process?
Do you have any? I was going to ask. And so how did it do?
Can your network detect trash? I haven't listened to your talk yet.
How far along are you? We're still in a fairly beginning stage.
Actually, just right now, I'm doing parameter tuning.
So I'm trying to get the accuracy of the model up. we can load up novel drone imagery and it's able to identify trash, The problem is that there are certain blind spots.
So I think one issue that's coming up is a rock looks a lot like a plastic bag.
So we need to feed it a lot more data that's intentionally placed in environments that we expect. and sort of pad out more of the examples of plastic and having rocks as the non-target or non-environment that we don't want it to classify.
And the drone is obviously above the water, I would assume, in the air.
But how much of the imagery is actually kind of underwater imagery that you're looking at things beneath the surface?
Or are these things at surface level? These are all from drones.
It's all aerial-based imagery. And then is it the trash or the rock that looks like a piece of plastic, of a plastic bag?
Are these things at surface level of the water or are they underwater and so you're also dealing with the way that light changes as you go Taking a picture that starts in the air but moves through the water.
I wish – no, we're not at that kind of a level yet.
We're just – so – when we do these drone flights it's there's a lot of regulation for for flying drones you can actually only legally do so much.
And I think a lot of it is just, we have the lowest altitude that we can achieve.
We have, we flush out a survey. perimeter and the drone basically just takes that bulk of imagery and then we come back and we basically get the imagery. and start you know either well at this point cropping it and then feeding it into a prediction doing some inferencing on it just to get a sense of how good the model is doing with novel data.
But yeah, it'd be nice if we could go lower and at some point go into the waterways.
These are certainly things that, We discuss and are sort of keeping an eye towards the future, particularly when you think of Internet of Things, like NVIDIA has the Jetson a development board.
That sort of technology opens up a lot of potential opportunity for real-time monitoring.
That's so... The whole point of these drone flights is that we're increasing that frequency of surveying an area.
So it'd be really interesting to... sort of get even higher frequency and even closer to the ground level.
Our guest today is Lorenzo Flores. Lorenzo is a web application developer in the Environmental Informatics Program at SFEI, the San Francisco Estuary Institute and the Aquatic Science Center.
We've been talking a little bit about the talk that Lorenzo gave at GTC Digital, which is online.
You can check it out, take a listen for yourself.
It's called Accelerating Trash Detection for the SF Estuary Institute with GPU-enabled convolutional neural networks by Oracle Cloud Infrastructure and Kinetica.
Lorenzo, let's take a step back for a second, and we chatted just a little bit before we hit record here, but I'm curious about your background.
I'm not going to ask a stranger his age, but you seem, you know, on the younger side of things, at least compared to me.
So I'm wondering... How you got into the work that you're doing with environmental informatics and studying the waterways and in particular in the SF Bay Area, but also how you got into the computer science, web app, data science side of things, and if those two were always interests or if at some point one made its way into the other one.
Yeah. I graduated from Berkeley in 2006 with a physics degree.
I studied physics and worked in a condensed matter physics lab, and that's where I basically fell in love with programming.
And so a little while after it went, worked for a bit in the field area and for a medical device company actually, and after that decided, hey, I need to step up my game and went to San Francisco State for a master's in computer science and that's actually where I really started to fall in love with machine learning and data science.
I think because having that physics background gave me the mathematical toolset to at least sort of approach some of these algorithms.
And at the same rate, just being a lot more of a computer programming has always been a passion of mine.
Sort of being able to marry those two was wonderful.
And so... I studied protein functionality prediction actually as my master's thesis.
We were looking at binding sites for proteins.
And that was a lot of fun. I really enjoyed that experience.
It was really a nice meshing of science and computer science.
And I got a lot of experience running experiments on AWS and sort of just on the cloud infrastructure, which is helping right now with Oracle's infrastructure.
During that time, though, I studied random forests and support vector machine, machine learning algorithms.
And so neural networks was actually quite new to me as far as this work.
And It's worth noting that while I was a graduate student, I did part-time IT work for the Berkeley Ecology Center here in California.
And so I think that's where I started to develop a deep passion for environmentalism.
And Berkeley Ecology Center is more about sustainable living and sort of the social justice side of things.
And so they run the curbside recycling program for Berkeley.
And in 2015 is when I really wanted to focus more on programming part of things.
SFI had this opening for the web application developer role, which is very broad.
Basically, just we have a lot of environmental projects that we're dealing with and we need to programming chops for a wide variety of things.
So kind of typical of a nonprofit, not a lot of people wear a lot of hats.
As a programmer, I wear a lot of hats. So How big is the, sorry to interrupt, but are you the only programmer app developer or is there a team?
Oh, yeah. No, we have a team, and that's actually what I think is fascinating about SFEI.
We're a nonprofit, but we have... I think so within environmental informatics, probably like 10 to 15 technology types, technologists that, do a wide variety of roles.
So we have some people that focus just on getting data from our sensors into a database.
Some people that deal with just the IT infrastructure of the organization, hats off to them lately, they've been doing double duty.
And then I'm part of the app dev group. And so the app dev group is three people and we deal with creating like user interfaces and some gis analysis for other parts of the organization so we kind of do support level work for other parts of the organization as well as just projects that are just a little bit more technically driven.
Yeah. So kind of turning back to the trash detection project, You mentioned, I think it's been about two years you started working on it in 2018?
Yeah, but it's worth noting, though, that it's really just a fraction of the work that I do.
It'd be nice if I could focus on this. to stuff full time, but we just, we have multiple grants and multiple projects that we're constantly juggling.
So I'd say over the last two years, it's been about a good 20 to 30% of my time, depending on when.
And a good chunk of that time was actually just letting the model train on my desktop.
It took a while to get the resources just to do proper training on a GPU.
Once we got that going, I just... had to let it train and Once again, given limited resources, I kind of decided to go with the more conservative approach and take the training schedule to a very small learning rate. and very small batch size, just really dial things down so that hopefully I can get the best results with just a time investment since we couldn't really just do a lot of parameter tuning, kind of what I'm doing now, actually.
So actually, along those lines, I'm going to put you on the spot.
Any other tips you can offer for folks trying to do deep learning, machine learning on a resource budget.
Like you said, you're training a model on your laptop, small batch size, Anything else you can offer up to folks who might be in a similar situation?
I think those are the two that I've learned so far, which is just the small batch size and low learning rate.
I think beyond that, you're going to be dependent on the architecture of your network.
We're going with the Object Detection API, so we got a pre-trained model from TensorFlow's Model Zoo and have leveraged transfer learning. in order to not retrain from scratch.
That's where it'd be nice to have a model developed with sort of aerial-based imagery.
This model was trained off of Cocoa Image Library. which I don't know what the potential impact of that will be, but it'd be nice to compare with something that's more aerial based.
But yeah, ultimately, I think. Those are the two things that I can advise.
To be honest, Those are things that I just really have been able to learn within the last year of just kind of experimenting and then also attending conferences. workshops that kind of deep dive into these neural networks.
Right. What's the remaining, I don't know if you know, but the remaining scope of this project, whether it's determined by the grant or just kind of how do you see, how much more is there to go on it?
And what I'm sort of leading to is wondering, You know, what are you hoping to to achieve?
Is there a goal, objective or is it more of kind of like you said, you know, this is sort of the first step towards seeing.
Is this a technique and a way of gathering data and starting to process it that not only might work, but also might be scalable? you know, on a nonprofit's budget?
Yeah, I think our big hope is to get somewhere to the 70 to 80% accuracy range.
And I think that's just because for us, it's okay if we're not identifying all the trash.
We don't need to be 100% accurate. So it is kind of an interesting space where there's a lot of room for just not being on the spot.
This is exactly the amount of trash here and basically having it one-to-one with the ground truth dataset that we have.
It's okay if there's some false positives or false negatives.
And so with that in mind, we think it is realistic that we'll be able to get to that level. with the proper parameter tuning in time.
And that's where, once again, being able to leverage these computational resources that we didn't have before, I'm hopeful is going to get us there.
Ideally, what we can do is have a model that then we have available for either us to support municipalities to leverage to do their own work. sort of trash assessments or for other people to take and do their own learning from.
Within SFEI, actually sort of We're kind of coming to the end of this grant cycle for this project.
As this wraps up though, the next step will be to We're going to be kind of retuning and tooling to identify tobacco, particularly cigarette butts.
So there's a big incentive to quantify the amount of cigarette butts that are out there.
Ideally, tobacco-related waste in general But it's actually, that's a really hard problem.
There's a wide variety of of products out there that look like a lot of other things.
But cigarette butts are pretty unique. And so we're sort of starting to focus with that.
So we're going to plan to take this model and cull some of those layers and then leverage transfer learning from there and hopefully continue this work, but just with that lens towards trash.
Also, I'm not necessarily part of this work, but I believe that we are doing some consultations with Denmark. to develop their own sort of methodology and algorithm for identifying trash in their waterways.
I think a big part of this work has been just the simple process of establishing a methodology where you do some sort of drone capture and then you take that imagery and you annotate it, and then you can run it into an algorithm.
These steps are things that I think most of the audience probably knows this is kind of the process we take.
But when you're talking to... policymakers and government officials you know this this stuff is they're still trying to they're still starting to get in tune with what this process looks like.
So I think just sort of guiding people along that.
That route is another aspect of what we hope to do with this work.
That's fantastic. It's important work. It goes without saying, but I wish you all the best of it.
I'm all for fewer plastic bags and cigarette butts. in the water for what that's worth so uh lorenzo flores for people who would like to find out more about the work you're doing on this project the other stuff that sfei is doing Are there places online where you can direct The number one website for this work would be trashmonitoring.org.
It has basically all the information about this project.
Since we are coming toward the ends of the cycle, I think soon we will have more information about the actual machine learning component and that's where the idea will be to have the model available for people to leverage.
In the future, beyond that, SFEI.org is for San Francisco Estuary Institute.
And that just covers the wide variety of work that we do as an institute.
And as mentioned at the top, your GTC Digital Talk will be available. for folks to check out online.
It may be by the time you're listening to this.
And you can search for Lorenzo Flores, F-L-O-R-E-S, on the NVIDIA GTC Digital site.
But I think it's the only accelerating trash detection talk at this year's conference.
So you should be able to find it. All right.
Well, Lorenzo, again, thank you for taking the time to come on the podcast.
Really again, best of luck with all the work you're doing.
I can't wait to see the results from this round and hopefully the project will be extended.
And you'll be able to share your work with your colleagues in Denmark and elsewhere, as you said.
But thanks again for the time. Thank you.
Thank you. Thank you.
