Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. From finding the answers to trivia questions to hunting down the perfect gift for any occasion, the utility of the internet almost can't be understated.
If you want to know something, you can probably find it out online.
But the very thing that makes the internet so useful to so many people, the vast quantity of information that's out there, can also make going online frustrating.
There's so much available that the sheer volume of choices can be overwhelming, but that's where recommender systems come in.
Our guest today is here to tell us all about recommender systems, and why they're so vital to each of us having uniquely useful online experiences.
Evan Aldridge is Senior Manager for the Merlin team at NVIDIA, and he's here to talk to us about recommender systems, what they are, why they're so great, and how they're shaping the future of how we all access information.
Even thanks so much for taking the time to join the podcast.
Thanks for having me on, Noah. So while I was doing that introduction, I was thinking to myself, I'm sure I've got some of the nuances wrong here and I know there's a ton I don't know about recommender systems. but we've got an expert in the house.
So let's jump right into it. Why don't you tell the audience what a recommender system is, and then we can get into Merlin and what NVIDIA is doing. in the field?
Absolutely. So as you pointed out just now, recommender systems are ubiquitous, they're a huge part of the internet and of most mobile apps. and really most places of interaction that a person has with their computer.
A recommender system really, at its heart is a system for taking the vast amount of options that there are available in the world and kind of boiling them down to something that's relevant to the user in that in that time or in that context and that's a really high-level broad task and a really significant challenge, both from the engineering side, which I hope we'll get a chance to get into in terms of the systems and the sort of the models that need to be built.
But also from a conceptual side, you know, really recommender systems in my mind are one of the most complex and significant machine learning challenges of our day, you're trying to represent what a real live human person is interested in, in any given moment, given some narrow context.
And that's not an easy thing to do, especially when that person may or may not know what they want in particular.
And that's one of the interesting kind of about recommender systems is typically they're being engaged with by somebody who isn't necessarily sure what it is that they want.
And so, you know, it's a very, very interesting space and one I'm quite passionate about and excited to talk with you about.
So let's start with that broad conceptual view first, and absolutely we'll get into the engineering nuts and bolts, so to speak, or you will.
I'll listen and nod and maybe ask a question or two, trying to keep up.
But broadly speaking, how would you define a recommender system?
What is it? Yeah. So a recommender system is a machine learning algorithm that filters content so you can query the recommender system, to narrow down the possible options within a particular context.
The classic view that most people or the classic experience most people have with recommender systems is online. where you're you're browsing for a particular item and you're seeing other items that are potentially, you know, useful in that same context or similar content and sites like Netflix and Spotify and some of these other kind of content distributors.
You're seeing content based on the content that you viewed in the past and The recommender system's role is to look at that experience and try and build a summary of your interests as a user of the site or the content and try and come up with the next relevant thing to show you.
And so in these different examples, you talked about it and a bunch more came to my mind and I'm sure, you know, to listeners' minds, everybody's experienced one or several.
Do they generally operate the same way across different, you know, different shopping sites or different content sites that I might go to? or are there different ways of approaching the problem?
There are patterns to the problem, but it's one of the more fragmented industries.
I think in part, Like if you look at things like computer vision or natural language processing, there are open source data sets that have allowed for a lot of significant advancements in the field and allowed for a standardization and a benchmarking and those fields have become pretty standardized because of that.
And in the recommender system space, the interaction data that your users are generating is in part the core value of your company.
And so most companies are reticent to reveal that data.
So there aren't a lot of great public recommender system data sets out there.
And that sort of What that means is that there's different solutions being applied in many different places and very different senses of scale from the very small to the massively massive when you're talking about like a small company sending out a newsletter versus Facebook's live recommendation feed, which has hundreds of millions of items being recommended.
So you're working for NVIDIA, obviously, and NVIDIA is not in the business of running a massive online shopping site or social media network where they're using an algorithm to recommend content, push it into your feed, those things you mentioned.
So what are you doing at NVIDIA? How did NVIDIA get into the business of recommender systems?
And what's the role that you and the teams you work with are doing right now?
Yeah, absolutely. I mean, I think for why NVIDIA is interested in the business of recommender systems, to quote Jensen, So recommender systems are the most important algorithm on the Internet.
They drive a lot of the financial and compute decisions that are being made. and really for NVIDIA, it's a very interesting machine learning workload that I think previously has been more applicable to CPU or has been done more on CPU,
And sort of that's where I entered into the picture is, you know, having come from a strong Rexis background, I joined NVIDIA to work at the nexus between Rapids and the data science ecosystem and the deep learning frameworks and deep learning ecosystem.
And the first thing I built here was this data loader that helped kind of improve the performance of tabular datasets, but ultimately recommender system datasets in deep learning training.
In that exploration, what I came to realize and what we began to explore was that there's a lot of a lot of complexity towards doing or building recommender systems correctly.
I think if you think about when the tools were built for most machine learning systems like these deep learning frameworks, And even the hardware when it was built, all of these were kind of built in the era of of computer vision, right?
And so, you know, the solutions being developed were targeting those kinds of workloads.
What's happened in the last five, six, seven years is both the hardware has changed dramatically.
And in addition to that, we've had to improve the software and make changes to the software and improve that dramatically. to try and help target recommender system workloads better.
We've gotten to a place where now recommender systems on GPU make a ton of sense and I think that that there's not many people who are trying to run large NLP models on a CPU.
There's not many people trying to run large computer vision models on a CPU.
And for Rexis, there's a lot of people kind of still focused on the CPU-based solutions.
And so... There's a strong motivation for us to get this right, both because we have a vested interest in selling GPUs, obviously, but beyond that, there's a similar degree of acceleration that's possible that led to the revolutions that have happened in computer vision and NLP.
When things can happen 10 times faster, then suddenly you're able to do much more exploration and much more kind of you know diving into the problem space and really you know the field begins to take off in a way that it hasn't before and that's something that our our team is really focused on is how can we enable these teams to develop recommender systems much more quickly and efficiently, both from the compute time perspective, and that's been really important in making sure that, you know, that you can develop features and train models and deploy to production really quickly and easily.
But the easily part of it is actually a very high priority for our team.
We're trying to build tools that are designed with MLOps in mind, that are designed with how the system gets deployed in the production of the design with how customers are going to be using these tools. to make it much, much easier for people who are using the tools to build and deploy these systems.
Part of that stems from viewing the space and seeing the degree of fragmentation in it.
There's a lot of companies out there at the largest level that are doing deep learning based recommenders and they're doing them on GPU and they're scaling it.
But they're doing that with 150, 200, 500 engineers supporting that effort. with a strong ML infrastructure and not every company has that, most companies don't.
And so what we're trying to do is, develop tools that make it easier and more straightforward that enable much, much more smaller companies to begin to explore these topics right the way down the stack, not just deep learning, but also the more standard traditional methods of recommender system models as well, making it just easier in general for customers or companies to go on their journey of developing a recommender system.
I think it's a wide open space and not a lot of standardization happening within the field.
Because all of these companies have had to figure it out on their own, nothing existed.
So we're working to try and build a framework that supports this ecosystem.
When a company or when you are working on a recommender system, are the types of Data that that are involved in sort of what the users looking for if I'm looking for.
I'm looking to buy a piece of clothing as opposed to a TV show to watch, as opposed to...
I don't even know some investment advice or even finding the right investment broker.
Is the problem basically the same on your end, on the design and computer science end, if you will, or are they are there different problems based on the type of, and I don't know if I'm using the word data correctly here, but the type of thing that I'm hoping you'll help recommend to me.
That's a really good point, Noah. And I think like it gets at the heart of why there's fragmentation and why there aren't these sort of these larger models that you can, you know, like, I guess to use the technical term that's used in the NLP space or in the computer vision space, transfer learning, which is taking a pre-trained model, that's been trained on a lot of data and then transferring it to your problem space doesn't happen within recommender systems.
That's because you can't take that recommender system that was trained to recommend content on Netflix and apply it to anything but the content on Netflix.
It's that specific. It's that specific.
It's learned the person's personal preferences within the context of and it's learned the content that's available on the site and how those two interact, right?
So you can't. That being said, the workflows and the patterns that are used, We've developed some design patterns.
One of the people on my team, Carl Higley, has this great kind of conceptualization of the four stage recommender system, right?
This is one of the things that we struggle with in the space and in the field is if you look up recommender system, more often than not what you're going to find is a recommender model example.
That's historical in the sense. But really, most production live recommender systems consist of four stages.
The first stage is a retrieval stage, and that stage is basically a simpler model that's looking at the millions, hundreds of millions, billions of items that are potentially available in the catalog and narrowing them down to a few thousand tens of thousands.
That phase is your initial phase to gather content that's going to be, in theory, relevant to the user.
The second phase is a filtering phase, and that filtering phase basically says, okay, let's take the content that we know is not going to be legitimate or relevant to the user and get rid of it.
Content that violates country restrictions, content that violates age restrictions, removing things that the user's already seen before, these kinds of things.
Okay. Those two stages all happened before we ever hit the ranking model, which is the model that determines the score or the fit for that user to that item. stage can include a lot of components including feature engineering and feature transforms and potentially accessing feature stores. and pulling in all the information related to that content.
And that can be, most typically tabular information, so categorical and continuous variables that describe the information.
But it can also include potentially text that's been embedded or images that have been embedded, and that's where a lot of the complexity and I think the future of the field lies is that multimodal, like how do we aggregate information from all these different sources?
Right. Then I guess ranking is the old term for it and probably ordering is a better terminology.
Then the third stage of the ranking model passes to a stage that does the reordering. you know there are many reasons why you would potentially want to reorder your search results you can imagine If the user is 70 percent into one type of content and 30 percent into another, the model may be recommending almost entirely everything from that first category.
But that's not... a great experience and so you may want to kind of maintain that 70-30 split and try and promote some of the other content.
Or the results may be so close that you want to do something along the lines of a shuffle that provides the content in a way that sort of different or more meaningful.
Right. Most of the time I want to watch comedies, but every once in a while I want to watch a sci-fi pseudo documentary.
So you got to filter those up every so often, even though they're only in that 30%.
Exactly. That begs a question that's been in the back of my head while I'm listening to you describe this, which is how do you judge the effectiveness of a recommender system?
Is it just on a, you know, sort of whatever the customer being the company that's building it, whatever they deem is most important or is it user feedback or how does it work?
I mean, there's a wide variety of things that are used to determine and compare the effectiveness. both offline when you're kind of developing the model and trying to evaluate its performance.
And then of course, online when you're running the model, production and the model serving to the customer.
There's a lot of metrics in that space. I don't think there's any one clear answer of this is what you need to measure and you're good to go.
I think a lot of different companies, at the heart of it, you're trying to measure user engagement generally long-term, but that's a very lagging signal and so you need to tie it to metrics that are much more immediate. like interaction and clicks on items and things like that.
I was just thinking like, well, one way to judge it is, how many Amazon purchases do I wind up returning?
Which is, I think more on me than the recommender system.
Yeah, and I think many companies will track those and forms of user engagement can be both positive and negative. and those cost the company a lot, and so they're probably being weighed.
It's very difficult to trace those all the way back to an individual recommendation. at the start.
And so it becomes one of the most interesting and complex challenges.
Like at its heart, a recommender system is trying to model your preference, right?
Right. And the preference of a human being in the world, in, you know, in, like their myriad of contexts, right?
And that context can change, you know, in ways that the model has no idea about, right?
Like on this podcast, you could tell me about an interesting book that you've read, and that's going to lead me to kind of like look it up on Amazon and dig through it and potentially order it.
That recommendation from you as a human kind of like, you're using your context that you have about me and about our conversation and about a bunch of other things.
And the system doesn't even know that that conversation has happened.
So it doesn't know that that's something that's of particular relevance to me. and the signals that the recommender system uses to determine what to share are generally the historical signals that the user has provided in terms of interaction.
Sometimes the system doesn't have any signals to work from and it's got to popular things or look at the sequence that you're interacting with most recently.
There's a lot of complexities and a lot of variation within the space for sure.
Our guest today is Evan Oldridge.
Evan is Senior Manager for the Merlin team at NVIDIA.
Merlin, of course, being a recommender system, that's what we're talking about today.
Evan, I want to take at least a couple of minutes here and ask you about your own background.
I know you've been working kind of more broadly than recommender systems by name, but in the intersection of human computer interaction and machine learning for a while now.
And if I've got this right, You were previously working at an online dating platform, which makes me wonder if you were using recommender systems to recommend humans to one another.
Maybe you can tell us a little bit about your background. background.
Absolutely. Yeah. So I did a PhD in computer vision from the University of British Columbia here in Vancouver, where I live.
That was pre-deep learning, so everything that I did in my PhD could be summarized in probably five lines of TensorFlow at this point.
But I went from that role into a job at Plenty of Fish, an online dating site.
And while I was there, you know, the initial –
Goal there was image summarization and trying to understand and deal with malicious actors within the system, people sending photos they shouldn't send and that kind of thing.
But, you know, along the way, like the core business value of what we were providing within Plenty of Fish was the users that were being recommended to each other in the recommender systems.
A small-scale organization, I think there was about 30 people when I joined and the founder had written all of the algorithms that were doing the recommenders.
And from there, I was the first data science hire and built that up.
And yeah, and it was exactly as you mentioned, recommending humans to humans for online dating.
And it's a very interesting space. It's a funny one in the sense that like, The users are the items, and so it's reciprocal.
You want to find and make sure that somebody may be a good fit for you, but you're not a good fit for them.
Yeah. There's lots of signals that we were measuring and tracking in terms of interaction and and sending messages and engaging.
And at some point, you would end up off the site.
And then there was great testimonials and we had this way of collecting kind of testimonial information from customers who, you know, ended up getting married based on the site.
And it was a very interesting place to be.
You'd leave work on a Friday night and head home and there was probably 50 or 100,000 people out on a date. because of an algorithm that you it's very very strange to think about you know like the number of potential new humans that are in the world, marriages, whatever else that just happened because of these algorithms.
I tend to think of Puppet Masters, if you will, as thinking about abstractly all of the butterfly effects their actions have on, you know, the world and people.
But I'm seeing you in that, you know, back then when you were with Plenty of Fish.
It's kind of literally being able to see the people out on dates, you know, if you look across the city or town you live in.
Yeah, it was interesting. And the data was driving it all.
And I think it was my first foray into recommender systems.
And then I left Plenty of Fish and went to work at, I guess I got into deep learning and fast AI.
I took six months off when I left Plenty of Fish after we sold to Match Group.
I spent time really getting into deep learning, which I hadn't spent any time doing and kind of Got deep into that through the FastAI course.
I wanted to ask you, and not having realized that about your journey, it answers the question. what's it been like either then or looking back, kind of going from doing your PhD pre-deep learning Now, you're at NVIDIA doing what you're doing in the thick of it and just I don't know, either mentally or sort of pragmatically, what that leap has been like for you.
Yeah, it's like, I feel like computer vision has kind of left me behind and I don't really spend much time thinking about it.
Rexis takes up the entirety of my... you know, my mental space, I would say.
And I think... deep learning based recommender systems aren't really proven out in the way that they are for computer vision or NLP. in some people's minds.
I think for me, I can show some pretty conclusive proof.
And, you know, I've talked with a lot of customers of the larger scale that are using deep learning based recommenders.
So I'm convinced of their efficacy, but it's not a foregone conclusion for a lot of people.
So it's interesting to be kind of back in that space.
My roots are also in tabular data, and I think tabular data even more so people are questioning the relative performance of tree-based models versus deep learning.
And so There's some interesting questions in that space in terms of the dominance of deep learning and whether it makes sense to use it for everything.
It's interesting to be exploring that in the Rexis field.
You mentioned earlier that NVIDIA has some tools available to make it easier We're talking about smaller organizations, but for anybody who wants to build a recommender system to do it, Do you want to speak to any of the specific things that are out there or maybe that you're working on with the Merlin team that folks can use?
Yeah, absolutely. Merlin, the team consists largely of people like myself who've built recommender systems in production in the past, and understand the pain of it.
It's really hard to build a recommender system.
You've got those four stages that I talked about. just deploying all four of those stages together is an incredibly complex nightmare.
If you look at, I saw recently, I can't remember the company that put it out, but the MLOps report that they put out, showed that it was something like only 11% of companies were able to deploy a model to production within a week.
And I think... If I remember correctly, more than 50 percent of companies took more than a quarter to get a model into production, and that was certainly my experience at most of my roles.
Plenty of Fish was a bit different in that it was an interesting system based entirely in Microsoft SQL.
We were building one of the more complex SQL servers and working some of these interesting algorithms in there.
We were deploying to production pretty regularly because we had access to the base code.
But when I moved to realtor.com, it took a long time to get models into production.
And I think most organizations are facing that.
It's a really big challenge to take a recommender system model and get it into production.
And so we're working on three main kind of premises, like make it Work we want to have a framework that really provides end to end recommendations so you could do all of the different stages.
All the things you need. make it easy so you know it should be straightforward to be able to do things that are commonly done within the space and so you know we're really thinking about like Where was a pain point in our past where it was a real challenge to use the existing tooling and how could we smooth that pain point over?
And then make it fast. We're NVIDIA. We want to make sure that this is performant at scale.
Absolutely. how these things scale is an incredibly important part of the problem space.
And I think we're starting to think more about companies in terms of their recommender system journey than in terms of like providing them with a solution that fits their needs. right now and thinking about where can we provide solutions all the way so that they can just adopt this framework and work their way through, you know, the exploration phases of trying to determine what model they want to deploy, and then get that model into production in a straightforward way.
I think a lot of libraries, especially in the REXIS space, but I think in general, target the data science, you know, and they're trying to help data scientists develop algorithms better models, but in my experience, the hard part about machine learning isn't the data science, right?
It's not that data science isn't complex or hard or challenging or interesting, but it It's that last mile of getting the model into production and keeping it there, keeping it up to date. maintaining the metrics, determining what's going wrong if something's going wrong.
Then a lot of side that's got me super interested in some of the developments within the space and and really from our perspective, thinking about how we can build tools that support not just the data scientists, but also the ML engineers, the MLOps teams that are supporting these systems in production.
One of the accomplishments that I'm most proud of that the team has done over the last year is, In conjunction with the KGMON team, the Kaggle Grand Masters of NVIDIA, we competed in the REXIS Challenge competition, the SIGIR Challenge. and a webconf challenge, you know, so all these kind of great recommender system challenges and one, all three of them In fact, for the Rexis challenge, have won it for two years in a row now on a very massive Twitter dataset.
And that's all, you know, joint work with the KGMON team, and they are incredible to work with.
And really... The amazing thing in working with them is we're able to take their thoughts and their ideas and their intimate knowledge of data science and provide some context around recommender systems and build upon that, and then turn around and bake all of that back into the tool.
So we've developed a bunch of feature engineering and pre-processing techniques that have been baked into our pre-processing library in vTabular that you can leverage automatically.
It's a single line of of code now to do what Kaggle Gradmasters have spent years developing.
It's super exciting to see that feedback loop of competing in a competition at the highest level, and developing these techniques that really enabled us to win the competition and then baking that into the product.
And one of the coolest things about competing the second year around was we found other teams competing using our code which we've kept open source. and leveraging those techniques.
So it's great to see that kind of those ideas expanding out and being built upon.
And I'm really proud of the accomplishments of the team It's pretty amazing to see what was accomplished over the last year in terms of Rex's challenge wins and Rex's competitions.
Yeah, congratulations. It's got to be really satisfying.
Thank you. Yeah, and I'm hoping it demonstrates to the community and the space that we know REXIS.
We know it intimately. And we're trying to share that knowledge and use that to build tools that will really kind of help companies and teams that are trying to develop recommender systems that maybe don't have the knowledge or maybe don't have that 150, 200 person engineering or maybe they do and they want to kind of leverage that org for something else because it doesn't have to be so complex. it really is a very fragmented and challenging complex space, but it could be a lot easier.
So, you know, we're putting in the effort to make that happen.
And honestly, that's, that's one of the things I love about working at Nvidia is like, because we're the AI company that works with every other AI company, like nobody else is motivated to do this work, but you know,
We really are. If we can change the whole field of recommender systems, it'll be to our advantage for sure.
I mean, selfishly, I'm most excited to develop these tools so that we can really dig in and explore the space in a way that isn't possible.
Right. That's great. So even you mentioned how the space is lagging is the right word, but a little bit behind computer vision and NLP when it comes to the move from CPU to GPU having happened more recently and that acceleration. and being able to get to a point where you're exploring new possibilities.
Where do you see the space headed over the next couple of years, five years, let's say?
Is it about you know, almost that you can look to computer vision and NLP in certain ways to kind of see where your space is headed because of that hardware and sort of paradigm shift. acceleration effect or are there other things you're looking towards as possibilities in the future that might really shape the space?
Yeah, I think it will head in that direction and maybe not as quickly because we still run into that issue I talked about earlier with companies not really wanting to reveal or provide datasets or the ability to share data in terms of a transfer learning situation.
That will always be a barrier, but what we're hoping to do with Merlin a bit is is provide a standard set of tools and a standard framework that everyone can use and think about. to be able to do accelerated recommender systems.
I think especially as we accelerate things in that 10x factor or more, It changes the pattern.
One of my favorite diagrams that I saw since joining NVIDIA is the diagram of the rapids difference between the developer who, at the start of their day, gets a coffee and then starts their feature engineering process and then goes to get another coffee because that takes so long and comes back.
Just this back and forth, you know, drinking six or ten cups a day and getting to the point where, you know, they can do – basically three things in a day.
And it hits home because that was me a couple of years ago.
I was intimately... you're experiencing that and it was so frustrating because you like you want to get stuff done but Because of that lagging signal, you're not nearly as effective when you get back to it to try and dig in.
Seeing the difference that it makes when you get to that, one, two, three-minute cycle where you're running something and getting the results, and running something and getting the results, and you get into that flow pattern, you're really able to explore things quickly. you get to the point where you're iterating so quickly, and then you can start leveraging the parallelization that happens on the GPU and begin to scale things up.
I think that's going to lead to a significant revolution in the space.
There's a lot of unanswered questions and it's pretty hard to do things in the space.
Most of the research being done is being done on the ranking stages, and I think there's a lot of work that could be done to think about the ordering, the filtering, the retrieval stages of a recommender system.
We're trying to develop tools that let our customers or the people who want to use the library, not just like... use it to do that exploration, but at the end of that process, have a model that they can deploy into production.
And I think that's a huge paradigm shift from what you see in most industries data science orgs where at the end of that exploration phase, there's many weeks or months long journey to take what's been developed and migrate that into a production-ready system.
I do think that aspect of it is going to pretty quickly revolutionize the space.
I love how well, I don't love that you went through the pain, but I love how, you know, the the the pain that you had to go through earlier in your career to do these things is this clear driver, part of your passion and making it so that The folks coming up doing this stuff now don't have to go through that as part of what you're doing.
It's great. Even for folks who want to find out more about the work you're doing, what's going on with Merlin, Where can they go online, whether to learn more or dig a little deeper into maybe some white papers, some of the more technical aspects of the work?
Yeah, if you go to our GitHub, github.com slash nvidia-merlin, That's a great starting point.
We linked out to a bunch of our papers there.
I have a bunch of talks that I've given about Merlin across various sources.
If you search for my name in YouTube or search for Merlin Recommender Systems, there's a lot of information that you can find out there.
One of the more interesting things that we've seen in the space, when we talk about the kind of the relationship between REXIS and other fields is the NLP space has a thousand fold more researchers in part because of those data sets that I talked about. really having access to publicly available data sets to train on advances the field. at a much faster pace.
And so what we see in Rex's space is kind of a copying of a lot of techniques, especially for session-based recommendations. you can think of like next word prediction from a sequence of words as being a very similar problem to next product recommendation from a sequence of products right it's it's Conceptually, it's the same thing.
That's called session-based recommendation.
What we looked at was, Within the space, there's all of these advancements in REXIS that are lagging behind NLP.
So you see the development of of BERT and then two years later, the development of BERT for RAC, right?
You can find that pattern consistently and we kind of highlighted this in our paper that we put out for Rexis this past year. and the repo that I'm going to dive into now, Transformers for RAC, that we're building upon and working into our new libraries. that we're developing under Merlin.
But the idea with Transformers for RAC, we sort of looked at like, There's a player within the BNLP space that's maintaining and keeping things up to date, a hugging face.
They're developing some amazing models. and the Hugging Feast Transformers library is really rapidly capturing the changes and the developments and the advancements in the NLP space.
What we did was looked at how do we leverage that research and that work and connect it to the Rexis community in a very straightforward fashion.
And so we looked at essentially building upon that library, providing the extensions needed for sequence of words and sequence of items.
It's identical, but often what you have with with products is not just the item itself but all the product information related to it right and And so, adding that slide information, adding that metadata, we did a paper where we, you know, sort of demonstrated those ties and talked about the library and then into a bunch of research efforts to kind of explore the space.
And that led to the library transformers for REC.
That's one of the libraries that's available on our GitHub repo.
Very cool. Well, even thank you so much for taking the time to come on the podcast and explain a little bit about what you're doing and why you do it, which came through and some of your stories you told.
We appreciate it. Thanks a lot Noah. Thank you.
Thank you.
