Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. The COVID pandemic is affecting people in different ways.
I haven't seen any data on this, but I'd hazard to guess that headphone and earbud use went up broadly speaking over the past few years.
I know that the four people in my own home definitely started using our earbuds and headphones more over the past few years than ever before. whether to escape reality into a playlist or podcast, or just enjoy a little noise cancellation enhanced peace and quiet.
Our guests today went through their own version of an earbud-assisted pandemic, but with a significant twist.
Rather than just popping earbuds in to escape the noise of a crowded apartment, they designed earbuds that could cancel out background noise even better than anything else out there on the market.
And they did it using AI. Maruchi Kim, Ishan Chatterjee, and Vivek Jayaram are roommates and graduate students at the University of Washington.
And they're the trio behind ClearBuds, a new wireless earbud system that's advancing the state of the... Let me say that again. that's advancing the state of the art in personal audio.
The trio is here today to tell us all about clear buds and maybe a little bit about the dynamics of living three roommates working together on a groundbreaking audio project in the midst of a global pandemic, but we'll kind of see where the conversation goes.
Marucci, Ishan, Vivek, gentlemen, thank you so much for taking the time to join the NVIDIA AI podcast.
Welcome. Thank you. Thank you. Thanks for having us.
So I will throw this out there and you guys can decide who's going to start answering the question first.
But let's just dive into it. Tell us a bit, if you will, about where the idea came to start working on ClearBugs and you know, what the journey was like, the first part of it, at least.
Yeah. Thanks, Noah. Thanks for having us on.
Yushan here. So as you mentioned, Clearbuds for us was born out of necessity.
So as you alluded to, in addition to being classmates, Maruchi, Vivek, and I were also roommates when the pandemic lockdown started.
So like your family and many others, we found ourselves taking calls in close quarters between kitchen and construction and conversations.
Our house was a pretty noisy environment for taking these calls.
So... We have collective experience across hardware, networking, and machine learning, and together we thought we could do something about it.
So we arrived on the idea for ClearBuds pretty soon after that.
And so you guys are, are or were grad students in computer science, electrical engineering.
What's kind of the collective background?
You mentioned hardware and software. Yeah, so I can start first.
We're all students at University of Washington studying computer science.
So I myself... I have a background in hardware engineering and hardware architecture.
I worked at Microsoft on the HoloLens device, the augmented reality device.
Mauricio, you want to talk a little bit about your experience?
Yeah, I'm an electrical engineering computer science undergrad from Berkeley.
Post Berkeley, I worked at Apple on AirPods Max for four and a half years.
And then now I'm in the PhD program with these guys in computer science.
Yeah, and my background is very much on the machine learning side of things.
So previously I'd worked on Other research projects related to speech enhancement and source separation, which was published in venues such as the International Conference of Machine Learning and Neuroscience.
Well, Excellent. So clearly, you guys were the trio to take on this task.
So what's the best place to start? My question that comes to mind is, what were or are the readily available solutions not doing? that presented a problem or an opportunity for ClearBuds to do better?
Sure. So just like you, a lot of us have commercially available headphones such as AirPods or Pixel Buds.
And... Even when we were roommates during the pandemic, we used to take them for phone calls and they just didn't remove the background noises.
I think all of us have experienced that.
If you're taking a phone call, even if you're wearing AirPods, If people are talking around you or somebody is using the vacuum cleaner, the person on the other end of your phone call is going to hear that.
Right. So I think intuitively we can all sort of grasp the problem here, which is that these systems aren't doing enough.
Now, there have been some other works that have incorporated into the video conferencing platforms such as Zoom and Google Meet. which try to use just your MacBook's microphone in order to separate out other noises.
But again, these solutions are really not doing enough in terms of separating out the background sounds.
I think the reason that the demo went so viral and was so compelling is that people just haven't experienced taking a call with somebody vacuuming next to them and not having it go through to the other end.
So it's such a simple concept, but people just hadn't seen this happen before.
Now, is there a difference between the kind of noise canceling, active noise canceling that I think of that when I'm listening... to whatever I'm listening to, music, podcast, whatever it is, it cuts out background noise so I can sort of hear and focus more on what I'm listening to.
Is there a difference between that and what you guys are talking about with removing, as we're talking, removing the noise from my end as I speak into my microphone.
And so, yeah, there's definitely a difference between active noise cancellation and what we're working on, which is called speech enhancement in the academic community.
And so in active noise cancellation, it's a very closed form problem.
It's very similar to kind of autopilot or like how cruise control works in your car.
You want to set something and you want to hit that target.
And so let's say you're wearing a pair of headphones.
You have some noise coming in. there's a microphone sitting outside of the headset recording that noise.
And there's a very calculated distance that engineers design where they can create precise anti-noise that'll cancel out right when it hits your ear.
And all that's done on top of that is that the music or the phone call is added on top of that.
We have a, much more difficult problem here where we're using microphones in your earbuds and it's already being mixed with the noise with your speech.
And so we don't actually know what the true noise profile is because it's already mixed in at the microphone.
Gotcha. I would say the intuition is that active noise cancellation is for... me to not hear my own environmental noise.
Whereas what we're doing is really the person on the other end of my line should not hear my environmental noise.
And that's a lot harder because they still need to hear what I'm saying.
So with active noise cancellation, you can just cancel everything.
Whereas with speech enhancement, you need to selectively cancel the noise while still sending my voice through.
And that's harder than canceling everything.
Got it. So if I'm understanding correctly, the issue, and this makes sense, I just hadn't thought of it this way before, is that When I'm just listening and I want to cancel out, as you said, you can just kind of cancel everything out and then... just pipe in whatever the material is that I want to listen to.
And that is a relatively easier problem to solve.
But when we're talking about canceling the background noise out when I'm speaking into the microphone, transmitting my sound across, You can't just cancel everything because effectively that would cancel out my voice as well.
So- There's a few reasons why the existing solutions such as AirPods or Zoom can't do this kind of speech enhancement.
Okay. Right now, when you use AirPods, even though you have a microphone on each side of your head, so each AirPod has a microphone, it's actually only using one side.
So it's only using one AirPod at a time.
And you might notice this, the battery will drain faster from one AirPod when you're talking because it's really only using one.
And so what we found is that by using two microphones, one on each side of the head, we could use that to sort of triangulate your voice location in the middle compared to just having access to one microphone, which gives you no spatial information.
So why is it that AirPods only use one microphone?
This is currently a limitation in the Bluetooth standard.
And so we actually had to develop our own Bluetooth protocol on top of the existing standards, which could stream time synchronized microphone data from both of your AirPods clear buds into the phone.
Gotcha. And this provided additional information to help isolate your voice.
And so then you can triangulate on the voice, isolate my voice going out from all of the other noise in the background that the mics are picking up. and then work on separating the two to cancel the background noise but keep the, got it, okay.
There's two aspects that we use to do the separation.
One is the triangulation, which I mentioned is only possible by using two microphones.
And the other is using deep learning we can outperform the traditional signal processing methods such as beamforming, which have been used in headsets such as the AirPods.
And we do this by collecting a large data set of things that are human voice and things that aren't human voice.
So for example, the vacuum, the blender, construction noises, and then we can train a neural network to separate out just the human voice by giving it thousands and thousands of such examples of these mixtures.
And one other innovation here is that we're also running this neural network on your iPhone So traditionally, neural networks are very expensive, and they have to run on the NVIDIA chips in the cloud.
And so by taking advantage of the increase in mobile compute, we were able to run it fully on your device in real time, which is necessary for a phone call.
So really, the innovation here was combining two aspects, the wireless time-synchronized microphones which stream to your phone and a neural network that runs on your phone.
Got it. So there's a neural network running on the phone.
There are the synchronized dual microphones, both being used at once, which is, I'm going to use the word innovation, but something that's not common in the commercial systems.
And then you guys also wrote your own version of the Bluetooth stack or an update to the Bluetooth stack as well.
Yeah. There's a lot going on there. You guys are busy.
Yeah. I mean, we started from the ground up, from the mechanicals, the hardware was all custom designed, fabricated in China, back here.
We wrote a custom firmware on the earbuds, the time synchronization, like you mentioned, and then the wireless protocol that we wrote on top of Bluetooth Low Energy. um to get the microphone synchronized sent to the phone um and then from there uh again is a machine learning network created the algorithm to suppress all the noise and enhance your speech.
As much as I want to keep asking questions about the hardware and the way the earbuds are constructed, because that's kind of my own...
I'm a little bit of minor headphone earbud nerd and really fascinated by this stuff.
I want to ask about the machine learning, deep learning aspects in particular.
What were some of the challenges or maybe unexpected either challenges or learnings or breakthrough moments along the way that kind of really, really kind of shaped, you know, how you arrived at Clearboards.
Yeah, so in previous works, there's been a lot of papers on using neural networks to separate human voice from these sort of noise source types.
And We built on these previous works. So what we started with is we took a state-of-the-art neural network for speech enhancement which was running on an NVIDIA GPU.
And we tried to make it run on the iPhone So obviously, it's too heavy and too clunky to run on an iPhone.
So what we did is we basically reduced the number of layers and the number of parameters aggressively in order to make this existing network run on the iPhone.
And when we trained it and we tried to deploy it on the iPhone, what we found is that by reducing the number of parameters so drastically, the quality of the output was significantly reduced to the point where you would hear unpleasant you know, pops and clicks and all of that, which sort of makes sense because neural networks are effective because they're so big, they have so many parameters.
And obviously when you reduce it, you would reduce the quality too much.
So one intervention was how do we basically increase the quality of the voice output without increasing the size of the neural network again, which we had reduced in order to make it run on the iPhone.
And the key insight here was that we were able to combine two different kinds of neural networks.
So one is a time domain network. It runs basically on the audio samples themselves.
And the other is a frequency domain network.
So it runs on a spectral representation of the audio.
And by combining these two, each of which could be very lightweight, they were able to complement each other in a way that produced a high quality output without needing a single very large neural network.
Now, you've been saying iPhones specifically, so I'm guessing, but I'm asking, ClearBuds work currently only with iPhones?
Yeah. So we were able to take advantage of the neural engine on iPhones.
A lot of Android devices have been... coming out with very strong neural engine processing capabilities, but it would need to be engineered.
Technically, it is possible to get this to run on an Android device, especially ones with the chipset comparable to the iPhone, but...
Given the overhead of developing the technology for a single platform, we decided to stick with one.
No, sure. It makes sense. And how far back?
What's the oldest iPhone that ClearBuds will run on?
Yeah, so when we developed it, iPhone 12 was the newest one.
Okay. we did is we made the network as large as possible so that it could run in real time.
And we targeted for the iPhone 12. And we showed that on previous devices, it's slightly slower than real time, but it could be modified to work on those devices.
So we really targeted it for a real-time deployment on iPhone 12.
Makes sense. Yeah. Were there particular situations and specific types of background noises or even specific types of... voice profiles of a speaker that gave you trouble that kind of posed some particular problems you had to get around?
So Yishan here, I think one of the hardest challenges for us were situations with two speakers. right?
Multiple speakers. This is not something, you know, Vivek at the beginning mentioned that Zoom or Google Meet integrate speech enhancement algorithms into their services.
But when you have one microphone, they just let through the loudest voice, essentially.
Kind of like humans in a meeting. Right, exactly.
And there's no way to necessarily parse out between voices, which is your target speaker, and which are the background speakers.
And so this was probably one of the more challenging situations, the one that's currently unaddressed by these type of mono-channel models, single-channel models. posed us the greatest problem because essentially all we have to work with there is just the spatial aspect.
As Vivek and Marushi had alluded to earlier, we had you know, both spatial as well as the audio signature that we're using to determine what voice to pick out and how to enhance that voice specifically.
But in this case, we only have spatial information.
So this is probably one of the more challenging aspects and one of the aspects that what's out there currently generally don't handle that well.
Right. Our guests today are Maruchi Kim, Ishan Chatterjee, and Vivek Jayaram.
They're graduate students at the University of Washington and the... brains behind Clearbuds, a new wireless earbud system that's leveraging deep learning to advance the state-of-the-art and personal audio, as I said at the top, but specifically it's a new method of Suppressing background noise to make the speaker's voice clearer in phone calls.
I assume this works on video calls as well.
It probably doesn't differentiate once you've got the audio signal coming in, but really making it a lot easier to do what we all do anyway. which is to pop our earbuds in and take a call, have a conversation from wherever we are, locked down in an apartment, walking on a busy street or standing in a vacuum cleaner store. as might be challenging.
So we've been talking about how the system works, what the impetus was to to start working on this problem.
I'm going to fast forward a little bit. You guys mentioned a demo that went viral, and I know that you've had some success publishing papers.
Where's the project at now and where are you looking to take it from today forward?
Yeah, so right now we've had, as Vivek mentioned earlier, we had a ton of interest.
We were at the demo session at our conference on mobile systems.
And while we were having people test it out, Vivek also took our Thank you. demo video and posted it to Twitter where when we woke up the next day, we were surprised to see that 400,000 people had seen our demo video, which was a wild experience.
Yeah. And so currently what we've done with With the work is we've open sourced all of the hardware, the software, the firmware, the mechanicals, the PCBs.
And we think this could be a real benefit to the community because it's kind of a very unique system.
It's different from what exists out there.
Because it has these hardware and software components, people in each of these domains might benefit from being able to have an example of that kind of cuts across these fields, right?
We can also see that the type of technology that we use might have applications in things like hearing aids, where part of what makes it difficult to be able to function with hearing loss and what these hearing aids don't necessarily address is picking out a target speaker, a particular person that you're talking to.
Hearing aids generally right now tend to elevate all noises, including background noises.
And this makes it difficult for one person to necessarily pick out, okay, in a conversation with a set of limited people, what are these other people, other conversational partners discussing, right?
And then thinking even longer term, right?
One could think of applications that build upon the Bluetooth technology that we discussed earlier.
We had this ability to synchronize nodes wirelessly to under 70 microseconds.
And that was part of the algorithm that we had to develop to be able to have the left and right earbud form a virtual microphone array across the head.
But what if we take those nodes and we don't necessarily have them in a left and right ear, but we have them on different areas of conference table, for example, right?
Now you can imagine, if we have multiple of these nodes, you can start doing things like spot forming, where a particular person sitting in a particular spot can be picked up by these wireless nodes, right?
And they can be beamed in a way that having a... single microphone array in the center of the table might not be able to do.
I was a stickler for using corded headphones and earbuds, you know, only for a long time.
In part, just because I didn't want to go buy a pair of Bluetooth and, you know, deal with the charging and all that kind of stuff.
But in part, because I It's kind of old school and you're going to get the best audio quality with a cable no matter what.
And I still use a cable. I'm using cable right now.
But I also am becoming a huge fan of using wireless earbuds for different applications.
And when you start getting into what you described, it's a system of hardware and software.
And folks like yourselves working with the software, with the hardware knowledge can start to come up with these other applications.
You know, you've built this core technology.
You've got this ability to wirelessly sync the stream.
And then you can see these other possibilities using it, which is, you know, which is really exciting.
So is there, you mentioned everything's open sourced at this point.
Is there a company? Are you guys keeping the band together, so to speak?
Or have you kind of hit a limit on, you know, We're happy with this, where we've gotten it to, we've published, we've open sourced, and you're moving on to other things.
Yeah, I think we're pretty happy with the open sourcing everything.
I think it'll be a real benefit to the research community.
There is a new standard coming out this year called LE Audio, which is Bluetooth 5.2.
And a lot of the work we did to get the synchronized wireless jam working on a phone, which is, I think, our huge competitive advantage, will be supported by the new standards.
Oh, that's great. I think we really like our research project and I think it's just kind of a glimpse of the future of what's to come.
Yeah, very, very cool. And just kind of for my own understanding, are the power consumption requirements greater on the earbuds.
I mean, you said that all the neural networks running on the phone, so you're leveraging the phone's compute.
Is there any additional, you know, compute needed in the earbuds themselves or really just that you're leveraging the additional microphones, but otherwise it's...
There's not too much additional compute.
You can imagine AirPods have already gone so complex these days already.
And so... We're using off-the-shelf chips to do what we're doing with the microphones and synchronizing them over the air.
So yeah, I'd say on the earbud side, it's already pretty lightweight, but then on the phone side, we're obviously running it all on the phone.
So there's a bigger hit there overall. Yeah.
Right. Very cool. So a lot of times we like to end these conversations talking about the future implications of what you've been working on You guys already hit that a little bit.
So I'm curious, what's next for the three of you?
Is there more, I don't know where you're at in grad school.
Is there more work or do you turn to theses?
Are you working on other projects? Anything you want to share about where you're headed individually?
Yeah, this work is part of my doctoral thesis, which focuses on source separation, both the type that you can run on a GPU, running it on a headphones.
The next work that I'm actually working on is using headphones such as AirPods or Clear Buds to actually map out how your head filter sounds.
And this is applicable for spatial audio.
So the goal is to help increase the fidelity of your listening experience to simulate sounds coming from behind you or above you.
And we can use the microphones to sort of capture how you hear sounds and create personalized audio experiences.
So this is very much part of my journey in spatial audio and source separation.
Very cool. Yeah, so same here. Part of my doctoral thesis.
I think my next project, so I'm kind of focused more on like intelligent wearable systems, kind of trying to push the envelope of, you know, we have these wearables a lot now and how can we better, how can we enhance those features?
And so I'm trying to design a ring with like a super low power camera to kind of control your devices without you having to speak to them all the time so you don't awkwardly invoke Siri or Alexa You can kind of point your finger at the device and kind of rotate your hand to increase volume or tap the ring to turn on and off light bulbs and stuff like that.
I would sign up for the beta of that immediately.
I've always, for whatever reason, I use it a little more, especially now that I have kids and they're just kind of native to it.
But the whole talking out loud to my device, I'd rather be stealthy about it.
Not that I'm doing anything particularly interesting.
Just, you know, keeping it low key. Ishan, what's next on your plate?
Yeah, absolutely. So I think Clearbuds is a good example of the type of projects that... excite me, like these ones that kind of cross hardware and software, but specifically that they light up a novel experience or a new experience that, you know, people can take a step back and say, wow, okay, that's something that's helpful, but also is not only novel, but also provides utility. right, is something that people find helpful.
Absolutely. You know, I'm interested in these technologies that are sort of right around the corner augmented reality and spatial computing.
A lot of companies are sort of developing these types of computers that are wearable and distributed around the environment.
And one aspect that I think we haven't thought so much about is how do we naturally interact with them?
How do we actually get input into our augmented reality devices?
So ClearBuds is one good example of how we might be able to get better voice input for one of these devices, even in on-the-go environments.
And I'm kind of looking at... at other methods, gestural eye tracking and kind of across the board and how do we fuse these?
Yeah, the less you have to think about how to get your input commands into the system, the better, right?
That intuitive human interface is so key.
Before we get into wrapping up, anything that you guys wanted to talk about?
I mean, this is from my perspective, this is great.
But anything you guys wanted to talk about that did not come up? one aspect that's unique in our design, you know, Vivek had previously discussed, taking something that was on a desktop GPU and shrinking it down to be able to put it on the phone.
Yeah. But this is an AI podcast and all the folks who have worked on an AI project know architecture is one thing, but data is a whole nother beast.
How do you train these networks? For us, that was also an interesting aspect.
We had to work with synthetic data, but because we're actually deploying this on real hardware in real unseen environments, how do we actually get enough data from the clear buds itself? how do we create enough environments to get this training data you know we're a set of three students working in an academic institution, we don't have necessarily the resources available to us that a much, much larger company might have.
Sure. And so here we sort of pursued a hybridized training methodology where we generated a large amount of synthetic data by mixing together existing datasets that have just individual clean voice samples of... different accents, different genders, and mixing that along with a set of different environments, environmental sounds, binaural environmental sounds, right? and applying different filters and different types of reverb to be able to increase the diversity of that data set.
But we also had to collect a lot of hardware data.
So we had a a setup that existed in one of our rooms for a long time of a foam mannequin head that we lovingly referred to as Frank.
We wore clear buds for hours and hours upon end while we played noise on a set of speakers around Frank's head.
Yeah. And so Frank spent a good amount of time collecting training data for us.
And then finally, we need to get also real voices.
So we had some of our friends jump into an anti-color chamber where we had them talking to clear buds to be able to collect, again, a diversity of different voices and what was the frequency response that our specific clear bugs had in the wild.
And by combining this data, then we could get something that was rich enough to generalize to in the wild environments, which is how we evaluated this.
Very cool. Is the system capable, either now or just theoretically of... leveraging the sound data that it's collecting just in normal use and then feeding that back into your data sets to train models and push out updates, I would guess?
Yeah, so certainly I think if this were to go on to be a commercial product, I imagine that companies would be interested in that sort of methodology to be able to collect real training data by telemetry.
For our research project, we didn't to pursue that path.
But absolutely, that's a good point. Gentlemen, this has been a pleasure.
It's been fascinating to hear about. And as I said at the top, you know, I've always had an interest and I listen to a lot of music, a lot of music. a vested interest in the technology you guys have been working on.
So, you know, a hearty thank you from all of us in earbud land for... for advancing the future of our own experiences.
But for folks who want to learn more, whether to dig into some of the published research and the open source stuff you were talking about, or even just to follow you as you continue your academic and professional careers, are there places online where we can send the listeners to learn more?
Yeah, so for this project, all the information is at clearbuds.cs.washington.edu.
And you can see demo videos, including the ones that went viral.
You can see the open source code. the research paper and other comparisons with existing methods.
My own Twitter handle is vivjay30. That's vivjay30.
And you can follow me to see research updates and other projects that I'm working on.
Ishan, places where folks can follow you and your work?
Yeah, absolutely. From the URL that Vivek mentioned, clearbuds.cs.washington.edu, you can find a link to my portfolio.
Excellent. And Marucci? Yeah. You can also find a link to my portfolio on the website.
It's also just maruccikim.com. And if the URL is a lot, you can also just Google Clearbuds.
I think SEO has put us at the top now, so you should be able to find us there.
Excellent. Sounds good. Well guys, again, thank you so much for taking the time out to come on and talk about your work.
It's great stuff. And I'm looking forward to seeing what y'all do next.
I have visions now of... transparent augmented reality that i can control you know with a very subtle low-key move of my finger and rotating my wrist that people won't even know about you know So I'm looking forward to the future more than I was 30 minutes ago.
Appreciate your time. Thanks for your time, Noah.
Thanks, Noah. Thank you. ¶¶
