¶. Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. With the big shift to remote work and remote school earlier this year, Zoom has become a newfound part of many people's vocabulary.
Last month, Otter.ai, a Silicon Valley tech company, announced live video captioning for Zoom calls, a potentially quite big feature. to boost remote work across many industries.
And while the news is quite big, Otter.ai has actually been around for a few years now. and its AI-powered transcription capabilities have made it a favorite of podcasters and other folks who need to transcribe conversations regularly.
Otter.ai CEO and co-founder Sam Liang joins the show today to talk about Otter's new capabilities, the challenges and potential of using AI to transcribe conversations, and his own history in the tech industry.
Sam, thank you so much for joining the Nvidia AI Podcast.
Thank you, Noah, for having me here. And so just to set the table for folks who have not seen Otter in action, before we hit record. you set up a live transcription.
So in a window that I'm hiding, so I don't distract or scare myself with my own words.
We've got a near real-time transcription. of our conversation running, just showing up on the screen in front of me, which is one of the things, like I told you, I'm not surprised because I know a little bit about Otter,
But it's still remarkable to seeing it happen.
So congratulations on all you've done so far.
Thank you. We actually have transcribed tens of millions of meetings already.
So let's get into it. Maybe you can tell the listeners a little bit about what Otter.ai is. does how long you've been around and then kind of lead up to the new announcement with Zoom.
Yeah, sure. Otter.ai is a startup based in Los Altos in Silicon Valley.
We started in 2016. We built the speech recognition system for meeting transcription, meeting notes from the ground up.
When we started actually, a lot of people asked us, didn't Amazon Alexa do speech recognition?
Didn't Siri do it? Why do you guys need to do it?
But the problem is that when you use serial Alexa, you ask a question like what's the weather tomorrow or set an arm at 3pm and then sirio alexa will answer your question but it's not a long-form multi-speaker conversation like what's happening in the meeting.
So if you use Siri or Alexa for your meeting, It doesn't work.
We have to build the AI model from the ground up. to take notes for complicated meeting situations.
How do you spec that? How many concurrent voices can Otter listen to?
What are some of those capabilities that make it good for meetings and kind of set it apart from the other services you mentioned?
Yeah, as I mentioned, when you have a meeting, you usually have at least two people.
Sure. Sometimes five or ten or even more.
So every person would speak with different accent, different pace, different volume.
On virtual meetings, people have different background noise as well.
So the situation is way more complicated. than the Siri use case where only a single speaker asking a short question.
And meetings usually run long. It could be 30 minutes, 60 minutes.
And people interrupt each other all the time.
People may speak a little faster or a little slower.
They don't always speak grammatically correct.
They may pause and hesitate and restart and suddenly change topic.
All of these make taking notes for meetings way more complicated.
That's why we built the AI technologies to optimize note-taking for meetings.
And so right now, how does somebody use Otter?
I mentioned the new Zoom announcement and then You're running this through, we're recording this on Skype, and so you're able to transcribe our Skype conversation.
Does it function as a standalone platform?
Is it a plugin? How does one go about using it?
Yeah, Otter is designed as a standalone product.
It is a virtual meeting platform agnostic.
We are using it for this Skype call right now.
And you can use Otter for any meeting platforms, including Zoom, WebEx, Microsoft Teams, Google Meet. or even your phone calls, regular PBX phone calls, you can use Otter as well.
Well, obviously, Zoom is one of the most popular virtual meeting platforms today.
Earlier this year, we heard they had 300 million daily active users.
That's just insane. Otter did have a special integration with Zoom.
When you have a Zoom call, order can be automatically started and it's a plug into the Zoom audio stream.
So the audio quality, both the audio and the accuracy quality is very high.
Separately, Zoom actually came to us three years ago. asking for help.
So we licensed a subset of our technologies to Zoom and Zoom could some meeting transcription feature inside Zoom, but they only made that available to their enterprise customers.
Okay. So even if you are paying a Zoom user, but if you're not one of their enterprise customers, you cannot get other feature inside Zoom.
So yeah, as I mentioned, order is a standalone product.
It can transcribe. but it actually stores both the audio and transcript for you.
And everything is searchable as well. For our own company, for example, we are four years old.
All our meetings in the last four years are actually captured in order.
So at any time, anywhere, I can... use water to search for anything I have heard before. in our four-year history.
I was going to ask you about this later in the conversation, but since you brought it up, How does that change the way that a company or even a person approaches meetings and approaches work?
Is there kind of a fundamental – mindset shift that you see, kind of knowing that you've got this automatically archived and searchable record of all of the meetings?
Yes, we do believe there will be a fundamental change, and it's actually happening today.
Otter has been growing steadily before COVID happened.
People have been using Otter on their laptop or using Otter on iOS or Android devices.
So this is a cross-platform product. use it anywhere.
If I was a student or a reporter in my former life, And I was out interviewing somebody.
If you and I were talking in person, I could pull out my mobile phone. and run the Otter app and get a transcription?
Yes, you can. For people who, actually this was, quickly adopted by a lot of journalists.
Because they do interviews all the time in the they had to hire human transcribers to transcribe their interviews, which was very expensive, and it was slow as well.
Now with Otter, It's fast, it's cheap, it's instant.
So this was quickly adopted by a reporter.
However, the main market we're targeting is actually the business and enterprise meeting market.
This is huge, as demonstrated by Zoom, Google Meet, Microsoft Teams, right?
WebEx as well. Just there are probably millions of meetings are happening every day.
In the past, if you think about it, most of the voice conversations they were all lost.
They were not even captured. So after a meeting, How much can you remember is doing the meetings?
Actually, people are pretty stressed. They're really afraid of forgetting things, so they have to take notes.
Like crazy, especially for complicated meetings where people discussing a lot of numbers.
A lot of facts people need to remember. So you have to either type or in the past, people use a pencil or pen writing on a paper notebook.
The problem with that is it distracts you.
It reduces engagement. You lose eye contact when you are watching your laptop taking notes.
No, I've been in meetings, I'm sure most people listening have, where there's been a mandate that, you know, nobody's allowed to have their laptops open because we want full engagement, which... you know, in some situations is impossible, but in some it is, but, but to your point, then you're kind of losing that archival ability, which can be so important when you need to refer back.
So the enterprise fit makes a lot of sense.
I'm wondering, NLP, natural language processing is obviously a huge field, a huge problem with many, many sub problems and applications that people, uh, under the umbrella of AI are working on from all different angles.
I'm wondering between the four years ago when you started Otter or even earlier when you had the idea, how much of a background did you or your co-founders have in NLP and what sorts of, challenges have you been using deep learning and AI to overcome as you're building the product and the company?
Yeah, for the AI technologies, to be honest, it wasn't built by me.
I did my PhD at Stanford, specialized in large-scale distributed systems.
So I'm actually a system person now by training an AI person.
I was the lead of Google Map Location Service for four years. handling location data and huge amount of map. and location data which is critical for google mobile map right but i got into this space when I was thinking about a new startup.
Actually, I quit Google in 2010. and build the first mobile startup in October.
It was acquired and then I was trying to figure out something else to do.
So one thing came to my mind was voice. One reason was that actually I always forget things after meetings and conversations.
I had a hard time remembering information.
Also, it's very hard to share information with the team when I talk to someone and then I need to uh discuss the issue with the team then how do i make sure i convey the information correctly So we figured that, okay, it would be great if we can capture all the voice in the world and transcribe everything.
And we did some calculation. How many meetings are happening?
How many words does every person speak in their life?
Do you have a number? Is there an estimate for that?
Actually, there has been In some research, I've seen data like in a person's lifetime, he or she may speak a few hundred million words.
The number I saw was 800 million words. Right.
Okay. I was thinking maybe a billion or two billion, but I talk a lot.
So that sounds about right, a hundred million.
So if you think about it, almost all this data is actually lost.
It's actually quite wasteful if you think about the amount of information, the amount of insights and intelligence that's embedded in that voice data. so that yeah that's the origin of the company so uh back to the ai part you know we actually when we started with uh my co-founder yun fu who is also a computer science PhD, and he's also a system person.
So we actually look at the market. We tested the AI technologies, the voice technologies, and speech recognition API from Google, from Microsoft, they actually didn't work for the meeting situation. due to the reason I mentioned earlier.
So we decided to do some experiment and we look at what we can build and also we started to Look for talents.
So very lucky. We actually find a few good people from Google.
We convince them to join us. And we started to build all this technology from the ground up, focusing on multi-speaker long form conversations, which is, you know, what happened? in meetings and in you know over the the last four years which has made tremendous progress We trained our model on millions of voice data.
And as I mentioned, since we launched the product in 2018, we have transcribed Over 50 million meetings, I think over two billion minutes now.
Wow, that's amazing. And just just the tip of the iceberg, I'm sure of what's to come.
As you were talking and mentioned the multi-speaker problem again, it made me wonder, is there an equation, even if it's fuzzy math, Is it exponentially more difficult with each new voice that's brought in, or how does that work moving from... single voice language processing to then being able to recognize and differentiate multiple users or multiple speakers.
Yeah, it's hard to give a number of how complicated is it, but as I mentioned, the complexity came from multiple angles.
Accent is actually really challenging. Think about my own accent.
And English by itself is a pretty difficult lot of words.
It's not the easiest language. a similar pronunciation.
And it really depends on the context. So this is where NLP and the language model and all that really matter.
So it's not an isolated speech recognition problem.
It's really... embedded into the context.
So it's a lot of work. you know, whether it's traditional speech recognition or it's a deep learning based, you know, we use a combination in the end to end It's complicated.
It's not just one or the other. You need to use the right combination of technologies for different problems. or how many languages does Otter support?
Yeah, at this moment, we're actually just focusing on English.
Okay. This is a huge market, obviously.
We did get a lot of requests to support other languages as well.
I'm speaking with Sam Liang. Sam is the CEO and co-founder of Otter.ai, a voice recognition, language processing, AI deep learning company that is transcribing audio conversations.
And again, we led with the news of their new Zoom feature, auto-captioning and Zoom video calls.
But as Sam mentioned, their platform-agnostic work across all manner of tech platforms and voice conversations.
It's really remarkable stuff to see. Sam, you You alluded briefly to your time at Google.
I wanted to switch gears just for a moment and talk about your background.
You also mentioned your company before Otter, Aloha?
Am I pronouncing that right? Yes. So you've got a little bit of a background.
Maybe you can walk us back to you mentioned you were a systems guy and your Ph.D. in distributed systems.
But kind of start from there and maybe briefly walk us through your career.
Yeah, I studied computer science in college and then later I did my PhD at Stanford.
I worked at Cisco before and at Google. But, you know, I am always itchy about doing startup.
So, you know, that's why I quit Googling 2020.
10 and built this mobile startup first. For that one, we were actually focusing on mobile location, contextual data, personalization, behavioral, But the important part of the startup was about getting large amount of data and analyzing the data.
So this first startup was about getting tons of the mobile and location data.
And other sensor data as well, we look at accelerometers and even a Bluetooth data as well, and use that to analyze the mobile behavior.
So after that company was acquired, I was thinking anything more crazy we can do.
And I realized that there's one sensor we were not using, a mobile device, that's actually the microphone.
And in the meantime, as I mentioned earlier, I realized that it just...
I really need to have my meetings transcribed so I can Search them I can analyze it and I share it with my team.
So then some multiple reasons motivated us to start heartotter.ai.
Auto.ai in some sense is trying to actually really capture all the voice information and make it useful for people.
And today with remote work and distributed workforce, most of the meetings are happening. on Zoom or WebEx or Microsoft Teams.
So people are finding Otter really useful to improve their collaboration remotely.
So that leaves me in the last couple of minutes we have here together.
You mentioned Otter was growing before COVID came about and certainly Since then, we're recording in mid-November, so over the past six months or so, you know, this boom in remote work and remote learning, remote everything.
Do you think whatever happens with people continuing to work remote or how many percentage of people, you know, go back into offices in due time?
Do you think that this trend towards capturing the world's information, in particular the world's audio data, And then having that searchability kind of change the way that we approach meetings and conversations.
Do you feel like that's here to stay? And even, I guess, more of my question is, where do you think this might take us and relative to Otter, the way we work, or even just kind of you know, the part of AI that's working on language and audio data related things.
Do you see any trends or any places that you think we're headed?
Yes, we do. I don't think this trend will stop in terms of using AI to help people capturing information, sharing information.
With Otter, now During the meeting, all there is your new meeting assistant.
People will have the peace of mind that They don't have to write down everything themselves.
They know that Otter is doing it for them.
So gradually people would just take it for granted that they will assume order will always be there and whatever they want. herd, they can always retrieve them later.
And on top of that, not just to retrieve the original thing, otter will actually use NLP to analyze the conversation as well.
If you look at your outer nodes, outer can already recognize summary keywords, for example, can detect the topics you are discussing.
And then over time, order or whatever other tools in this domain can summarize your conversation, can detect important information.
For a product manager, for example, in this week's meeting, an author may remind them of the action items They discussed last week and can help you manage the agenda for this meeting.
And suppose you're not in the meeting, but somebody mentioned your name in the meeting, maybe you'll get automatic notification.
Right, right. Okay, say, Michael assigned this action item to you, Noah.
It's due on Monday. So that notification can be sent to you directly. from honor right yeah there's a lot of a lot of potential i am i'm preaching to the choir i know you know but there's there's so much potential now to act on all this information it's something For folks who want to find out more about Otter.ai and try it for themselves,
Obviously the name of the company otter.ai is the website, but also they can just go get the mobile apps.
Yeah. You can use ordered AI in your web browser or you can download it from Apple App Store or Google Play Store onto your mobile device or iPad, Android Pad.
So again, it's cloud-based service. No matter where you use it, you can search for information anytime, anywhere.
Fantastic. Well, Sam Liang, thank you so much for taking the time to join the podcast.
And it's remarkable what you guys have done in four years.
And all the best on the next For and Then Some.
Thank you, Noah. Really my pleasure to be here.
Thank you. Thank you.
