Innovation is cumulative.
And by that I mean that the ways we solve problems now couldn't be effective if not for the ways we solved them before.
And while these days, the words training data are typically used in the context of AI development, it's worth remembering that users are also consumers and retainers of enormous amounts of training data from years of discovering and adopting every piece of software they've ever used.
So while we're busy obsessing over use cases and new features for our own AI products, users are following a different script.
They're operating with preferences, habits and, most importantly, expectations they've picked up since the very first time they opened a web browser.
My guest today is Dhruv Bhatia, co-founder and chief scientist at Utori.
As you're about to hear, Dhruv's experience in AI research development, training and leadership spans over 20 years.
So, as you can imagine, he's got more fascinating insights on the tech than we could possibly cover in one episode.
With that in mind, when I asked Dhruv what he'd be most like to communicate to product leaders, he didn't hesitate.
He told me, it's that AI capabilities are extremely jagged.
And you're about to hear exactly what that means for your users, for your organization and for the near future of product.
Let's jump in.
Oh, by the way, we hold conversations like this every week.
So if this sounds interesting to you, why not subscribe?
Okay, now let's jump in.
Welcome back to the Product Manager Podcast.
I'm here today with Dhruv Bhattaraj.
He's the co-founder and chief scientist at Utori.
Dhruv, thank you so much for joining me today.
Of course.
Thank you for having me, Hannah.
So let's get started with a little bit of background info.
Can you tell us a little bit about your background and how your journey through AI research, from deep learning into today's generative revolution, has shaped your perspective on where we are now with this tech?
So I'm an AI researcher, been in the field almost 20 years at this point.
AI research in modern discussion seems to start around the 2022 ChatGPT revolution.
I entered the field in 2005. before the last epoch of deep learning.
I got my PhD at CMU working in core machine learning problems applied to computer vision problems like detecting objects and images.
Over the years
I've built chatbots, built the first systems that could answer questions about images, hold a dialogue about images.
I was a professor at Georgia Tech for many years.
I created the deep learning class.
I also spent eight years at Meta.
I was a senior director. leading FAIR Embodied AI.
FAIR is Meta's fundamental AI research division.
Embodied AI is AI for robotics and AI for smart glasses.
So one of my teams at Meta had built the earliest version of an image question answering model that shipped as a multimodal assistant on the first version of Ray-Ban Meta sunglasses.
Other teams of mine built the world's fastest 3D simulator for training virtual robots in simulation before we deployed them on the Boston Dynamics smart robot.
I've sort of seen the spectrum from computer vision chatbots robotics, and I've just sort of fascinated about intelligence and building intelligent systems.
And that's what has taken me on my journey today to Utori.
Clearly, you're a very qualified person to speak on this topic, which is something that I think all of us want to know as much as possible about.
And I'm really excited about today's topic because we're going to be looking a lot more closely at the expectations versus reality when it comes to the state of AI technology right now, which I feel like you need to have a certain set of qualifications to really be able to speak to this topic and answer some of these questions that are really on our minds.
So we're going to be looking at it today from three angles the user side, the business side and the technology side of AI.
Starting with the user side,
Right now obviously, we're caught in a huge hype cycle around AI, but users can often experience some really wildly inconsistent results depending on the tools and the use cases that they're pursuing.
So what do you think is causing the gap right now between what people on the user side expect AI to do and what it can actually deliver right now?
I think that's a great question.
This speaks to a problem that lies at the heart of, you know, not just building products, but also AI research.
And that gets to this nature of what is often referred to as the jagged nature of intelligence.
As with many of these topics, there's a famous XKCD comic of a PM-like figure asking an engineer-like figure to hey, can you build me an app?
Every time someone, a user, takes a picture of, I want to know whether that picture is in a national park.
And the engineer responds, sure, that sounds like a simple GPS-based lookup query into a database.
Give me a few hours.
This should be doable.
And then the next sentence from the PM is, let me know if the picture is off a board.
And the response from the engineer is I'll need a research team 50 million in five years and maybe we can answer that question.
Now, The specific example in that story is no longer valid.
Computer vision has made enough progress that now we consider detecting species of birds or dogs a solved problem.
But I think the point it's trying to illustrate is that there are extremely sharp transitions from trivial problems to impossibly hard problems.
And that sharpness is difficult for people to conceptualize, to predict.
This is true not just for users of technologies, but it's also true for builders of technologies, builders of products, and it's also true for AI researchers.
It's not really the credentials that matter, but when you spend time building this technology.
Different researchers end up building mental models of what machines can do and cannot do.
And in today's world, for example, we joke about how we have built chatbots that can answer International Math Olympiad questions.
Yet simultaneously, they make mistakes like saying 9.11 is greater than 9.9, that no human would do.
But that just is a kind of mistake that chatbots make.
So where does that leave us?
First of all, why does that happen?
Where does that leave us?
That happens for a few different reasons.
We are building intelligent systems that are at a different point in the intelligent landscape.
And humans approach intelligent systems with their human understanding from dealing with other humans.
When I talk to a person and they tell me that they have gone to high school or college or a university or have a PhD at different spectrums of their expertise, I expect different things from them.
When someone says that they have a PhD in chemistry, I don't expect them to make a mistake of the kind that you know.
911 is greater than 99.
I just expect them to be mathematically numerate, generally well-informed about the world, and so on.
Those expectations break down when we're dealing with AI systems.
And they break down because we cannot rely on the same shared assumptions.
Performance on certain tasks requires training for those tasks.
And even though we have built general purpose systems now over the last few years, there is a very specific thing that we mean when we say generality.
And that makes it very hard for consumers to build mental models.
And so there is this, perhaps a frustrating experience, where people arrive at a product it says it can do a lot of things.
You ask it to do the thing that is listed on the website from the manufacturer of that product, and maybe it does it.
You ask it a slight variation of that question and it's unable to do it.
And so that can be a frustrating experience.
Absolutely, yeah.
And this is such a new consumer behavior as well, whereas we're kind of trained on features that are very specific, very intuitive, very easy to use.
There's definitely this matter of people calling upon their existing training of interacting with chat or with another human being and applying those expectations to a feature that's kind of largely undefined.
We don't really quite grasp the limitations in different, you could say, competencies.
So yeah, very complex technology that we all are kind of learning how to use together.
So when we think about everyday tasks that AI could automate things like booking travel, managing schedules, those kinds of things What makes tasks like that harder to solve than people assume?
So I'll use Utori and what we're building as an example of these things.
So at Utori, we're building personal assistants that can automate mundane workflows on the web.
Our first product is called Scouts.
It's a team of agents that monitor anything on the web for you.
And it was extremely important to us that we clearly state this expectation that you monitors a piece of information.
You cannot ask it to book anything, buy something for you.
It will not create slides for you.
It will not do your homework.
It will not write code for you.
It's not everything that you can do on a browser.
But what it can do for individual consumers is let me know when my favorite artist comes into town.
They might announce it on a few different websites.
I might go to those websites at some frequency.
I would just like you to check it at that frequency.
Maybe I'm looking for reservations for something that requires filling some lightweight form on a browser clicking buttons.
I would like this agent to do that for me and then tell me what information is available.
Maybe I'm a recruiter and I'm tracking changing roles by a particular set of people.
And if they announce it on X versus LinkedIn versus their blog post, let me know when that happens.
So why is this hard?
This seems like a trivial thing.
Humans sit down, open their browser, go to a particular page, fill out certain things.
Why is this hard?
Fundamentally, these problems are hard because they are what are known as sequential decision-making problems.
You are in a particular state.
Maybe you're on a webpage.
You have to take a few different actions.
Your websites are laid out for human consumption.
Reading code on HTML.
It's wildly inconsistent about how buttons are annotated or labeled on webpages across websites.
And so fundamentally, this is a perception problem.
You have to click a button, then something happens.
You maybe scroll the page, fill out something, something happens.
Any mistake that you make along the way simply cascades.
And earlier mistakes lead to later failures.
This is a similar kind of problem that robotics and self-driving industry dealt with.
We realized that if robots make a mistake, those mistakes cascade on themselves.
If you're slightly deviating from a lane.
Now you are no longer at the center of the lane.
You have to have a course correction maneuver.
Similarly, browser automation agents that we are building.
If you have ended up in a part of a webpage that is either hung or you're not supposed to be there, you're not going to find that response.
So there's an error recovery that you have to learn when you're operating out in the wild.
There are also read-only tasks versus write tasks.
If you fill out a form and you click Submit, some websites will not let you go back to fill it out again, which means that's an irrecoverable mistake that you have made.
Training for irrecoverable mistakes is difficult.
And for that, you have to create replicas of the real world.
This is what roboticists deal with, with creating 3D simulators of the world, almost like virtual gameplay, training virtual robots and simulation, then deploying them in the real world.
This is what we do with browser automation agents as well.
When we have to train them to fill out a form and click submit or maybe buy something on the web.
Those are going to be irrecoverable errors if you make them.
So you have to train in simulation.
Those are some of the things that cause these problems to be difficult.
And it's often difficult to know, or to know what you did along the way as an AI agent that contributed to your successful failures.
And that's known as the credit assignment problem.
All of these things that, as humans, you know At this point, we're all so well trained to do some of these procedures.
It seems like a simple task, but on a technical level, we're looking at a much more complex.
I know that doesn't even take into account things like preferences.
You know what time, where in the restaurant do you want to sit at the bar?
You know, there's all these other kinds of matters that I can imagine are just impossible to do from a coding perspective.
Here's a small example I'll give that communicates this.
Humans are used to certain design patterns.
For example, when you go on a webpage and maybe you're trying to book a reservation or book an appointment.
There's a common design pattern that if a date or a time slot is grayed out or has been struck through, you understand that it is not available.
Even though there is no text above it or around it saying that this slot is not available.
You understand it because you've been exposed to that design pattern, online websites that you have seen or in various pieces of writing that you've seen.
Across the things.
How do machines understand this?
Well, they may have read a lot of books, but you have to interact with websites to understand that grayed out text means something.
And this is just one example of the kinds of design patterns that are meant for human consumption, that machines have to absorb, that clicking on this button will not do anything and there's no text describing its purpose.
You just have to know what this means.
Yeah, this is so interesting.
This reminds me of a conversation way back with Nima Priyal, who's the founder of Horde, when we talked about the evolution of user behavior and how these incremental changes and how we understand UX elements and just the general layout and design of websites and technology over the years is kind of this compounding asset that all of us really take for granted.
And this is something that's very difficult for us.
It's kind of a shared language at this point that we've kind of shared over many, many years of technology progression.
It's very difficult to communicate to a machine.
So I just think this is such a fascinating area.
And I want to dig a little bit deeper into the consumer behavior side as well.
So as an extension of some of these kinds of behaviors and patterns that we've kind of internalized over time now this is an ongoing process.
So what are some of the shifts in how people interact with technology that product leaders should be preparing for in the near future?
The emergence of AI products on the consumer market, has certainly shifted people's expectations.
There are now children growing up who will just expect to be able to talk to machines.
There's always this Futurama episode or science fiction thing that children growing up in technologically advanced civilizations, if they're exposed to an older technology, wonder why can't I talk to my TV?
Why is this not understanding me?
I think we're seeing that shift in expectations from consumer behaviors as well, that you just you feel like you want to be able to express yourself.
You feel like, I should just be able to talk to the machine.
It should have a certain general purpose capabilities.
It should be able to hold a coherent dialogue.
It should understand my usage patterns.
And that's kind of what motivated our work and vision and tutorial as well.
We see the evolution of the web in the last 30 years as just incremental advancements over a core technology of connecting content and services to humans.
The web is primarily laid for human consumption.
And that has been because it has just been human eyeballs on the web.
Now people just expect to be able to tell the machines what they want the machine to do on their computer and on their browser.
Why should I, as a person, sit down?
Click buttons, fill out my name, my address, credit card details in order to buy something or to procure some information?
This should be something that should be automatable.
And I think that's what we're seeing as a shift in consumer behavior.
This idea that I should have 30 tabs open to search for this one item that I'm looking for.
Read 20 different reviews.
Already people just want to ask a deep research system or a monitoring system, why don't you let me know when this happens?
The next step after that is okay.
If you have told me that my favorite artist is coming into town and they're performing this on Friday, why don't you go ahead and buy the tickets for me?
Why am I sitting there mucking through forms and things?
I think the shift in consumer expectations is moving up a level in abstraction, talking to software, expecting software to automate the mundane pieces of their lives.
And it almost sort of becomes a task list with superpowers.
And I think there will be a notion of proactivity.
We don't want to sit down and explain every single time.
Here is who I am.
Here is what my preferences are.
You know, there's a notion of memory.
Once you understand memory and personalization, why don't you do something proactively?
Why am I having to sit down and say things?
So it's sort of like everybody gets an assistant and a superpowered employee or a chief of staff.
Well, and you can see how some of the technologies that we interact with every day kind of contribute to that as well.
You think about the For You page on TikTok, where there's the technology that learns your preferences, learns the things that you're likely to interact with and engage with?
And we kind of apply that same logic to the technology that we're using now that we know knows a lot about us.
We know knows a lot about our preferences and habits.
So yeah, it's interesting how kind of our technology landscape is sort of training some of these expectations.
I think that those are interesting relationships to be paying attention to in terms of anticipating what consumers are going to expect, which I think is a great segue into the business side.
So right now, There's no surprise here.
We see a lot of companies that are rushing to market with AI products and AI features, and they're very quick to promise transformative capabilities to varying degrees of success.
What would you say are the biggest mistakes that you see product teams making when it comes to scoping and positioning their AI features?
I think this, again, goes back to the question of the jagged nature of intelligence.
You have to be extremely careful.
This affects not just the consumers, but also the builders.
You have to be extremely careful not to promise the sky, because you will not be able to deliver it on day one.
Yet at the same time, the expectations of generality from consumers are climbing.
They expect you to be able to not be extremely narrow applications, because ChatGPT answers any questions.
So why don't you do anything?
And so there is this trap of falling into this design pattern of a text box as the entryway into anything, and you don't tell your user anything.
You promise the world, my agent can do anything.
That leads to frustration from the users because they're not going to.
You know.
First of all, they are dealing with the problem of just a blank canvas.
What are the kinds of things that I can ask here?
And if I'm not calibrated, I will ask for things that the agent will not be able to do, and they will be frustrated and your users will cheer.
This is one of the reasons why, for our first product, we crafted a fairly narrow scope of capability.
Scouts are agents that monitor anything on the web for you.
They don't log into services and they don't take right actions.
It's a read-only monitoring product.
However, we did not say this is Amazon price monitoring or this is Ticketmaster event monitoring.
Any digital information that is available on the web that you could open up a browser and get access to.
These agents will tell you and you will get an email whenever that happens.
So just tell us in natural language and at whatever frequency you'd like to be monitored.
The reason why we did that, it was extremely important for us to deliver this capability.
This is a read-only capability.
We're not making irrecoverable mistakes.
If we make a purchase decision on your behalf, you will get frustrated if the wrong thing is purchased for you.
Yet there is a certain generality here in the kinds of queries you can ask and the surfaces you may be expecting this information to appear in.
And from there on, you have to climb the staircase of trust.
Initially, we delivered value without asking you for any logins on credit card information.
However, after you've seen certain value, your users will naturally expect you to be able to do more.
If I'm tracking an artist coming into town, the next step is get me a ticket.
If I'm tracking the availability of a restaurant reservation, the next thing is to make that reservation.
If I'm a recruiter and I'm tracking the movement of a particular candidate, the next step is drafting an outreach email.
And so from a builder's perspective, I am a newbie.
I'm an AI researcher.
I don't feel like I'm in a state to be able to give advice.
I can just point to the caution that we adhere to, which is that there is a jagged nature of intelligence.
You're not going to meet some tasks.
You're going to be able to solve some not.
Typically, the tasks that you are able to solve are the ones where you have the ability to practice.
And therefore, it has to be tasks where the mistakes are not too costly.
And you build from there. incrementally.
Okay, this is very wise words.
And I see this often where I completely see what you mean in terms of frustrating the users with having limitations that are just opaque to the user's experience.
They're walking into a chatbot and they're going to naturally interact with it, the way that you would expect to chat with a live agent.
And that can lead to a lot of frustration which there's a cost benefit analysis to not limiting the scope of what's possible and taking the risk that your consumers are going to take a wayward turn and have a lower degree of faith across the board with the technology.
And may churn.
If they don't find value in what you said you could do, they will have suboptimal experiences and they will churn.
They will not come back to you.
Yeah.
Okay.
So let's dive a little bit deeper in that.
I kind of think the trust is sort of a core element of why people are churning at that specific critical moment.
How should product leaders think about building trust with consumers without overpromising the capabilities that aren't ready yet or that they can't deliver on?
I think this goes back to the previous question that we were talking about.
People have to see value before they hand over credentials for sensitive information like credit cards.
If you're asking consumers as a very first step and there are certain apps out there that have tried playing this where give me your calendar, give me your email inbox access, give me a couple of other logins, before you even see what this product is capable of doing.
That is an extremely risky strategy.
It might make you go viral on social media, but imagine yourself.
Put yourself in the shoes of that user.
Do I really want to hand you my work email that has sensitive documentation on it or my credit card, without knowing what you can or cannot deliver?
And so that's why where we started was no authentication, no writing or no changing the state of the world.
Initially, it's just reading.
And what that means is that you can actually.
You know, AI isn't promising you 100 accuracy, which means you can retry if you have made mistakes.
Retrying is possible in a read-only product.
It's not possible in the right product with irrecoverable mistakes.
So those are the kinds of things I would and we certainly keep in mind when we have to climb that stages of trust with our users.
I can see this kind of notion of.
This is an unrelated example, but I feel like it illustrates a similar kind of point.
I had a friend who moved to Canada from Brazil and she thought Canada is the safest country in the world.
Everywhere that I go is safe.
And then her first week in Canada.
There was a robbery on her street and suddenly she thinks everywhere is unsafe.
So it's the similar kind of notion of... Time to value, time to delight.
Time to delight, time to value, but also how fragile and kind of fraught that initial trust-building period can be when something interrupts your expectation of what is going to be delivered and kind of shakes that foundational trust.
So i want to dig in then to the technology side.
I think this is a really good time to kind of switch it up from your perspective as an ai researcher.
Which problems would you consider largely solved today and which would you say are and i know that this is loaded language to say it's on the near-term horizon, because that can always shift but what feels like it's very much on the near-term horizon versus, you know, let's say, perpetually a few years away.
The concrete Example I'll use here, because it's on my mind, is of a problem of answering questions about images.
The reason why this is on my mind is last week I was at a conference, ICCV International Conference on Computer Vision.
My collaborators and I received what is known as a Mark Everingham Prize for work that we had done a decade ago.
The work was called visual question answering.
We introduced to the community a data set, a task, a benchmark and a set of methods for building the first generation of agents that could answer any open-ended question about any natural image.
And over the last 10 years we've helped the community track progress on this by organizing annual competitions.
We stopped organizing that competition in 2021 because initially, when we started in 2015, most methods were incredibly poor, as you can imagine, at this task of answering questions about images.
In 2021, on that data set that we had created, we basically met human accuracy.
We had reached inter-human agreement in answering these questions.
And so we stopped organizing that competition.
And, as I'd mentioned, it's an interesting lifespan where, in the course of last 10 years, I found myself leading a team at FAIR that built modern methods, that shipped on Ray-Ban meta sunglasses, where you can invoke an assistant and you can say hey meta, take a picture.
Tell me more about this monument.
That problem is a good loop closure from 10 years ago.
And when we started, there were just a host of problems that were completely open-ended problems.
Answering questions that required reading text in the wild was hopeless.
If you asked, what does the sign say?
Most of those methods weren't running OCR or optical character recognition.
And so they couldn't read the text on the sign.
And so what methods are having to do is answer based on priors.
What do signs usually say? signs say stop or go.
And so they would just make common guesses.
We found that this was a common problem, that most image question answering models were heavily dominated by linguistic priors.
So if you ever took a picture of bananas and you asked what color are the bananas, the model was most likely going to say yellow, because most bananas in the world are yellow.
It knows that from the training data set.
It actually can't see very well.
You can think of it as squinting at the image.
And so it could be a picture of the green banana.
It'll say the banana is yellow because most bananas are yellow.
We had extended question answering to dialogue with chatbots and co-reference was an extremely hard problem.
If you ask, is there a person in the image?
Yes.
What are they doing or what is he doing?
Now that is a co-reference into a visual entity that we just talked about in the previous round.
That was a hard problem.
The model was already confused.
It didn't know what he referred to or what they referred to.
Those problems today are considered solved.
To the degree that we can measure, these problems are not open problems.
However, there are still some problems that are extremely open problems.
Counting objects and images is still an open problem.
Take a picture today where there are more than 10 people in a crowd.
And I say 10 because small number of objects you can make reasonable guesses on.
But take more than 10 people or a crowd shot, upload it to your favorite chatbot.
Ask it how many people there are in the image and just look at what response you get.
And that is still an open research question.
Asking about 3D spatial understanding.
For example, take a picture where there's a table at the far end and maybe a bookcase closer to the camera.
Ask it about the height of the table and the height of the bookend.
What most chatbots today will answer is pixel heights.
Like whatever seems closer to camera, they will say that one is taller because they don't have a spatial understanding of depth, that things that are really far away can actually be taller than things that are closer to the camera, because there's a 3D world behind the scenes.
Those are still open questions.
And more closer to AI and building agents, for example web agents.
There is still a notion of drift that happens in agents over time.
That you may.
You know Utori builds monitoring agents right.
Our monitoring agents are running for months monitoring certain topics.
So you may ask for a particular news topic and you may monitor that over months.
And what can happen is that slowly, the agents will drift into tracking something that is deviating from your original request.
Because nobody has built agents that run for months at a time and then we can do credit assignment and reward them for success or failures on those long horizons.
This is a really interesting pocket here, because I hadn't thought about what is the criteria for how we tell an agent that has done something correctly.
Generally speaking, when you get a correct output, you move on with your life.
You don't necessarily say good job, that was great, or give it details about what it did, correctly or not.
So this is interesting about feedback loops for the technology.
Is there a user behavior that we should be adopting in order to better train the models that we're currently depending on?
Or is this something that I feel like this could be a whole other show?
But how do we close the feedback loop?
Internally, we do evaluations and feedback at multiple levels of abstraction.
So we will browser automation agents or browser use agents.
We have to manually annotate, for example, every click it does.
Was it the right thing to do at that webpage or was it the wrong thing to do?
Now that you can imagine is too low level.
It's too noisy.
Sometimes a task can be done in multiple ways.
Maybe sometimes you search, type an item in a search query on a webpage.
Maybe sometimes you click on a tab directly to go to that source of information.
So that's too low level, too noisy.
However, after a task is done, maybe you asked an agent for whether a 630 pm reservation was available at a restaurant or not.
And it clicked through a bunch of buttons, found a restaurant and said yes or no.
At the end of that trajectory is a proof of work.
And in many of these tasks, there is what is known as a generator-verifier gap.
Verifying proof of work is easier than actually solving the task.
Because solving the task requires clicking lots of buttons, typing the right thing in the right search box, going to the right place in the website.
But if an agent comes back to you and says yes, I have found a 630 pm reservation, there's a screenshot of a webpage.
We can read that webpage.
We can see whether the agent was right or not.
And so we build evaluations based on that.
Then, of course, there are consumer-facing feedback mechanisms where, every time our agents send emails to people, people can do thumbs up and thumbs down.
This then naturally ties into personalization.
Sometimes there aren't right or wrong answers.
Sometimes there are just preferences.
If you are tracking a certain news item and maybe you're just bored of a certain direction that news is heading into.
If you were talking to a person, you would say no, I want to see less of this.
I want to see more of that.
And ideally, you want to say that in natural language because that's convenient.
And so you have to build in feedback mechanisms that take in natural language feedback and change the course of agents operating in the future.
This is so interesting.
I'm thinking about this in terms of also how people tend to look at the outputs of something like ChatGPT through the lens of are we happy with this as a service, rather than are we happy with this as a technology perspective?
So, for example, if I'm next to a colleague and the two of us are generating, let's say, meal prep plans for the week, and I generate a meal prep plan and they generate a meal prep plan and it's the same meal prep plan, but I feel like it's not really got the right macros for what I was trying to target.
And it's just from a service perspective, it's a thumbs down.
But the person next to me says, well, they did exactly what I asked.
The output is technically correct for the specifications in the prompt, this is a thumbs up.
It sounds very confusing.
And there's a real nuance there about how we understand what our role as givers of feedback to these machines is.
That's a whole other data set that can be very difficult to work with.
The traditional notions of A-B testing.
You know, anytime you're building a feature, split your users into two groups, show one set of responses to one or, as strategy PT does show parallel responses to the same user.
Ask them to pick one.
That breaks down in a lot of cases.
For example, anytime you have to execute a workflow and buy something or do something, there's no A-B test, because you're not going to do the same thing two times.
Your user is frustrated by that.
And the second sometimes thumbs up and thumbs down or A-B reactions are just too coarse for a user really to convey what they're trying to say.
And Sometimes you want to give them the ability, if you're sending them textual responses is to highlight something and say not this or be more editorial in their feedback as they would to another human.
You know, if you were taking a pass on someone's Google Doc, you wouldn't just give them a thumbs up or a thumbs down.
Oh, that would be horrible.
Thumbs down.
What part of it is thumbs down?
Exactly.
This is like, you know, going to your editor and just hearing, no, fix it.
Fix what?
Do the whole thing over again?
Yeah.
Oh, wow.
And of course, you know, that's asking a lot.
Even if we did have the capabilities to say, you know, go into an output on an LLM's output or an LLM's response to us and give that kind of critical feedback, that's a lot to ask of a user that just wants it to work.
Generally speaking, it's difficult enough to get people to respond to surveys, even if they've got a referral code or some kind of bonus in it for them.
Yeah, it makes a very difficult co-authoring task to ask folks to help you co-author.
Anyway, what I'm getting from this is that we really need to expect a lot less from users when it comes to hair technology.
And I think they have to feel like that feedback is an investment into personalization of the chatbot, towards something that they want to be able to do.
That the user cannot be your QA engineer.
They cannot tell you all the things that are wrong with the product.
But what they can... the experience you want them to have is that this is my assistant.
And therefore any feedback I give to this assistant is personalizing it and making it better and aligning it to my preferences.
That feels like the right relationship or the engagement mechanism why they might put in time.
Yeah, I would agree.
I tend to feel that I'm a lot more patient with a model that is more forthcoming with trying to learn my preferences.
Like I feel often when you're doing a query in some kind of an LLM And it right off the bat asks for questions to further refine, even if they seem a little trivial to me, it kind of reinforces this behavior that whatever I'm telling them has to be a lot more specific than I anticipate in order to get the output that I want.
And it kind of prepares me for disappointment if I'm not quite on base.
So I think that this is something important to kind of keep in mind when we're developing features that require some sort of give and take from a user in order to kind of get the output that we all want.
This is where we go back to the climbing the staircase of trust.
If the very first thing you do to a user who is just trying to experience your product is give them a 20-question questionnaire, they're just trying to get to delight and they want to build a quick model of what you can do, and then they will iterate from there.
So it can't be the very first thing you bombard your user with but as they see some value you can inject in those questions and preferences.
I'm curious at this point, why now?
Why have you started Notori now?
Why is this the moment that you've decided to start this project?
And what's fundamentally different about the landscape right now that makes this possible to exist versus.
You know, you've been in the game for a long time.
Why not then?
I am, of course, susceptible to hindsight bias, but my feeling is that this is a unique time in building certain kinds of AI-powered products that we couldn't build in the past.
I was doing robotics before.
I could have spun out and started a robotics startup, and there's no shortage of those.
There's plenty of those.
I don't think this is the right time to start a consumer or an unstructured-focused robotics company, because the kinds of problems to be solved there are still decades ahead of us.
I think people forget.
2004 is when this government agency called DARPA organized the DARPA Grand Challenge, asking universities to build an autonomous self-driving cars that could drive from point A to point B in a desert.
2004 is the first time that was organized.
No car finished.
I was at CMU at 2005.
CMU, you know, went the furthest in the first attempt.
2005 is when multiple university teams finished.
Late 2000s is when a bunch of Stanford researchers get absorbed into Google.
It becomes initially Google X and then mid 2000s, the Waymo project.
And then ultimately, in 2023 or 2024, is when we get consumer-facing apps where, in San Francisco, you can call a Waymo to your doorstep or to certain pickup locations.
Think about that journey from 2004 or 2005, which was the first research prototype demo, to a consumer-facing product that is available, at least in certain geographic locations.
It's still not been universally rolled out.
Universal rollout is still maybe another decade ahead of us.
That is the challenge of hardware plus AI.
In software plus AI, development cycles are much closer.
We are finally at a place where AI systems can talk to people.
So there's broad knowledge of the world being able to hold a dialogue.
Perception systems have matured, at least on the web.
So we can take a screenshot of the web and we can know how websites are laid out, which buttons do what.
There's certain broad-based common sense understanding.
And finally, the third thing open source models have been released over the last couple of years which allow smaller players such as ourselves to at least get started.
A few years ago, if we had to build a web automation or a web agent startup, we would have to start from pre-training.
Pre-training of language models and vision language models is an entirely different enterprise with an entirely different set of capital investments and compute requirements and data requirements.
Today we post train models, meaning that we start from open source models that have been open source vision language models and we post them for browser automation, clicking buttons, filling out forms and so on.
That would not be possible a few years ago.
And simultaneously.
This is not a problem that has long iteration cycles and is still a decade plus ahead of us.
The way I think of it is.
There is no possible world in which robots have arrived into our homes, but we're still sitting on our laptops typing our names into browser fields.
Like digital assistants will arrive before physical assistants do.
And digital assistants, because it's a purely digital realm, the world of bits just moves faster.
It has faster iteration cycles.
And a lot of the substrate on which we can develop these intelligent systems has commoditized, so we can focus on the last mile problems.
Oh, that's a very eloquent answer.
Also love the use of the word substrate.
This has been a thoroughly fascinating conversation.
I feel like we could have gone so much deeper, and I'm sure that a lot of folks would love to do that.
So where can folks follow your work online?
I'm available personally at dhruvbhatra.com.
That's my webpage.
My work is available at utori.com.
And our product is called Scouts.
It's available at scouts.utory.com.
Amazing.
Well, thank you so much for joining me, Dhruv.
I really appreciated this.
Thank you for having me.
This was wonderful.
Next on the Product Manager Podcast.
Leading product in the age of AI means solving many of the same problems in very different ways.
From development processes to distribution tactics.
Just about every playbook that worked a year ago is already outdated, which Webflow CPO Rachel Wolin considers both a challenge.
You'll get the answers and clarity you've been waiting for on topics like answer engine optimization, build versus buy and the right way to enter the AI market.
Subscribe now so you don't miss it.
