At this moment in time, the most coveted product jobs are all about AI.
And historically speaking, it's a particularly unusual moment because the core competencies required to compete in this arena have only existed for a couple of years.
Stranger still, the field has evolved so rapidly that any experience you might have gained when chatGPD first hit the mainstream — barely two years ago, by the way — may already be outdated.
So, how can PMs looking for jobs in this space get up to speed right now?
And maybe more importantly, what habits can you implement today to stay current as the technology evolves?
My guest today is Loraelyn Mckew, lead AI product manager at Superhuman.
Loraelyn's career journey is pretty freaking incredible, so I'll let her fill you in on that shortly, but it's also a testament to the real key skill you need to be successful as an AI PM — adventurousness.
So as you listen to Loraelyn's tips, I suggest getting in the mindset of an adventurer because this is a field of uncharted territory, and the first thing you'll need to succeed is a willingness to explore.
Let's jump in. Welcome back, listeners to the Product Manager podcast.
And I am here today with Loraelyn Mckew.
Loraelyn, thank you so much for joining us today.
Oh, thanks for having me.
So you have had a really interesting journey to becoming a lead AI product manager at Superhuman.
And I think that we usually try and kind of linger just a little bit on this section.
But I would really love if you could tell us a little bit more about your background because it is so cool.
My background is very, very strange.
So I started as an Apache helicopter pilot, which is like, you know, obviously a very normal entry route into tech.
I was in the army. I did that for about six years.
And then I went to teach physical education at West Point for the last part of my time in the army.
Then I went to business school and ended up at Slack as a product manager, first on their growth team and then on their platform team.
And then I went to a small machine learning startup called Empira.
And now I'm here at Superhuman working in AI.
Such a cool career trajectory.
I'm just obsessed with it.
We'll get right into things because today we're going to be focusing a little bit more on just AI and kind of how a superhuman came to be and the whole journey that you've had to becoming a lead AI product manager.
So if we talk about AI kind of in the context of superhuman, can you tell us a little bit about a moment when you saw AI significantly evolve and elevate the user experience for superhuman users?
Yeah. So let me tell you a little story.
For me, the most important thing is to save users time and to let their brain do more important things.
So for me, superhuman AI lets your brain do more important things.
I'm going to tell you a little story.
When I was in Iraq, I was given this one mission to fly in circles around a base for three hours.
I just flew in circles around a base for three hours doing nothing.
But then eventually we got a new mission, which was to go fly really low, find these bomb caches and then blow them up, which was way more interesting and a much better use of the most technologically advanced helicopter the army has ever produced.
And in a really, really weird way, I think that I fell in love with the concept of AI before it was even a thing because I could see how much better use of our helicopter was to use the second mission versus the first mission.
So AI is kind of like that and that AI takes care of the circling around the base for three hours at a time and lets you take your own human brain to do the more interesting tasks.
So AI is a huge time saver for users.
When we looked at one of our features called instant reply, which is a little quick reply that you use to send a response to someone.
It saved users two minutes an email to reply.
You don't need your brain to spend two minutes to compose a reply when probably something that's kind of boilerplate would work for most of your emails.
To me, that was like this big aha moment that like, okay, AI can take the drudgery off your plate, give AI all the road tasks, let it circle the base for three hours and you go to the cool stuff, you go find the bombs and blow them up.
No one has ever used that analogy on this show before, but I'm here for it.
Glad to be the first.
Okay, well, we'll come back to AI in a moment because there's just so much to unpack there.
There's so many different use cases that we can really get into.
But I want to talk a little bit about your career.
So when we talk a little bit about your career progression, going from Slack and along that line into AI product management, how have you kind of come to wrap your head around AI as part of your career, like kind of going from like your initial role to where you are now, and how has the integration of AI
into kind of the fold affected your approach to product strategy?
Yeah, that's a really good question.
So I think the most important part for me has been to optimize for learning.
That is something I learned at Slack, but it is something that is it's maybe 10 times more important for AI, you have to optimize for learning.
So a good example of this is we launched a product called Superhuman Ask AI recently.
And this feature is so cool.
Instead of searching a specific search query, you can just type in a question, you can say, hey, can you please summarize the most recent feedback on instant events?
Can you give me the top three most positive quotes in a couple of negative quotes?
Something that would take you 20 minutes to do now takes just seconds, which is amazing.
Or what am I meeting with Hannah or when did I last email so and so?
So we were initially going to take like three or so months to build this product.
But what we decided to do is put out a version after one month, which did not feel ready for us.
And we put this version out to a small group of people.
First of all, we put it out to our AI team, then we launched it internally.
And then we launched it to an increasingly larger amount of beta users.
And what we found is that each phase, we learned something new that completely changed the way we thought about the product and the way we were going to build it.
So by the time three months have passed, a lot of people had used this and we finally built the right thing, which we might not have done from the beginning.
AI is such an unknown territory that you really have no idea how people are going to use it sometimes.
And so you have to optimize for learning, you have to launch a little bit, and then iterate, and then launch, and then iterate, and then launch and then iterate.
Yeah, I can see how that accelerated cycle, it gives you so much more almost failure data to work from, which is gonna be oftentimes so much more useful than just, you know, if you have nothing at all, right, and you've just taken such a long time to kind of exhaust your runway.
That's really interesting.
One of the things that we've mentioned, we had a conversation before, and you really emphasize the importance of really baking AI into the product strategy rather than kind of like slapping it on top, which we kind of see often with some of the AI features that are integrated into current products.
So what were some of the decisions that you made to make sure that the AI in the product felt really seamless, fully integrated and not just kind of like bolted on?
Yeah, I think today, a lot of people are putting AI in a sidebar, like this little chat area, you have to integrate AI seamlessly into your product.
My dream user interview goes like this, hey, have you used this AI feature?
Oh, I didn't know that was an AI feature.
I think that's so exciting.
Because when you integrate AI seamlessly into the product, it just feels natural.
It feels like what you need is there when you need it.
So a good example of this is we just yesterday launched a feature called instant event, which I'm super excited about.
Personally, especially you get an email, we're both parents, so like you get an email that's like, hey, don't forget to bring this to school on this day.
And you're like, Oh, now I have to go open my calendar, see what I'm doing that day create an event.
It's really frustrating.
But now in superhuman with one click, you can just have an event created with the right date and the right title and write all the details that are there, the right people in there.
It's so useful. And when we first launched this feature, we had it in not a seamless location.
So we had it in command K, which is kind of like your command menu for superhuman.
So you had to know to hit command K and then type what was at the time create event with AI.
Obviously, like very few people are going to find this.
But because we had it there, we were able to test it and do that first principle of optimizing for learning.
So we were able to do this on a bunch of emails ourselves, figure out what was working, what are the weird edge cases.
But that wasn't the final step.
We had to put it somewhere that was seamlessly integrated into the product.
So now it's in multiple places.
The first place is you can use the normal calendar shortcut, which is B in superhuman, and it'll automatically create an event with AI instead of just a blank event, which is super cool.
It also has a little button on the bottom of an email.
If a date is detected that says, Hey, do you want to create an event on this date?
You click that button, and it pops up.
It does not make that noise.
By the way, I did not win that battle.
Or when you hover over date, that little cursor will change into a hand.
And when you click on it, you'll notice that an event was created, which is so cool.
And so we wanted to give the functionality a try, put it in command K, but the final version was so much more seamlessly integrated into the product.
And we're just finding that users are using some of the ways we didn't even announce in our announcement email.
They're just finding it naturally, which is exactly what I want to happen.
Like you see a date, you click on it, and an event happens.
It's perfect. Web designers, this one's for you.
I've got 30 seconds to tell you about Wix Studio, the web platform for agencies and enterprises.
So here are four things you can do in 30 seconds or less on studio, adapter designs for every device with responsive AI, reuse assets like templates, widgets, sections, and design libraries across sites and share them with your team, add no code animations and gradient backgrounds right in the editor and export
your designs from Sigma to Wix Studio in just a click.
Time's up, but the list keeps going.
Step into Wix Studio and see for yourself.
We recently had an episode about empathy, kind of part of the design process and it's kind of interesting how those kinds of intuitive features, I think that's really the key to creating something that's easily adopted.
It's like making those kinds of features so natural and so seamless that people just kind of can't help but discover these really amazing features.
So I want to talk a little bit about data and how we use metrics and kind of shape our decisions around them.
So let's kind of use example of superhumans summary processing automatically so that users don't need to request things.
So what metrics are insights which have shaped this choice in this product?
You might want to give a little bit more background about that feature.
And how do you decide which feature should be proactive versus user initiated?
Yeah, so there's actually two features I'll mention with this because there's a really interesting story about the second feature.
So when you receive an email, before you even open it, we calculate the summary and we calculate some suggested instant replies for that email.
So the automatic summary is a one -line summary which is present in every single email.
So you give a summary upon click, which makes sense.
It costs a lot of money to summarize every single email that you get.
Some of them you don't even open.
But for us it was so important that those two things were available the moment you open the email.
Because maybe you're in a rush like you were saying earlier.
You have to have empathy for the user.
They may not have time to click on that and wait a couple of seconds that it would take to come in.
The second feature is instant reply.
And you asked about what metrics shaped our choice.
We really wanted to use a more advanced model for this feature, this instant reply feature.
And the more advanced model was just too slow.
It was like a turtle.
It was like the turtle bot.
It had these brilliant answers.
It's like a turtle genius.
There's this turtle genius bot and, okay, maybe it makes sense to have the user request them and then they get this turtle bot genius answer and it really makes sense and it's great.
And the summary is amazing.
And the instant reply is amazing.
But they're going to have to wait a few seconds and it may not be ready by the time that they actually click open that email.
Or we have another option which is like, I don't know, it's kind of like your fast, awkward uncle.
It kind of gets the job done.
It makes some mistakes every once in a while.
A great example, before we were able to figure out how to solve this problem, for instant replies, there was this one use case that drove me absolutely crazy.
I would stay up until midnight every single night trying to figure out edge cases like this.
So this particular one was somebody would send a goodbye email like, I'm so happy being at this company, but it's time for me to go.
And then a bunch of people would say like, oh, I'm so sorry.
We missed you so much.
This is great. You've been wonderful.
And then you're like so stupid.
You're kind of fast but kind of not so smart.
Uncle would be like, dear Johnny, thank you for saying goodbye to Sarah.
And you were like, no, that's not what I'm going to email.
I'm going to say goodbye to Sarah.
That's what makes sense.
The super smart girl is over there like, yeah, I know I got this.
Of course. But it takes me long to figure it out.
So we were like, okay, how can we do this?
At some point, we realized we have to use your a little bit stupider uncle.
We chose that, but we figured out how to make it smarter.
We can fine tune the model.
We can send in more examples of that particular case when we're doing our fine tuning.
We can also decide what kind of parts of the thread we send in.
We can change our little bit.
But the bottom line is it had to A, be good enough, but B, it also had to be fast enough.
That was like the most important thing.
Okay. So I want to kind of dig into that a little bit further because this is kind of like a skill that's more unique to AI product management that I think is kind of relevant for so many people who are interested in deepening their skill set here.
So when we talk about kind of like these technical aspects of kind of balancing, you know, fine tuning models, kind of trying to balance latency and cost effectiveness, like all of these decisions, what's kind of like the nuts and bolts process that you have to evaluate when you're kind of making these
kinds of decisions?
And what kind of resources do you need in order to kind of understand better, like how to make really informed decisions when you're kind of looking at all these factors.
I like to use a tool that allows you to put multiple prompts side by side, along with multiple models.
So you'll have prompt A with your super smart turtle on the left, and you'll have the same prompt with your kind of stupid fast on goal on the right, and then you'll run them with a data set.
And just to throw back for a second, how did you get that data set?
Well, you launched earlier, and you had your internal users do thumbs down or thumbs up on the examples that worked well for them.
And that becomes your own data set that you had to test on.
Which, yeah, your brain could probably come up with a data set of examples, but having real live examples from your own internal users is so helpful.
You take that data set and you run those two prompts, you look at the time that it takes, maybe you figure out how long it takes for users to open an email.
Okay, that's my benchmark.
I got to get this ready within like one to two seconds because sometimes people click on emails right away.
Okay, which side is winning the two second rule?
Okay, this side is as money the two second rule.
Okay, let's now look down every single one of the responses, which one's janky?
Oh, that one's really janky.
Why are you thinking, you know, so and so for saying goodbye to Sarah.
Okay, one more column, let's open up on your column.
Now let's change our prompt a little bit.
Okay, determine the focus of the conversation, figure out who is the best person to respond to, you run it again.
Okay, you look at the examples, okay, iterate again, it's just like you have to iterate, iterate, iterate, iterate.
And you may not even know what you're looking for, in the beginning.
But as you do it, slowly you start to figure out what is your criteria for success, you may already have some criteria for success, it has to be this fast, it has to have this amount of accuracy, but like, what kind of accuracy on what kinds of problems you get an intuition for that as you actually do
it and actually play with it.
Okay, so this sounds like a huge emphasis on prompt engineering is kind of like a key skill set to really master in order to be effective in this role.
How did you kind of hone that skill set is with just such a new practice?
Did you have to take a course?
Or was it something you learn on the job?
So two things definitely learning on the job.
I mean, literally, it was me, late at night, trying a new thing, every once in a while ago, like search the internet, like, is there something I can do here?
Like, what is going to solve this problem?
We actually had a close relationship with open AI, and they were able to give us some tips.
Some of them are funny, like, say this in all caps.
Really? I can like, yell at the LLM?
Okay, I could do that, you know, or like, repeat this a second time, or make sure you put it at the end as well.
A great tip was, you know, make sure there's an example there, a one shot example there.
Make sure you're fine tuning has examples of all the different options that you're having trouble with.
Okay, these are really interesting tips.
I think that prompt engineering is an interesting field because, because we interact with these interfaces so much with plain language, it seems like it should be a very natural process in order to create a really effective prompt.
Do you have any other unexpected things that you've learned as you've kind of refined and kind of honed your ability to kind of zero in on the most effective prompts in order to kind of get the result that you're looking for?
Yeah, one of the tricks that I used that didn't usually end up in the final prompt, but I used while I was making the prompt was to have the LLM explain itself.
So let's take that instant reply as an example.
I would say, okay, who are you replying to?
And then I would say, why are you replying to that person?
And then it would say, I am replying to that person because of XYZ.
And I'd be like, oh, that's what is throwing you off, you very silly slow uncle.
Okay, now I can change the prompt a little bit to get a little bit more specific about that.
Okay, explain yourself again.
And kind of having the LLM explain itself helped me understand the thought process a little bit more, so that I was able to make tweaks to the prompt.
Yes, they explain yourself option was pretty helpful.
Yeah, I think that that can be really helpful.
Because sometimes the temptation is just to go back to the drawing board, like, Oh, I messed up the prompt somehow, we got to just rewrite the prompt.
But okay, well, this is really practical stuff.
I kind of want to come back to sort of like the business elements of AI.
Obviously, right now, AI features are kind of becoming table stakes with regards to most digital products.
But there's also a lot of complexities you have to kind of balance as you're kind of integrating AI.
What are some of the challenges that superhuman faced kind of in a business context when deciding which AI models or tools to implement?
Yeah, so those who, you know, know AI, you probably understand that I was referencing GPT 4 versus GPT 3 .5.
So GPT 4 is the like, slowed turtle and GPT 3 .5 was the fast, not quite smart uncle, I will say now, this is not so much of a problem because GPT 4 .0 and GPT 4 .0 many they're so fast, and they're really, really good, which is interesting because the problem that vexed me a year ago is like, kind of like,
this is not as big of a deal now, which is so interesting.
But back then, it was incredibly important, especially when you are talking about a scale on the magnitude of summarize every single email that someone is getting.
Like, that is a huge scale.
Like, that costs a lot of money.
And we're really fortunate and that our CEO, Raul, he is all about quality.
Pick the best option, get the best option.
But what's interesting is the best option isn't always what gives the best response.
It's, you know, also the latency, like what meets the criteria for our users' speed needs.
Speed correctness. So for us, would we have been willing to pay for GPT 4?
Yeah, we would have been willing to pay for GPT 4.
But it just wasn't going to meet the needs of the users at that time.
So okay, well, let's use GPT 3 .5 because it's available at the time.
That is good enough.
And our latency requirements.
And with a lot of prompt engineering, and you know, PM time, and QA time, we're able to get it to the right quality bar that we need to meet the needs of the user.
The business aspect is really important.
But we were willing to pay the cost, whatever it did, whatever it costs to like meet the needs of the users was the most important thing to us.
I'll switch gears a little bit.
I want to talk a little bit more about your career development into AI product management.
Because it's just like I think I've told you before, everybody is really clamoring to just develop the skill set and understand how to enrich themselves.
So their resume stand out in this kind of, you know, like it's like an AI free for all right now.
So what are some of the skills or now that you've worked in the field, now that you kind of let a team in this field?
What are some practical skills or areas of knowledge, or even designations or courses that you would recommend for folks who are looking to develop AI centric products?
I would say I think hands on learning is the most effective thing right now, getting into it, keeping on top of the different trends, the different things that are being released, there's like something new being released every single week, it feels like once that comes out, play with it, use it, I
would especially say, find ways to use AI in your daily life, figure out what works.
So a good example of this is I have a very, very cute couple of boys, and they love coming into my office while I'm working.
Guys, like just give me like a couple moments, I'm almost done, I'm gonna help you.
Well, that never works.
So what they really want is they want me to print them some coloring pages, and then they'll go off and like color it.
Okay, but at this point, I have exhausted every single monster truck and World War Two tank and battle scene on coloring pages .com.
So I like no longer have anything left.
They're like, No, no, no, no, no, okay, well, like let's go to chat GPT, let's see what we can do.
Make me a coloring page of a monster truck.
Okay, bro, are you interested in this?
So does this meet your requirements?
No, I want 10. Okay, make it 10 monster trucks.
Does this meet your requirements?
No, I want them to be racing.
Okay, and then you start like going around.
And then at some point, you know what your kids want.
And you're able to say, make me a kid appropriate coloring page of 10 monster trucks on a track.
Make sure not to include any race cars, you know, you figure out like what works.
And then you're able to, you know, you're able to print it.
But I would say like by finding these use cases in your everyday life, let's a personal example, let me find a professional example.
So I do this with copy sometimes, like, okay, we need to explain to users and one tool tip, how this feature works.
Okay, you can only be so many words.
It has to use this language.
Let me put in a rough example, and then give me like 10 different variants.
Okay, cool. Let me try another one.
There's something about using AI in your daily life.
That is that just I don't know, it gives you this knowledge, this understanding of what's possible.
A new feature comes out, okay, give it a try.
A new technology is available, a new model is available.
Another company comes out like, okay, give it a try.
And a good example of this segment of perplexity and said, okay, list all of the different models that are out there and how much they cost per million tokens, make it a table like there goes like, thank you, that's great.
I don't know you have to just do it.
I think that's honestly the most useful thing.
Is that like a helpful answer?
It's not a course you can take.
But it's definitely something that you can do in your everyday life.
Very, very useful. And I also, I wasn't expecting the parenting tip, but I will definitely take that one.
Yeah, yeah. My son is interested in the same stuff.
So that'll be a fun little treat for the weekend.
Well, thank you so much for joining us today, Laura Lynn.
I really appreciate you coming and sharing some knowledge about AI product management.
Hottest tip of the decade right now.
Before we wrap up, where can listeners find your work and learn a little bit more about you guys are building at Superhuman?
You can look at our Superhuman work at our Superhuman blog, superhuman .com, click on blog.
And then for me, I post on LinkedIn so you can go to my LinkedIn, Laura Lynn Michio.
Cool. Thank you so much for being here.
Thanks. Thanks for listening in.
For more great insights, how -to guides, and tool reviews, subscribe to our newsletter at theproductmanager .com slash subscribe.
You can hear more conversations like this by subscribing to the product manager, wherever you get your podcasts.
