not to get all check your privilege on you but if your organization has an in -house research team or works with a research firm or even has just one UXR on staff you gotta count yourself lucky according to the 2024 state of research report by user interviews for every one dedicated researcher there
are five PWDRs that stands for people who do research so by my math that means that there's a one in six chance that one of those PWDRs is you so if you do identify as a PWDR you're likely in a situation where you're doing the absolute best job you can doing UXR off the side of your desk while painfully
aware that you don't know what you don't know about doing it better.
And since one in six of us are in this exact position, we held a phenomenal panel event with three renowned user research experts who really get it and want to help.
In this recording, you'll learn what good, decent, and great user research looks like, the traits that distinguish good, decent, and great UX design, and useful strategies to connect UX insights to your product's unique selling proposition.
Let's jump in. Today, we have three really fantastic guests I'm very excited to introduce to you.
We've got Laura Klein, who's the author of Build Better Products.
Laura is a seasoned UX expert.
She's known for her ability to bridge the gap between research and actionable design.
And I'm going to start everybody off with a little bit of a jeopardy question.
So Laura, I'll follow the first one to you.
So you've got some pretty helpful insights for anyone deciding whether they should be living at the top of a mountain or at the bottom, what would you say would be the best place to live?
Oh, I mean, this is not a tough one for me because I live at the top of a very short mountain, and I really love it.
But I think the most important thing is whether you live at the top or the bottom of a mountain, you should always have a funicular.
A funicular is the correct way to get to or from your home.
That's the most important thing.
Top or the bottom. And yeah, I live in an apartment building, and I think I should also have a funicular that's much more zip line or fireman's pole I think I would be I would much prefer that to the staircase to walk up you'd be the most popular person in the in the apartment I think so too we also have Steve
Portigal joining us today he's the author of interviewing users how to uncover compelling insights which is a must read for those who if you're not familiar with the work you should check it out so Steve is a master of user research he's renowned for his work in uncovering insights that drive meaningful
design decisions as the book title would suggest so steve your jeopardy question for today you just took a week off to camp and airbnb around oregon which place should we check out next time we're in the area i've got to and they're both connected because they're just i don't know whatever the barnacle
bistro in gold beach oregon has got just the best sign i encourage everyone to google it it's like a very strange character playing banjo and you can get food there but they also have just the craziest sign And along those lines in Medford, Oregon is Blackbird, which is this hardware store with a parking
lot with a giant kind of sculptural Blackbird that is a good roadside attraction kind of place.
So we went to Blackbird after having been there maybe 20 years ago.
So we went to see how the bird was doing and the bird is doing well.
We also have Thomas Stokes joining us.
He's a principal of UX research and digital strategy at Drillbit Labs.
Thomas is a leader in product strategy and specializes in creating compelling user experiences that are tightly aligned with business objectives, which of course is highly relevant to what we're talking about today.
Thomas, you just took a week off to hike part of the Appalachian Trail.
Did you grow out the grizzled Appalachian Trail beard like you see in the before and after pics of others who have hiked it?
I gave it my best shot, you know, but I really grow a better mustache than a beard.
It's pretty full here, not so full here.
So I shaved really promptly on my return, but it's going to be back.
and thanks for having me here Hannah.
Yeah we're looking forward to it.
Just a few other words about the product manager membership before we get started with the discussion.
If you are not yet a member of the community first of all welcome thank you for attending.
This is one of our monthly sessions that we conduct that we invite members and non -members to attend but if you'd like to learn a little bit more about membership and some of the exclusive events that we host please learn more at theproductmanager .com slash membership.
We would love to have you on board have a lot of fun And with that, let's get into the discussion.
So this discussion is going to be taking part into three parts.
We'll have a discussion on UX research, so what separates the good, decent, and great UX research from the bad.
Also, we'll be looking at what separates good, decent, and great UX or user experience from the bad.
And we'll also be looking at how we can connect the dots between those kind of two significant areas to create really meaningful experiences for our users.
So to kick us off on section one, so what separates the good, decent, and great you are from the bad, I've got Steve.
So Steve, what would you say to that?
Right, research is a big set of activities.
So if you say sort of good research or bad research, I think we often start with kind of data collection or interviewing or testing, whatever your kind of method is.
So good versus bad, poor quality interviews, interviewers that don't have a lot of experience or training and asking good question that don't ask follow -ups is kind of the first part.
But then even stepping back from that, what to research is an area to think about what is good and bad.
So I think sometimes there's a naive application of research to the challenges that businesses face, like where all you do is test.
You make decisions about what to make, and then you test.
Or research just means, hey, show people the prototypes that you're thinking of shipping and seeing if they give you a thumbs up, thumbs down.
And you hear those phrases like, oh, we don't want our customers to tell us what to do because of something that Steve Jobs maybe said or didn't say or whatever.
So those sort of naive understandings of how you would use research to make what decisions I think limit the quality of research.
I think sometimes we do research and it gets treated like stenography.
Again, that's that customer telling us what to do.
the idea that, you know, sort of requirements gathering, like you just ask people and then you kind of tabulate what the requests are versus thinking of research as something that feeds into this very active, very creative synthesis process.
You know, you make things called insights.
Those are not quotes from people directly.
And I think if you don't know that is possible, you don't know the research can do that because you've never been exposed to that, then your application of research is sort of limited.
Yeah. Another piece, I guess, from that is that research is a way of driving change in the culture.
And so you hear these things like, oh, well, is bad research better than no research?
And this is a wonderfully messy kind of hot topic.
Bad research sends you down the path of making bad business decisions.
So that's not good.
But getting companies out there, people in companies, teams, stakeholders, whoever, kind of out there seeing real people and learning from them, even if it's not good by whatever measure of good you have.
That's good for the organizational culture that, hey, we don't know everything.
Our customers are going to tell us things that we don't know.
And then, I don't know, there's just this piece around what makes research good or what makes research not good that I actually disagree with.
And probably the other folks can add way more nuance than I can.
But there's this kind of self -owned that I see researchers doing, which says that my research isn't good if someone else doesn't adopt it and integrate it.
I think, yes, that's why we're doing research.
But I think when you put that in front of you, like I'm not successful in doing research if somebody else who I can't control ultimately doesn't make some decision or doesn't build something, doesn't implement something.
Research has a lot of other sort of softer outcomes than just another person taking an action.
So I don't know, I like to kind of keep it open to, I think the question of how do we assess if research is good is not so blunt force as just, did the feature we recommended ship?
I think there's, we have to look for other kinds of signals before we can even say like, was this piece of research good?
Anyway, rant over for now, but that's my take on that.
Does anyone have anything they want to add to that?
I want to address that because I think that I have made the comment before that decent research gets used.
And so I think I actually sat and this is I'm sorry, Steve, but I think that I actually agree with you.
I think that there is a difference between bad research.
Maybe it's the whole like good, bad, decent thing.
Like you can do great research and it doesn't get used.
And that is I totally agree.
That's not your fault.
We should never I mean, we should never blame the human when a system fails.
But I do think it's a systemic failure if you're paying people to do all of this great research and learn all about your users and bring all this great insight and you're not using it.
So maybe my framing is sort of like, that's not bad research.
That's bad company, right?
That's bad product decision -making to have all of this insight and just be like, whatever.
Like my gut says we should do this other thing.
So I never, ever want to blame the researcher in that case.
And I think the researchers, I have found that researchers tend to blame themselves a lot for stuff like that.
And I find it so sad.
And I always say the question that I always get asked anytime I talk about research is, how do I get people to actually listen to me and do the things that I recommend?
And obviously, there are a lot of great answers to that.
There's storytelling, there's how to share the information, and there's stakeholder management, and there's all this stuff.
And also, sometimes you are dealing with people who aren't going to listen to you.
And it's very important that you not internalize that and make that your fault.
You can do everything right and have that not turn out well because of systemic reasons.
So I just wanted to say, I do think it's important that research get used.
And again, I also agree, it may not get used in the, you built this feature, it may get used in the, now we all kind of generally know more about the people and the context and the flow and what they're doing and why they're doing it.
And maybe it helps us make a better decision next time.
So sometimes it's a little slow.
That's a really good thing to point out as well, is that sometimes that the research, even if it doesn't get used in the moment, it's still valuable.
There's still a lot of value to conducting it and conducting it properly, but it doesn't always go well.
Thomas, I'm curious to hear your thoughts on some of the things that can go wrong throughout the research process.
Yeah. And I want to add one thing to what Laura just said real quick before I talk about that.
I think there's a useful frame of mind to think about this.
And I don't I'm not often into sports analogies, but this one is a sports analogy.
There's this coach.
I think he coached for like the 49ers or some football team, but he had this thing that he said to his players that said the score takes care of itself.
And the reason why he did that was so that they stopped paying attention to like the end score.
So in this case, I guess the score would be, OK, they ship something because of my research.
and he specifically wanted people to start focusing on the things that they could control kind of like their process their input maybe the way that they execute a play or whatever again kind of talking outside of my expertise here on the football aspect but with the research aspect i suppose what that could mean or what that could look
like is okay focus on my process what can i do as maybe the person who's helping to actually do and then deliver research findings and that's all the stuff that you were talking about, right, Laura?
It's, okay, what's in my control?
Clearly articulating my findings, doing it in a way that's convincing, putting it in a format that we know people will access and will listen to.
So that has the maximum potential of actually converting and making that effect.
So I think maybe that's the nuance of the conversation there is there are things within our control.
We can't always pay attention to the end result, but in theory without organizational misgivings kind of set aside there's things that we could do to make it so that they will act on a research funding right so i think maybe that's the element and that's i think hannah you're asking you know i have got
this way of thinking about ways research can go wrong where we talk about what we do before during and after conducting a study all that has to do with after once we have findings how do we carry them forward but we can also think about what we do before a study.
That's talking about planning.
Are we doing research on the right things?
Are we selecting methods that are appropriate?
And do we go in with unbiased objectives so that, you know, we don't have a finger on the scale, so to speak, and unduly influence results so that the findings never really mean anything because we went in with an assumption we were trying to prove.
And we can also talk about, I think, Steve, you mentioned this in your initial response, right?
Once we start to do the study, are we doing that well?
Are we experienced enough?
And do we have enough knowledge about the method so that, like you said, we're doing good interviewing techniques, we ask follow -up questions, we ask questions in a way that's not leading, that sort of stuff.
So we can look at before, during, and after conducting research to kind of frame up where things might go wrong.
Can I just make one mention on the methods thing?
Because you all both brought up methods, and I think that is so unbelievably important.
I'm a huge, huge, huge fan of mixed methods and specifically bringing in quantitative data and qualitative data together, but also just knowing all of the amazing opportunities that we have to do different kinds of experimentation and testing and research and ethnography.
And these are all really useful for very different things.
And if you are trying to A -B test your way through when you should be doing ethnography, or vice versa, you're probably not getting the results that you want.
And that's the time that you really need to kind of bring in an expert who maybe does know the sort of, oh, you should actually be doing, you know, oh, maybe this should be a diary study, not yet another prototype test.
I want to recommend Christian Rohr's article.
I think it's called When to Use Which User experience research method.
And I think it's a Nielsen Norman article that's been revised like steadily over the last 10 years or something.
But there's a great matrix in there that shows different methods and sort of what they produce.
And then the whole article, I think in one revisionally sort of talks about what question do you have or where are you at in your product maturity and what types of methods are used to answer what questions.
Because to your point, Laura, like that's expertise.
And not everyone's going to read Christian's article and be like, yeah, I know how to use that.
But at least says like, this is not a hit or miss.
There is a process here.
And Christian, my experience has done the best job at kind of documenting that in a way where like, well, we can actually choose.
Awesome. While we're shouting out additional resources on some of Laura's thoughts, Laura has actually joined us on the product manager podcast on basically that exact topic.
I believe the episode is called How to Have Fun with User Research.
If you're interested in kind of hearing some of Laura's more elaborated thoughts on that topic, please check it out.
It's a great episode.
And that kind of brings us to section two, which is what separates good, decent and great user research from the not so good.
So Laura, I actually have you set to take this one on head on if you like.
This one's for a user experience design, right?
Yes. So it's so funny because when you first set the name of the panel, the, you know, how to delight users that I'm like, you know what I find delightful?
I find it delightful when things work, which is often a thing that people, it's so funny because we always talk about, we need to delight users.
And people kind of think this needs to be like, oh, it needs to be fun and enjoyable.
And I'm like, yeah, it needs to do the thing that I want it to do and get the hell out of my way.
So I think there's actually this, which kind of brings me to the point that good user experience takes into account what the user is trying to get out of the product, if it's a productivity app that I have to use every single day for my job.
I mean, if it's a game, I play a lot of video games, right?
If it's a game, it should have a very different kind of user experience than a CRM.
I get a little bit annoyed when people try to force all of the same things into the user experience.
It should take into account, Again, my context, the flow of what I am doing, it should not interrupt me.
It should not force me to work a different way if possible, unless that way is a way that you can teach me in such a way that I'm like, oh no, this is really much better.
Good design is about behavior change, right?
You are changing my behavior, but you should be changing it in such a way that I get what I want out of the product.
And I should get to be or do whatever I want to do within reason.
But mostly, I don't know that you need to delight me.
You just need to make it work.
You need to figure out what I'm trying to do, how I'm trying to do it, and help me to do it the correct way.
So I think bad user experience often tries to be, I mean, sometimes it tries to be too clever.
Sometimes it tries to be too pretty.
Sometimes it tries to be, you know, sometimes it tries to be too minimal.
It tries to be too, the misuse of the word lean.
You know, sometimes it does a 10th of what I want it to do.
And then I'm like, well, that doesn't really help me.
So the funny thing is, a lot of us are willing to put up with suboptimal user experiences for products that actually help us do things that we want to do.
Doesn't mean we should have to.
It very much depends on how important the thing is that we're trying to do with the product.
It depends answer. There you go.
I think this is a very common theme with many, many things.
But in this case, yeah, the context is really everything.
Thomas, I know that you had had some frameworks in mind that were kind of applicable despite the fact that there was obviously a lot of context that comes into play with each individual situation.
Yeah. Yeah. I think one that we can plug into this conversation really well is Aaron Wolter's hierarchy of user needs.
And if anyone hasn't come across that before, it's very similar.
If you've come across like Maslow's hierarchy of human needs, it's similar in idea, but it's recontextualized to the context of user experience.
So the idea is that there are some very foundational needs towards the bottom of the hierarchy and higher level needs as you advance.
And there's kind of four levels going up.
There's that the design is functional.
It does what it's intended to do.
So next step up would be reliable, not only does it, but does it consistently at all times.
And then, of course, from there, not only is it functional and reliable, third level would be that it's usable.
It is very usable. It is user friendly.
And then the highest level, Walter argues that it's pleasurable.
So in theory, if we want to take that on its face, we could say good user experiences achieve those higher level goals, less good ones don't achieve the higher -level ones or maybe don't even achieve the lower -level ones.
And I think, Laura, one of the things that you said that really stands out to me is that if someone tries to be or a product tries to be too clever, like too beautiful without actually addressing foundation of usability that's below that or even reliability or functionality, then what's the point of that kind
of pleasurable element if it's not actually meeting all the other needs that support it?
I think we can do all of them if we're conscious of it, but it's just a matter of recognizing that there's kind of foundations, right?
We've got to build from the feed up an experience that achieves all the four of those elements.
I agree that we should strive to achieve all four.
I actually struggle with it being so linear because I actually think this is a weird, this is a weird thing that most people don't talk about.
I think different types of products, those may be in a different order.
Like if you are doing something that appeals to, you know, like shopping or clothes or makeup or beauty or whatever, actually, that making it beautiful may be more important to your users than having it be super reliable.
I play, like I said, lots of video games, and I have a few that are what I would like to call not unbuggy.
And you know, they're not super reliable, yet they're still fun enough that I play them and I enjoy them.
And would I prefer that they work all the time?
Yeah, absolutely. 100%.
But they're fun enough that I'm willing to kind of forgive that.
That stack I think applies great to something like, again, a productivity app or email or something that you have to use.
But for things that are more, you have to figure out where you live in that product and how important your product is to the person and what they are trying to get from it.
But I agree that everything should hit all four of those ideally.
Yeah, this is a really interesting insight, the idea that the hierarchy of needs itself can be contextual.
That's an interesting takeaway.
Just to ensure that we're kind of hitting all the marks before we start to get into Q &A, I'd like to move into section three, which is how do we connect the dots to elevate and experience and offer this unique proposition for our product that kind of combines all the elements of our research to create
these great experiences.
And since we haven't heard from you in a few moments here, Steve, I'll get you to start off with this one yeah and i think it builds nicely from what we're talking about right the the unique aspect i think is key here and laura and thomas are kind of talking about what should it hit but also what does
that mean for you and your product and you know i think because we're speaking broadly to a broad audience about broad topics we're using words like pleasurable and delight and efficient and so on and laura's kind of hinting at like it changes as you kind of move around from category but beautiful for you
know when we say beautiful we think of sort of something visual but there's beautiful when two pieces snick together properly and you just kind of go like uh like it just feels good i think there's sort of beauty and delight that you know we have to be kind of diverse in how we think about what that looks
like i feel like a hundred people have written in the comments it depends it depends in like the very enthusiastic cheerleading way like I feel like we're at a rally here where we could just say it and everyone would feel in the phrase, it depends.
And Laura, you're getting at this as well, right?
Who are you and what do you kind of stand for?
I was thinking as you were talking, Thomas, about Slack, like Slack has a way, and this is more about content than experience.
But if you update the Slack app, they have a very specific tone in their contents that's they're consistent and they don't sort of write like anybody else.
That's kind of irreverent.
But it says a little bit about who they are, who they think you are, what they think their relationship is with you.
I don't personally think Slack translates that into their user experience necessarily.
But there is something about being a brand, having a personality, knowing who you are in a consistent way and knowing who your users are.
It's just tied back to research.
Like you have to do a lot of work on yourself, just to sort of therapize the language here, and do the research to sort of understand everybody else and then do the design work to kind of connect all those in a way that you build in a way that's consistent.
And we're not talking here about like, I mean, you talk about unique selling proposition.
We're not even really getting to like doing the thing that you're there to do that people care about accomplishing, but just what outfit do you wear while doing that, I guess, is really where I'm talking.
So knowing yourself, knowing your customers, kind of integrating those consistently.
That's the connecting the dot piece, I guess that's kind of part of your question.
Yeah, I'm kind of running out of steam here.
Somebody else jump in.
Yeah. Thomas, did you want to add a new insights?
Yeah, I'll throw one in.
Steve, you said a really important line in there.
You said, do the research.
And one element of that is that that research should happen across the product development lifecycle at all different stages, right?
It's not enough. If we really want to have a USB, it's not enough to just like usability, test and flows before things go live.
that misses the mark.
If we're really going to draw a circle that says this is what users want, or this is what people need, and this is what we're good at, and we're going to find out what's in the middle of that, we have to be able to draw that circle if this is what people want and need.
And to do that, we need very early discovery foundational research.
Also, we understand that what people want, or if we want to keep saying it, what delights people is like a moving target that changes over time.
It's not always going to be the same.
It's not static. So we have to actually have like strategies to measure things after they go live.
I'm a big believer that a good UX measurement plan helps you keep your finger on the pulse of how people are receiving what you're putting out there.
And so having actual post -live measurement will help you understand how things are actually shifting.
So at all stages, whether or not you're not even sure what to build, if you're building the thing or if it's even out there, You have to be doing research on all of it to actually have that USP that people are after.
I think it also had some comments about like, that's kind of one part of it.
But then there's the element of what's feasible.
There's other matters to take into account.
Yeah. And I think this comes full circle to something that we said or that Steve said earlier.
Steve, you often say like research isn't stenography.
It doesn't tell you or you don't take it on its face.
It doesn't tell you what to do directly.
Further to that, right, we're talking about using research to essentially prove out user desirability and usability, right?
But I mean, I'm talking to a bunch of product managers.
I'm sure everyone's heard this before, but it's worth just giving this caveat that there's going to be two other elements in addition to that user desirability that we've got to balance.
There's obviously feasibility.
We've got to be able to build the thing.
it'd be great if user research reveals that we could build something that you click one button and your house is clean and your laundry is done and you've got groceries.
But maybe that's not one button click, right?
So it's got to be feasible.
And it's also got to be kind of viable for the business.
You work within a organizational system that is going to select four things that advance its mission.
So the business viability, the technical feasibility, and the user and desirability kind of have to all come together.
And that involves a bit of decision -making, right?
Laura, did you have anything you wanted to add to that?
Yeah, I have a kind of a weird side take on this, which is I think it's much easier to deliver great user experiences that fit with your business needs if everybody's incentives are aligned.
And that has to do with your business model.
So if you have a fundamentally, what I would consider to be a more ethical business model that says we are going to deliver a great product that is so good that people are excited to give us money for it.
Then the better you make that product, the more people are going to want to give you money for it.
And everybody's incentives are aligned and that's fantastic.
And obviously that's simplistic.
And that's not also the complete definition of ethical.
You can still hurt a lot of people doing that.
But that's sort of the baseline of if you have a business model where your user's incentives are aligned with your business incentives, you can make a much better argument for we want to make this better for users because that's going to make it better for the business.
When you maybe don't have that or you have, you know, the customer and the user, which happens a lot in B2B, where the person who's buying isn't necessarily the person using it, then you have to be a little bit more creative about about connecting those dots between if we make it better for the user,
it actually makes it better for, say, the organization, which is a good reason for you to buy it.
So you have to make that sort of leap.
And then you also have the products where those things are just very disconnected.
And there isn't really a great argument for making this a better product makes us more money because it might not.
I don't know. I don't think those are great places to work personally.
And I don't I think those are great.
They're definitely a great place to be a researcher or designer.
So keep that in mind when you're looking at your next job.
We did get a question that seemed to be building on what Thomas mentioned, which was where would necessity come into play?
I'm just trying to see if that user who posed a question maybe can elaborate a little bit more on those comments.
But does that kind of provoke any kind of response from any of the panelists?
Necessity for whom?
Or for what? Like if you're, is it like the thing, like if you're forced to use the product, Then, again, it's, I mean, it would still be great if it were easy to use.
Yeah, I suppose there could be that user -buyer disconnect in a lot of B2B spaces, right?
Where like they build stuff for the person who's actually in the sales cycle, the one who decides on purchasing the experience for the team, all actual end users be damned.
So I suppose that would be a negative implication of necessity if you're required to use it in a B2B space.
But even then, I think we could all argue that it's to the benefit of the folks who are building those experiences that we would in theory win out if we support the end user and we can show, you know, there's some meaningful case studies or whatever that by supporting the end user, we actually achieve
better outcomes, which then influences the buyer's decision, right?
So maybe it's a roundabout type thing, Laura, I'm not sure.
All right. So let's move on to the Q &A.
So our first question from a member is, how do I get people to answer surveys?
I send Slack DMs, email surveys, without a budget to properly pay and incentivize customers, it's tough to get their time.
People are stretched really thin these days, so it's understandable, but it makes my work very tough.
I think Thomas had some kind of preemptive thoughts about kind of how we can manage this a little bit.
Yeah, I think there was a key line in there about not having budget to incentivize people to participate.
So I've got two main points.
One, specifically about budget.
If we're not actually incentivizing people for the time, we don't have the appeal of monetary incentive, we have got to appeal to something else.
And one thing I've found is that in those situations, you might be lucky enough that you're in a situation where your customers actually really care about impacting the direction of the product.
And so you might appeal to that.
You might actually really emphasize in whatever recruitment channels you're using that, hey, this is really something we are going to use to drive forward product improvements.
You got to make good on that promise, but use that to appeal potentially to potential participants.
The other thing is, you know, if we're not actually incentivizing them for their time through a sort of reward with money, be relentlessly, I guess, scrupulous and just really look over the survey that you're building.
Just prioritize really what you need.
It might not be the perfect survey that you'd have if you had unlimited budget and time, but narrow things down so you at least get a good response rate when the people who click in thinking, yeah, I'm going to help the product direction.
If they see a million questions, they're probably still not going to answer.
So go with both of those.
Try to peel something other than just monetary incentive and really relentlessly prioritize your survey down to what's essential for the decisions that you might make off of it.
Great answer. Sure.
So our next question from our members is, my challenge is in getting the truth from users about their reasoning and intent for using the software the way that they do.
We hear tons of stories about how they think things should, quote unquote, should work or look, but it's hard to get them to open up about what they're actually doing using the product.
Steve, did you want to take this one on?
Yeah, I agree that interviews should cover what people are actually doing.
You know, I think the fun of this format is trying to infer some more context from the question.
Like, I really wonder, are these interviews?
So I'm thinking about like, how are they being set up?
Who's being asked to participate?
It's kind of what you were saying.
But the dynamic that we have with our research participants, I think an interview is the same thing is true with surveys.
Like what's the expectation?
Who's being asked to participate when and how?
Like are these people calling in with tech support questions?
And then they're being kind of escalated to, to interview.
That's a different context versus, Hey, we want to talk to you and learn, how is the interviewer introducing the subjects of interest?
We want to talk to you about X and Y.
And how is the interviewer asking those questions?
Because I think I usually ask questions about what are people doing?
What are you doing?
How does that work?
Before I get to anything about what would you want to see different in the future?
It's a great question for me, because it makes me wonder, well, why isn't that happening to begin with?
So again, I think just expectation setting, what questions are you asking?
What follow -ups are you asking?
I think, and yes, every interview, no matter how well you set it up, somebody is going to come to it with a different sense of what the purpose of that is.
And sometimes you just got to let them share what they want to share rather than kind of squelch them into your model of the conversation.
Let them share what they want to share and then say, this is great.
We've got some other things we'd like to know?
Can we talk about what your workflow looks like today?
I want to go right back to the beginning.
How do you configure this?
I think you can keep asking questions to build to that kind of outcome.
How are they working?
It's not saying, how are you working?
It's many, many, many, many smaller questions to get at that information.
So I think it goes back to what's good and so on.
like it's technique, it's expectation management, it's all kind of that stuff.
So just go based on what that question makes me think of.
I don't know. That's my kind of my riffing on where to think about making improvements.
So the next member question is about the administrative pains of research.
So ongoing customer meetings to discuss their needs and desires and hopes and dreams is a painful administrative task full of cancellations and rescheduling.
Is there a way to ensure user turnout for these meetings?
Does anyone want to take a crack at that one?
I'm not really a user research, like a research ops expert.
I don't know if Steve or Thomas is either.
This feels very much like, I mean, so can you make people show up to things?
No. I mean, there's all sorts of things that you can do to make it better.
You can send reminders and make sure that you're offering an incentive and all that kind of stuff.
But like people are people and they're going to people.
So you're going to get ghosted.
It's going to happen.
But I think that having a good research ops organization, if you can, or a person who is responsible and who can sort of help guide that and it at least takes away some of the hassle if it is somebody's job to be able to say, oh, we got you an extra person and you can't make people show up to things
they don't want to show up to, especially these days.
I'll give one too. There's a useful stat I think we can start off with.
You can expect 10 to 20 % no -shows and cancellations.
That's generally true.
So you can look at your own practices.
If you're trending below that 10%, great work.
If you're over that 20%, maybe there's a lot that you can do to bring it down.
Like Laura said, if you establish contact, if you do things, you know, ops -wise, like actually maybe automate reminders and that sort of stuff, make sure that the incentives are around the right level to encourage people to show up, that's good.
But I saw this one trick.
I'm wondering how well it works for most organizations.
I'm not sure if anyone else has seen this one, but it's exploiting consistency bias.
I don't know if anyone else has seen this, but essentially like a screener or a recruitment thing, you get people to agree or disagree with a scale question that's like, I'm the type of person who typically keeps my appointments or I show up on time, something like that.
I've seen people try to say that putting those in and if people agree, they're more likely to show up to your actual sessions.
It sounds fun and interesting.
i'm yet to test it out myself but who knows it's worth a shot that feels like one of those things that people that you know end up getting written up in a very popular airport book and then later on it turns out that it was you know exactly eight people all of whom were college students at duke and it
is not at all applicable to anybody else in the world or whatever it is but uh but uh it's i mean it certainly sounds fun it seems like an easy thing to try yeah i would love to run that experiment yeah so we run that experiment experiment get back to us i don't know yeah well you know that's i think
it's very interesting to kind of play upon people's you know uh self -awareness and their their perception of who they are as a person as a means to incentivize them to show up for an appointment uh maybe i should do that on myself okay we'll move right along here this is an interesting question it's
a little bit more about bias management.
How do you avoid the biased voice of the angry customer who's reaching out to vent and or turn the call into a support case?
I guess this is about doing research based on stuff that folks have volunteered.
Does anyone want to take this one on?
I might start here.
I think it's just like the other question about what we're not hearing from people.
And I had a lot of questions about context here, right?
You're in control of your sample.
So, you know, this to me is this, it's a sampling question and it's a method question.
So Thomas kind of made a reference to screening.
So how are you screening people in or out to participate in research?
Asking a question about your disposition towards the brand or the quality of experiences you've had is you could filter in or out people to get kind of a balanced sample, right?
We do research on individuals, but we do research on a sample.
And so somebody has a perspective.
I don't know, is an angry customer a biased voice or is an angry customer an angry customer who has, you know, a lived experience in today's parlance.
And maybe it's the second piece of the question is, so should we include angry people that don't like our product?
Yeah, but maybe not exclusively unless that suits the objectives of what we want to research.
People wanting to turn the call into a support case, I think that goes back to the thing I was saying before, you know, expectation management.
Who's asking for the call?
What's the purpose of the call?
Let's reiterate that at the beginning of the call.
And I think this is really important.
Let's live up to that, that behavior.
So I see researchers telling participants, okay, first of all, there's no wrong answers.
And then go on to be very excited or not about the responses, which basically tells people, yes, there are right and wrong answers.
So how do you sort of set and live up to that expectation in your interactions with the participants?
And if someone has something they have to tell you, a wise woman once said, people are people and they're going to people.
You can't stop someone and nor should you from saying the thing that they're, you know, they're the they're the rage character from inside out.
Like, if that's where they're at, like, meet them there.
That doesn't mean that's the tone or the content of your entire interaction with them.
Like, that's just a starting point.
So lots and lots of interview participants come with an expectation, and it's a bias about the interview.
And then you're like, yes, and yes, and tell me about this and tell me about this and tell me about that.
So I think it's very manageable, but it just takes a little bit of know how.
Yeah, shout out to the improv tactic there, the yes, and very good points.
We talked a lot about some of the methodologies around, you know, how we're thinking about things and conducting research.
But I also want to make sure that we're mentioning some of the more specific tools and that kind of thing that we use in order to be effective in our roles.
So what are some of the go -to research tools that you folks are using for capturing insights and gathering user feedback, and especially at scale?
I feel like this is a very different answer for people who are on teams and large teams versus small teams and teams that have research ops versus teams that don't and consultants versus non -consultants, you know, in -house versus not in -house.
Anyway, it's not a question that I can answer specifically.
There are a ton of tools right now that are available.
And I would say that as with anything else, you need to look at what does your team need the most?
Do you need to, you know, sort of democratize the responses?
Do you need to make things easily searchable?
Do you need to do management of participants, like what is your specific problem?
And there's so many more tools out there than there were back when I was, you know, doing user research every every week.
But they each kind of fill a specific niche and just make sure that that's the niche you need filled.
Yeah. And I'll also say, I've got a rule for myself that I don't answer this question in a public venue with people coming from many different backgrounds and organizations.
But I do answer this question in private channels.
So if anyone does want actual kind of contextualized advice around tools, I will help you out with that.
You can reach out. But yeah, like you said, Laura, there's just way too many things that influence whether someone should choose this tool or that tool.
There's even just like elements of the way that that tool does business might not work with your organization and the way that like your procurement works.
So there's all these thorny details about actually getting in and starting to use a software tool in the BDB space that just makes it a slog.
No single person has used all of the ones available.
And there might have been one released yesterday that solves everybody's problems.
And that's wonderful.
And I wouldn't want to miss that one by saying, use this thing that I used five years ago.
I think we have time for one more.
So we'll just go ahead and jump into it here.
It's actually, it's coming from an anonymous user.
How can we make UX easy for a novice user, but at the same time, get out of the way of an expert user?
Oh, this is an interesting one.
That is also its own whole thing.
Here's the thing. It's not like advanced users want things to be harder.
It's just that sometimes they are doing weird things with your product.
I think a lot of times people think of easy as we are going to strip away all of your choices or all of your options, or we are going to shove everything into a settings thing.
And I think the most important thing is just to understand what is it that people need to actually sort of get started and be successful with your product in that onboarding experience.
By the way, it's probably not five screenshots with little arrows saying, hey, do this.
Hey, we added five new features, check it out.
It's probably not a product tour like that.
It's probably some sort of process that helps them get to the first time using your product while training them the steps that they would need to go through to be successful.
And then having there be the next time the option to maybe sort of move through that a little more quickly.
But it's very much understanding what is getting in the way of the novice user.
what you know you don't drop them into a giant complicated interface and be like good luck and then understanding what makes a power user what is a power user trying to do that is fundamentally different from what a novice user is trying to do and how do we make that you know again easy the first time
even for power users sometimes they might be doing something with your product that is their first time doing that thing you still need to make that easy as well so So don't think about it so much in terms of like noobs versus, you know, old folks or whatever.
It's how do I get people started doing the thing that they want to do in the way that helps them to learn what they should do the next time they want to do it.
And you can have settings and you can have things for like actual power users.
I'm an ex -engineer and sometimes we want hotkeys.
Don't take them away from us.
Panelists, this was exactly as fun as I knew it would be.
So thank you so much for being here, for giving your insights and your great personalities.
We really appreciate your expertise and your time.
