Things aren't always as they seem.
From bait and switch subscription pricing to the people who swear candy corn tastes good.
We have no shortage of reasons to be skeptical of everything, especially right now in product.
The speed of change in this industry is moving so fast that trends can start looking like best practices and the real best practices can start looking outdated.
But when the goal is to build a product that will scale over time, it's on us to check ourselves from getting sucked into hype train hypnosis.
My guest today is Matt Graney, CPO of Sligo.
Matt has spent over 20 years in B2B product and almost nine of them scaling the product team at Sligo from four to over 45 people.
And while AI has proven to be an extraordinary opportunity for the company, he's noticed a gap between expectations and reality when it comes to some of the shiny new tactics blowing up our LinkedIn feeds.
From prototypes masquerading as production-ready code to too-good-to-be-true tools, you'll hear his take on where to tread with caution and the time-tested wisdom he's counting on to make it through this AI transformation alive.
Let's jump in.
Oh, by the way, we hold conversations like this every week.
So if this sounds interesting to you, why not subscribe?
Okay, now let's jump in.
Welcome back to the Product Manager Podcast.
I am here today with Matt Graney.
He's a CPO at Soligo.
Matt, thank you so much for making time in your schedule to talk to us.
Thank you, Hannah.
Great to talk to you today.
So can you tell us a little bit about your background and your journey to becoming the CPO at Soligo?
Yeah, so I've been in B2B product for a long time now, about the last 20 years.
Soligo is an integration platform.
And so I've been with Soligo for eight and a half years, joining just after Series A funding and another five and a half years before that, also in the integration space.
It wasn't really by design.
It's just how things worked out.
Before that, I was involved in products around the software development lifecycle, including UML modeling tools for those of you who might remember what those were.
Originally I was a software engineer back in Australia, working in telecom and defense, and first came to the US to work as a sales engineer, actually for a product that I'd become a power user in while I was at Motorola.
So kind of a diverse background, and then took a tour of duty in product marketing and then finally made it into product.
A tour of duty.
I've never heard it described that way before.
That's funny.
So today we're going to be having a little fun with a very quasi-Halloween-themed episode.
We're going to be focusing on the theme of bad product plays in disguise.
And we're going to start by digging into vibe coding.
So lots of opinions being tossed around in vibe coding in the product community.
And I'll be the first to admit, no-code platforms are amazing tools.
I use them all the time.
Not what it seems.
So what's been your experience, Matt, with vibe coding within product teams?
Tell me the good, the bad and the ugly.
Yes.
And we would have got away with it if it wasn't for you pesky kids, right?
So I think it's democratization, which is incredible.
But we have to also think about the delusion.
You know, with that power to make something look good, it doesn't mean it's necessarily built right.
And I think there's a lot that has to happen under the covers.
We shouldn't confuse perhaps quick prototypes with production-ready code.
And so, okay, citizen creators, but you also need citizen architects.
And in the context of the business we're in, for example B2B, we're dealing with infrastructure software.
I mean, we're talking about billions of transactions a month.
There are only certain parts of the application, perhaps on the very front end, where we might be comfortable vibe coding anything.
And so I think it's got to be about the right tool for the job, just as it's always been.
And while also making sure there's a culture of experimentation, making sure that we're encouraging the team to take risks, to try new tools and stay current with the latest developments at what is truly a groundbreaking time for the whole industry.
So just to go a little bit deeper on that, when we think about the various levels of understanding of the technology all along the organization, it can be very tempting for folks who are less experienced either with the technology or new founders themselves to kind of see what looks like functioning code and just kind of run with it and puts a lot of pressure on engineering teams to kind of match that speed or be able to develop at that level that quickly and make it look so shiny and new.
So how do we help stakeholders understand the real constraints and considerations between the prototype that we're seeing from Vibe Coding Tools and production-ready software?
Yeah, that's such a good point because speed to demo is not the same as speed to production.
And especially, we're talking about 5,000 customers, again, with B2B workloads.
So we sometimes talk about what we do in infrastructure is the plumbing.
So, okay, AI might paint the house.
It's not necessarily going to plumb the house, right?
You want to be sure about some of these things.
And there are some challenges because, whether it's pressure on engineering teams or even pressure from the exec team, Recently I've seen internally fairly senior members of non-technical staff vibe coding proofs of concept to show new capabilities that are much needed by their customers.
It's certainly fired the imagination, but it's also creating this unspoken pressure that maybe this is accessible to everyone, that this is something that we can rush into production.
And if we're talking about buildings, if I go back to painting houses or plumbing houses, I mean these are not necessarily load-bearing walls, right?
These are the facade.
It might look good.
And again, that's not to say there isn't a place for it because the speed of POC is just incredible.
And we need to be embracing that at every turn, whether that's to help a product manager better explain requirements, whether helping a designer to show alternative workflows, whereas before they might have been going through designing many different screens in their favorite design tool.
The power of working code is undeniable.
And we need to be looking at embracing that at every possible turn, while recognizing that it's not the same as production-ready code.
Yeah.
And something that's been on my mind as well is that these are tools that invariably they're going to get much more sophisticated.
And we, as users, will also become much more sophisticated at using them, which means, of course, we're trending towards this cost of building approaching zero.
So for yourself as a product leader, what are the concerns that you have about that trend?
And what have you been doing to mitigate those concerns at Sligo?
Yeah, Hannah, I think that's a great mental model, a great thought experiment to run.
Like what happens if the cost of building goes towards zero?
Okay, maybe it's fast approaching the generations of these tools, as you say, the ability of users, skilled users, just as we see improvements in sort of the usual chat, GPT kind of experiences.
So we can expect a dramatic decline.
So it becomes so cheap to build.
I think that actually counterintuitively puts even more pressure on product managers to make sure we're building the right thing.
So it doesn't absolve us of all the right things we should be doing looking at product analytics, quantitative and qualitative user research, customer interviews, product advisory councils all those things, proofs of concept.
A, B testing, working closely in a triad of product managers, designers, and engineering, right?
So none of that goes away.
And I think we have to guard against that, because to go to a Halloween theme couldn't help, with a bunch of zombies running around like zombie projects, things that were so easy to build, Maybe littering the product with all these ideas that never really quite made it.
And so I think again that some of the older disciplines of product management really come back to the fore, because we have even more choice now.
I think the ability to make decisions, to drive the right kinds of outcomes, I mean all that has to remain.
I would agree.
On the topic of maybe ill-conceived ideas, I have certainly been seeing an explosion of tools flooding the market that seem to offer, let's say, enchanting benefits, while hiding what maybe not even hiding, but just offering some very serious risks.
Like I've seen some fairly egregious concepts circulating that I could just so clearly see an opportunity for bad actors to just manipulate in a way.
That's not really what we want.
So what are some of the things that you're seeing in the space that give you pause?
And what role does product judgment still play that just can't be automated away?
I think we've always been looking for maybe the magic eight ball of product management, whether that's scoring methodologies like Rice.
I've seen them abused as well, because it still leaves a lot of latitude for product managers to have their thumb on the scale to influence the scores.
And it doesn't take too many rounds of it to figure out exactly exactly how to move the needle and tip the scale in your favor.
So I think we'll see the same sorts of risks here as well.
I think there are some tools that promise maybe they're going to vacuum up all the Intel product telemetry, every conversation there ever was.
But I think we all know there's plenty of things that happen out of band.
Observations that maybe it comes from a session replay tool, but it's not written down in a form that an LLM is going to understand, right?
So I think there's no substitute for sound product judgment.
And while maybe we have more data than ever, I think at the end of the day, it's incomplete data.
And there still needs to be a vision, a strategy.
And in product management, there are bets.
And yes, we take bets, knowing that we're going to be able to measure the outcomes hopefully, if we do our jobs well.
But it still has to be an iterative process.
And I don't think there's any magic answer that AI gives us to suddenly produce an infallible roadmap.
Yeah, I tend to agree.
I was speaking to a guest who has not yet been on the show.
It was coming up quickly, but something that we discussed was this concept that AI is kind of a jagged technology in which there are some things that it's very good at you know we can exploit those advantages but then there's things that it's not so good at, but it looks like it's very good at.
And so yeah, it's a matter of the product sense, but also understanding the technology well enough to be able to kind of check yourself on.
Like, what are you really relying overly relying on the LLMs to do for you?
And I think even with basic chat interfaces.
I think we've seen plenty of examples of some AIs being quite sycophantic right.
And if you're not really awake to that, you might begin to think you always have great ideas.
And so personally, I like to spice it up a little bit and make sure that I have sort of an alternative view and ask for a hypercritical review of the ideas I have, because otherwise I'm always sounding like a genius when I talk to my AI.
Yeah, they love us, don't they?
So let's kind of move past by coding a little bit.
And I'd like to talk about some other, let's say, like attractive but ultimately unsustainable shortcuts that we're seeing product teams take right now.
What are some of the bigger offenders that you've seen around?
I think one of my favorites is OKRs.
I think we've got sort of a troubled relationship with them, maybe at a company level, not too bad.
But I think in product, it can be a bit challenging.
And I think it's what happens when you try to import something from a fang, in this case like one of the big name companies, without necessarily having the rest of the culture to go with it.
I think that's in general, you just can't graft on a limb.
Okay, now we're going to talk Frankenstein's monsters, right?
If you don't have that sort of as part of the culture, These things are never really going to knit together properly, right?
So that's one.
I think there's also we've all seen metrics theater, a bunch of vanity, metrics that don't really tell us much or don't drive better decisions.
I think that's a gotcha that's been around for a while.
Maybe we talk about feature factories or feature farms, and maybe now we have the ability to farm by the acre, right?
Because Because of the scale, again, our ability to produce is going up.
How do we make sure we're producing the right thing?
So I think any of these sorts of things are essentially shortcuts.
Again looking for magic solutions that apparently work somewhere, because someone read them on Hex or on LinkedIn somewhere.
And without that sort of rigor behind them, it's just going to erode trust, I think.
Yeah, I would agree.
On the topic of eroding trust, I think that one of the ones that comes to mind for me is how it's affected the UX research community.
I know that UX researchers, I think, have long suffered as being sort of the underappreciated aspect of the product process.
But now LLMs, I think we kind of get an even more muddled view of how to conduct that correctly and kind of the role of AI in assisting with that process.
That's one like I wish I could shout out from the rooftops that you just cannot substitute User research with LLM.
And our head of user research recently was telling me the same thing that the AI transcripts are great.
Okay, you know, verbatim, this is what was actually said.
But the insights often miss the subtleties, certainly at the moment.
And maybe that'll change.
I think, as we say, there is continued generations of this technology.
I think we can be optimistic about the future.
But for now, I think it's most important to understand the limitations and guard against them through old-fashioned craft of good user research.
So switching gears a little bit, let's talk a little bit about your experience scaling teams.
So you have been doing it for a while.
You've scaled your team at Sligo from 2PMs to 10 times that size.
You've got lots of experience in building processes.
I'm sure that there have been missteps along the way.
So can you tell us a little bit about what has been sort of your, or a few of your, best takeaways in terms of scaling teams from small organizations through their maturity?
What have you kind of learned that you think still holds true even today, in this fast moving time of AI?
Yeah, so it has been a journey, as you say, Hannah.
So I joined the company, inherited two PMs and two tech writers.
So a team of four.
And now we're about 45, close to 50, right?
So it's been a lot.
And it's a team of PMs, designers, researchers, technical docs, and product operations.
I think what I've learned is maybe the order in which to do things, right?
So at the very beginning, life was simple.
You know, by my side on the more the platform side, working directly with our CTO and three of us around a room prioritizing an entire backlog right.
It was as simple as that, but that clearly doesn't scale in the long run.
Dealing with offshore teams, both PMs and engineering being offshore, complicates things.
And we gradually added process.
At the beginning, design was just the best we could do with the tools we had.
So it was PMs doing their best.
I always think it was almost like stitching together screenshots like a ransom note.
That's kind of how I think about it.
It was so grungy and almost embarrassing.
And then really, I think our first... foray into professional designers, we didn't do that well.
I felt like we used design more like an agency model.
Hey, make this look pretty instead of really thinking about it as user experience, as opposed to just design.
Docs, we've always been fairly good and that has continued to evolve, So we've added process as we needed it.
I'm not saying we're perfect, but it's tended to work for us pretty well.
And maybe haven't got all the right ceremonies in place.
At times it feels like, but I think sort of directionally it's been correct.
And it's been a case of just enough process and responding to the needs of the business and providing room to grow for all members of the team, and so on.
But it's been a journey.
I've never run a team this large before, right?
There are a lot of firsts here and I have some battle scars and gray hair to prove it.
Oh, really?
Where?
Yeah, I'll blame my kids maybe.
Okay.
Yeah, you can blame everything on your children.
I do it all the time.
So to expand on that a little bit and kind of tie it in with what we were talking about before, I'm curious about some of the tried and true product management practices that you think maybe are actually more important now than ever, even if they seem a little old school.
Are there any concepts or frameworks or practices that you found yourself returning to more than ever or emphasizing with your teams in the age of AI?
As I said, as the cost to build maybe approaches zero, I think it actually puts more onus on us to prioritize.
So I'm thinking more in terms of the tools for prioritization.
So some of that has to be alignment around a vision and strategy, making sure that everyone on the team is clear enough to be able to make localized decisions.
No substitute for firsthand contact with users.
With customers, including.
You know.
I feel like I made my early days of my career with hostage negotiation, like unhappy customers talking them off ledgers or whatever.
I think there is no substitute for that, because it really helps inform the full picture of what the product is about and gives the PM tools they need to better understand how they ought to be prioritizing.
So I mean there's no substitute for ruthless prioritization.
At some level, you're going to run out of capacity.
I sometimes look with envy at much larger companies, but I know somewhere they have exactly the same sorts of problems.
I know all PMs do.
You never have enough capacity.
So it just comes down to prioritization and all the usual tools apply.
I think, as I said, where AI comes into play okay, maybe to help understand a whole and digest a lot of information indispensable when it comes to research.
And then now, as we talk about with vibe coding, when put in its place, you know, for rapid prototyping.
I think what some people forget about prototyping too, is the original ideas of prototypes is that they're meant to be thrown away.
They're not meant to be the basis for what goes into production.
And if you can do all those things, I think, then the tools are there to really accelerate the way we work and again, to assist with prioritization.
Yeah, well said.
Well, to close on an optimistic note, where would you say are the most legitimate high value opportunities for AI to enhance the work of product managers?
And what advice would you give to product leaders who really want to embrace AI in a way that's thoughtful and without falling into any of these traps or potential mask off moments we've talked about?
Yeah, I think really one of the big ones has got to be around research.
I think the ability to do competitive research for PM these days it's a tool I wish I had in the past.
Sure, I think.
Part of it, of course, is vendors tend to be a lot more public, like most docs for products are available now.
But you really need to be doing that.
That is a huge one.
I think, obviously, quickly putting together documentation.
A lot of people talk about writing press releases first or FAQs first.
I think this is a great opportunity.
It might have seemed laborious to do that before, but it's a great way to get started today.
And I think it's about using tools like this to help expand the imagination.
What am I not thinking about, right?
I think AI, when prompted in the right way, can be really good at identifying some blind spots.
And yeah, it might hallucinate sometimes, but even out of that sometimes there can be insights or lateral thinking.
Perhaps that hadn't come to mind.
You know, I think AI though can be a bit of a funhouse mirror, right?
If it's messed up to begin with, if you don't have your discipline in place, then it's only going to make it worse.
But when done right, it can provide that focus that product managers need.
Yeah, I tend to agree.
Well, Matt, this has been wonderful.
Thank you for sharing all of your knowledge and for you know sense, checking some of these things that we're seeing so much of in the space.
I really appreciate it.
Where can folks follow your work online?
Best to find me just on LinkedIn.
I seem to be on there more than anywhere else.
So look forward to catching up with people there.
Awesome.
Thank you so much. next on the Product Manager Podcast.
If you thought this episode went hard on expectations versus reality.
We are about to deep dive into LLM technology in an episode that will challenge everything you think you know about AI.
While the potential of the tech is limitless, the current limitations are far more complex than we realize.
And so are the impacts on us as both builders and users of AI products.
This one is going to hit hard.
So subscribe now to jump in with us next time.
