I'd say the beginning of this year, we were truly hill climbing, building an assistant.
And for anyone out there who has tried to do this, it is the easiest thing to demo and the hardest thing to build in terms of production.
Because when it comes to conversation, the user could ask anything.
And it is so difficult to succeed and provide good answers across all the topics that could be asked.
It's so easy to make a demo and so hard to get right.
And so I think that's what we really learned.
And one thing that was a big unlock for us, which people don't normally get to see.
And I think it's such a good demonstration for how creative PMs have to be in this day and age is how we do our evaluations.
We'd create this judge that is like ever evolving, right?
You're basically embodying all of your product intuition about how you want this agent to behave and you're putting it in this LLM judge.
We would take that and we would actually then give it to the entire development team.
And so every time any developer sits down to build on Sidekiq, and every time they say okay, I did the prompt, I added a new tool or I'm testing a new model, they would then run it against a bunch of simulated conversations.
And then they would have the LLM grade the conversations against that candidate build of Sidekiq.
And so we do this every time anyone sits down to build on Sidekiq.
We also run this LLM judge on live conversations when we launched to see how is this trending based on what users are actually asking.
So this becomes the tool that you use to guide the team.
It's so much more than a spec, because it becomes the measurement of whether or not we are achieving what we want as a product team in this agent, which is fascinating.
Creating great products isn't just about product managers and their day-to-day interactions with developers.
We'll dive into this system from every angle and help you think like a great product leader.
This is the Product Thinking Podcast.
Here's your host, Melissa Perry.
Hello, and welcome to another episode of the Product Thinking Podcast.
Today, I'm excited to have Vanessa Lee with us.
Vanessa is the VP of Product at Shopify, where she leads innovative efforts in agentic commerce and AI integration.
I'm thrilled to dive into how Shopify is setting the pace in agentic commerce and what lessons we can learn from their journey.
Welcome, Vanessa.
It's great to have you here.
It's good to be here, Melissa.
Thanks for having me.
Yeah, I'm so excited about this topic.
You guys have been really full-blown going crazy at Shopify with really cool new products.
Yeah.
We've been a bit loud about it from all respects.
So this is a meaty topic to dive into for sure.
I love it.
So can you tell us a little bit about how did you become the VP of product at Shopify?
What do you oversee and what was your journey to this moment?
Yeah.
I've been a founder a couple of times.
So I'm an engineer by education and I did a couple of startups.
I worked at Microsoft for a little, like a hot minute.
And then when I joined Shopify, I actually joined as a senior product manager.
And so I have seen every level of product management at Shopify.
I worked on a few different things at Shopify.
I started first building our developer platform and essentially our developer ecosystem in 2017.
And then I worked on our online store, which is what Shopify is known for.
It's our bread and butter.
And then I worked also on some of our AI initiatives.
So like things like Sidekick and what we do with generative AI, like media generation.
And then, so recently, I've gotten a chance to pair with, to be honest, like most of Shopify, which has been really fun.
And yeah, so it's been a journey.
It's been almost nine years, which is wild.
Wow.
That's awesome.
And so what have been like some of the key moments when you look at Shopify's journey, for you that define how you think about product management?
Oh, I feel like product management is different in every organization.
It's probably the one craft that swings wildly depending on the culture of the company.
Us in particular, I'd say also a very specific flavor of product management, but one that I personally couldn't imagine not working in.
So we are a very in-the-details product management culture.
I think it flows... from Toby, just all the way through our entire company.
We are the type of product managers who just absolutely love building and being in the details.
And our team is actually quite light when you look at ratios between PM and engineering.
So I think that kind of sets the tone for how much we expect PMs across the board, even all the way up to director.
I have a bunch of product directors who are way more in the weeds than you would expect a product director to be.
But I think, if you love building and you're super curious, Shopify's PM culture really speaks to those types of product managers.
And you are working right now on one of the hot button topics, I feel like.
I feel like it's like the hot button topics of all of tech.
So I feel like it's funny.
It's like almost a game now.
It's every time you say AI, I do something.
Take a shot, yeah.
But I know Shopify has been doing a lot of work around agentic AI.
Can you tell us what you've been working on lately and what's making you really excited about this?
Yeah, we are the Toby and Harley, and folks like to say we are the entrepreneurship company, which means our goal is to make sure that merchant's our merchants succeed.
And we believe that there's more entrepreneurs in the world than they are out there today that are just waiting to find their passion.
And so one of the things that we've really started out in now, I believe, a couple years, we put out this, essentially this vision for what became Sidekick, which is our AI assistant, to make sure that merchants have the someone by their side to help them succeed.
So, if you think about AI in that respect, this is like a co-pilot or an assistant that you can chat with that knows Shopify better than any other assistant out there.
Knows your business better than any assistant out there.
And that knows how to do things on your behalf.
So like it's able to proactively create products and collections for you, help you set up your payment methods.
Basically, anytime there's a hurdle in entrepreneurship, we wanted Sidekick to be there to help you through it, knowing how lonely entrepreneurship can be at times.
And In the early days with AI, outside of just smaller projects that we had around product description generation or image generation, we really envisioned a lot more than just the one-shot generation.
We also envisioned this assistant, this entity beside you.
And yeah, Sidekick really kicked it off.
And I'd say lately there's been a lot of advancements that we've seen over how Sidekick can go also beyond just someone you talk to and now moving into an assistant.
That is a lot more proactive, that folks turn to in a lot more of a general sense, versus just creating products, creating collections.
So I'd say, a lot of it in the early days was around our soul, our heart and soul as a company, which is how do we make more entrepreneurs successful?
And with Sidekick, I think people could understand it a little bit and think maybe it's just like a chatbot answering.
But it goes beyond that.
What are some use cases people will turn to Sidekick for and how does it work?
I think the best, well, Sidekick is just embedded inside of Vibe.
And I think one of the things that we're still working on is really how Sidekick will evolve over the next couple of years in terms of how it changes the UI dynamic, on how someone using Shopify day in and day out to run their business experiences Sidekick.
But to be honest, Sidekick is actually, from a UI perspective, pretty simple today.
It really is just embedded inside the admin.
But what was magical is that over the last couple of years, in the last, I'd say, the last 2025 in particular, we saw Sidekick really take off, And I think that's a mixture of two things.
So one, we have a audience, a merchant base who is super hungry for this companion.
Entrepreneurship is a very lonely journey.
It's not easy.
The cards are often stacked against you.
And so we had people who were just really hungry for help.
And so that was one piece.
Was we had just this user base who was ready for Sidekick to come and help them?
And then we also just hit it in terms of foundations.
Sidekick really improved in terms of answers and what it was able to do.
So Sidekick's able to do things that you would expect it to be able to do in Shopify things like creating a product, creating a collection, creating a discount for you, helping you navigating Shopify's admin easier.
But now we've started to see people really leverage Sidekick for, I'd say, like higher order tasks, things like hey, tell me about my sales over the last year.
What trends are you seeing?
What should I be doing in the next year to help me grow?
What's an attainable goal that you would see for my type of business in my industry, considering I had X number of dollars of revenue in the last month.
And I thought there's like one really key conversation that's such a good example.
There's this merchant who just went back and forth with Sidekick on like a Friday night over a hundred times.
And you really saw them start to just even talk to Psychic truly as if it was another member of their team.
And I think that was incredible because it meant Psychic.
For so long we had worked on the foundations, for a long time, and we finally broke through as to okay, now we're really providing useful answers for our merchants.
And that journey was not an easy one.
But yeah, I'd say like merchants use us a lot for higher order tasks, which was also a surprise.
It's like anything, like a data analyst, an advisor, a financial advisor understanding what marketing tactics they should explore.
I'd say that those are some of the surprising, most useful use cases for Psychic.
I love that it's basically an e-commerce expert embedded into the product, which is really changes the user experience for people right, who are not, let's say, e-commerce expert.
They're small business owners that know their brand or design clothes, but they're like how do I actually sell this?
So I love that you're giving them that whole experience in there too.
Yeah yeah, it's.
It's been such an amazing, probably one of the most rewarding products I've had a chance to work with, to build in my career.
Like even just watching merchants, they're like yeah, I just I've want to talk to Sidekick all the time.
And the fact that Sidekick just knows so much about their business is, I think, what unlocks a lot of that versus any other assistant that they could reach for that is around the internet.
Yeah.
And it has all the data.
So how did you come up with Sidekick?
Tell us a story about how you built this, how it got started, what that process was like.
Yeah.
So this is If you go back, you'll see Toby's original post.
I think this was, I can't remember the exact date, but I think it was in 2023.
And he really set the aspirations for what we wanted this to be.
I think like originally, even in Toby's video, I don't think you... see exactly where we're going.
But he already painted the picture pretty far into the future at that point.
So Sidekick really kicked off then.
I think once we saw LLM start to kick off ChatGPT, there was a moment, I think, that everyone who was working on especially SaaS-based software knew okay, this is a moment to pay attention to.
And I'd say that was like I'd say, at that year 2023, we were all just finding our way like at that point 2024, we started to be like okay, you started to see companies truly come out and say okay, we're going to start to have assistance, and assistance inside of our platforms.
And we're going to give that a go.
And so really for a while, I'd say, since Toby sent his video out and up until even, I'd say, the beginning of this year, we were truly hill climbing, building an assistant.
And for anyone out there who is tried to do this, it is the easiest thing to demo and the hardest thing to build in terms of production.
Because when it comes to conversation, the user could ask anything.
And it is so difficult to succeed and provide good answers across all the topics that could be asked.
It's so easy to make a demo and so hard to get right.
And so I think that's what we really learned.
And one thing that was a big unlock for us, which people don't normally get to see and I think it's such a good demonstration for how creative PMs have to be in this day and age is how we do our evaluations.
So when you're building an assistant, it's such an intricate system of what models am I using?
What Context am I providing the model?
Are we using several models that are working together?
What data is feeding that model?
Is it a fine-tuned model?
Is it a model that you're getting off the shelf?
And then also, how does it integrate with the platform?
There's so many variables in that.
And every time you change any of those variables, which your developers are doing all the time every time they do a PR for a new iteration on the prompt, they are changing a variable.
How do you know that it's not just, like, how do you test it, right?
When we build software up until this point, you have unit tests, you have your set of tests.
That you know.
Okay, it works.
How do you test when it's not a deterministic product?
And so what we did was we also had the cold start problem.
Everyone who builds an assistant has that problem where they just don't have the data yet.
So we had no data.
We had no example conversations.
And so one thing that we did, which I think is so ingenious and I definitely can't claim credit the team really did this.
I watched folks build a simulator for a merchant.
So you take all these topics and basically ask LLMs to help us generate all of these topics of give us like a thousand questions that merchants might ask in their journey.
And we would do it over and over again for different verticals of merchants, different maturity of a merchant, a scaling merchant, a merchant who's just starting.
And then we'd take all of those topics and then we'd feed it to another LLM that we have prompted to be a merchant.
So we said, you are a merchant.
Yes, you are a merchant.
You are a brand new merchant.
You are working out of your garage.
You are aged...
30 years old.
You like answers that are simple.
Your goal is to build your, I don't know, your, your lip gloss business out of your garage.
And then we would do, we do many merchant profiles.
We'd give them all of these corresponding topics.
And then we'd have them talk to a candidate of psychic, like our first build a psychic.
And we get all of this synthetic conversation.
We get these LLMs talking to LLMs, which is also hilarious, because you'd be surprised watching LLMs talk to each other.
They're just so willing to please that they just kept going on and on.
So there's a lot of work you have to do to actually get that to simulate, to be a real topic, like humans do.
But we got to a point where we'd be able to have these thousands upon thousands of synthetic conversations.
And then we would take them and we would actually start to mark them as a group of humans.
So they're like, let's say we had five humans and PMs do this all the time.
I think this is like one of those things that PMs don't realize when they build these products.
But the way you actually spec out what you want your agent to do is actually through this grading process, because you said hey, this conversation the agent went sidekick, went too long, did not answer the question correctly and created a product instead of creating a collection, which is what would have been the right answer.
I'll grade this and let's say Melissa, you're also a PM on the team.
You would also grade the same conversation.
And we would say, okay, what we care about for this product is like on five dimensions.
Did it serve the merchant's goals?
Did it follow our safety standards?
Did it actually use the right tools?
Was it grounded in the right way?
Did it have the right tone?
And so we would define all of these dimensions.
Then me and you would grade this particular conversation.
And funny enough, even humans don't agree with each other all the time, okay?
And so we would take this overlap and we'd say okay, Melissa and I, on average we have a agreement overlap of about 60, which means 60 of the time.
We would mark the same conversation the same way, across these dimensions.
And then we'd take that and we'd say, okay, we're going to build an LLM judge.
So this is like not proprietary.
I think this is just fascinating because if you read papers out there, you will find this.
This is not...
We are not the only ones to do this, but I don't think people tell this story enough.
Yeah.
We take an LLM judge and we actually say can we prompt this judge so that it would also overlap at 60 percent?
We want it to be able to slot into the team and be like, Vanessa's not here today.
So, you know, this LLM is going to judge.
And the way that they judge basically simulates Vanessa enough.
So once you have this LLM judge tuned means, you prompted, you iterated on it and you've ran it a couple of times again against all of the same markings and it turned out it would grade it the same way, me and you would, with the right overlap.
We call it Cohen's Kappa.
And then from there, we would say, okay, now we have our judge.
That process that I just called to you, we do that all the time.
So every time there is a new feature in Shopify, every time there, we believe that actually let's revisit.
We don't think that merchants are starting to change their behavior with Sidekick and we need to start to create essentially new features.
Ground truth sets for our judge, because people are asking now about marketing tactics.
Oh, we haven't graded marketing tactics before.
So me and you would do that again.
And then we'd create this judge that is like ever evolving, right?
So this is almost the new version of a spec.
When you think about PM, you're basically embodying all of your product intuition about what you want this, how you want this agent to behave, and you're putting it in this LLM judge.
I think it's so fascinating because you're getting to what I was going to ask about this, which is it's a different tool set and a different way of working, I think, with PMs, for PMs who work with more generative LLMs and agents, training agents.
Tell us a little bit about, now it's not a spec, is it evals?
Like what do we have to learn?
What are we doing here?
That's a different tool set.
Yeah.
So that LLM judge that we just trained, we would take that and we would actually then give it to the entire development team.
And so every time any developer sits down to build on Sidekiq, and every time they say okay, I iterated the prompt, I added a new tool or I'm testing a new model, they would then run it against a bunch of simulated conversations.
And then they would have the LLM grade the conversations against that candidate build of Sidekiq.
And so we do this every time anyone sits down to build on Psyche.
We do this many times.
We also run this LLM judge on live conversations when we launched to see how is this trending based on what users are actually asking.
So this LLM judge becomes the tool that you use to guide the team.
It becomes even more.
It's so much more than a spec, which is so fascinating because it becomes the measurement of success whether or not we are achieving what we want as a product team in this agent, which is fascinating.
Yeah, that's really cool.
So it's pretty much like designing the agent like how you want it to think and then embedding that into it.
And then it's basically you make the product for it, though, too.
Yeah, you're basically imparting all of your product intuition into an LLM to then shape yet another LLM which is so mind boggling.
But this is the type of creativity that I think.
When teams truly hit an acceleration with AI or building AI products, this is the type of creativity you need.
You do not find data just lying around.
You do not manually, okay, let me test this over and over again.
That's not going to scale.
And so I think that this is one of the fascinating stories that I love telling, because it's just such an amazing story of ingenuity.
Yeah, it's really cool.
It's extremely creative about how you handle that lack of data that we were talking about at the.
I get that question from newer PMs or people launching products so much where they're saying hey, if I have no data, how do I even start this?
How do I look at it?
You guys figured out a whole system on how to generate it.
I think everyone should start that way.
LLMs, they have so... much to offer in terms of capabilities.
I know we are all used to chatting with agents and using them to help us learn faster.
And I think those are all great applications of AI.
And we definitely do that a lot internally, where we expect gut checks, sense check, ideas with agents, especially as they've gotten better, especially this year.
But I also think that LLMs, we often forget, are useful tools in creating data and cleaning data.
And that's what with how the team has built psychic.
Probably at this point, like this is not, this is tribal knowledge.
I think for the teams who have built agents at scale, but I think for people who are just getting started.
You don't get to see that every day.
And I think that is, that's such a valuable lesson.
I'm excited to share that.
I'll be joining product weekend in New York city on November 14th, an incredible event powered by localized
In the last edition of may, I had the chance to meet a few people from the localized team and some of their clients, and it completely changed how I think about localization.
It's not just about translation, it can be a real growth driver.
In fact, some companies attribute up to half of their revenue growth to localization.
Localize helps teams do this at scale, with over 3000 companies using their AI powered localization platform to speed up translation, ensure quality and deliver better experiences worldwide.
Head over to localizecom to learn more and check out their localization revenue report to understand more about how localization can help scale your business.
People out there who are trying to get better at doing product management when it comes to generative AI and using LLMs.
What types of new skill sets would you advise them to look into to be able to do these things correctly?
Where do they start?
What should they learn?
Yeah.
To be honest, this part is... is hard.
I think these stories and why I love telling this one.
I tell it internally at Shopify all the time because, to be honest, even at a company as large as Shopify, this knowledge that I'm telling you is not uniform.
But I think the biggest advice I have is definitely everyone plays with AI.
I feel like that's not a new thing, but also see how you can use AI in more creative scenarios.
I think that's when you start to just get better ideas.
So like we started using AI even before Sidekick, we went through that loop.
Simultaneously, I had another team who was building our product taxonomy, which is a different application of AI, where we take LLMs and different ML models and we try to basically get a better idea of the products that merchants are selling so we can help them find the right channels to sell those products.
And every product is very different.
And Every merchant holds their product data in Shopify differently as well.
So sometimes you'd be like, okay, is this a garden hose or is this a lipstick?
It's funny because we were able to use LLMs to give us product attributes and suggest it to merchants to be like all right, it's a garden hose.
We can guess from the description and the title and the image.
But also garden hoses should typically have a length.
And it should have a color, and it should have.
These are all the attributes that you should fill in so that it can actually be sold properly when you connect it to marketplaces SEO, et cetera.
And we used LLMs also in that respect to help merchants basically accurately label all of the attributes of their product, which now have also helped merchants sell more accurately when they go to connect to other channels.
So these things where you're like, this is not like a normal application.
I'm not just chatting with an LLM.
I'm using it on a specific goal.
I think once you get that muscle going then you'll draw a lot more creative conclusions on other applications you could use it for.
But I think you just got to start.
Yeah.
You have to see what it's capable of doing.
Yeah.
I guess there's the limitations part of it too that we talk about with generative AI and LLMs.
Is there anything that you would call out that say hey, this is not well-suited for it, or here's some limitations that we found that we try to steer clear of?
To be honest, we use LLMs so ubiquitously across Shopify.
I'm trying to think of an application that wouldn't be good.
I think that's actually funny enough.
That's more challenging nowadays than finding applications which AI and LLMs are good good for, to the extent where I'm going to have to get back to you because I actually don't know what that would be.
I think everything in Shopify has been completely redefined and retouched in some way or other by LLMs and AI.
Oh, that's cool.
That's also saying like they're getting better and they're super smart at this point to lean on them.
Yeah, I recently had to onboard very quickly to our inventory system, for example.
And I wanted to be helpful to them.
And there's this funny story where I'm sure Toby would be fine if I told him this, but me and him were both going back and forth on the design of Shopify's inventory system.
And how do we make that better?
It's a really important part of Shopify.
And we started passing each other, essentially like chats that we had with AI about okay, what if it worked like this
And if we had virtual inventory groups and then we have a ledger and we actually use LLMs quite a lot in how we learn and how quickly we can ramp up in areas that we we either want a deeper understanding of or we could be new to.
So I think that's, it's a fun story.
Everyone uses it.
There's no ego in Shopify about not using AI to help us do our jobs.
That's for sure.
That's really cool.
And you're operationalizing AI at a massive scale.
Shopify, if it's touching everything, you have it across the platform.
What do you think, it's such a complex challenge.
What gives Shopify a unique edge in overcoming, doing this at scale being able to actually implement AI across such a large platform.
I think, thanks to just a lot of curiosity from Toby and from a lot of folks here, we've been willing to cutting edge on stuff that may or may not have been ready.
We were super early on AR, VR.
We were super bullish on QR codes before COVID.
And I think that even on things like, you know, stable coin and then now AI, I think there's a big belief that if we are on the cutting edge, then our merchants will also be on the cutting edge.
And so we're always bullish on.
Okay, let's do a 3D model and make it possible for someone to upload a 3D model in their online store so that you could see the details of this stroller.
I remember doing that and that was way before, I think, merchants even understood.
How do I get a 3D model of my products made?
But I think we always trusted that if we did our jobs and made it easy on our side of the platform, that over time a lot of these bets will pay off.
I think AI was just one of those where very early, we could tell, okay, this is a bet.
Oh, this is more than this bet is very likely to hit.
And then I think we just really encouraged everyone to play with it and to use it and to not be afraid of it.
And I think that comes all the way from Toby all the way down.
And I think that's really changed the way that we use AI and then how we build with AI as well.
When you think about the vision in the future for Shopify, and especially when it comes to agentic commerce, how do you see that playing out?
Can you tell us a little bit about what agentic commerce actually means as well?
And where do you think this is going in the future?
Yeah, I think...
If you Google agentic commerce, you will come up with many results.
So agentic commerce, I think, is really like this big umbrella term about how are we going to change shopping in the advent of agents.
And it goes everywhere from how are we going to change how we discover products to buy and how we live our lives?
For example, the ones that I'm personally excited about are things like helping people spend less time searching.
I want to be able to say, hey, it's my five-year-old's birthday. this year.
Help me plan it and help me procure everything.
Okay, great.
That looks good.
Purchase.
And that just saves so much time.
And so I think that there's just this advent of okay, there is an assistant that is helping you with all aspects of your life, including the parts that perhaps require a bridge into physical goods, right?
Purchasing things.
How does AI play in that space?
This is something that we've been excited about for a long time, to be honest, because we do think that is.
Humans are anything but very smart and lazy beings, which is what makes us great tool makers.
And so we trusted that this would happen.
So we started planning for this a while ago, a couple of years ago, even with the taxonomy, the product taxonomy work that I told you about, like understanding what products are what, so that we could serve up to an agent okay, this is a garden hose.
It's good.
It has great reviews.
It's green.
It's eight feet, right?
If we could understand our merchants' products better, then we could better help integrate them in a future not even that far in the future where agents were helping to actually purchase on behalf of users.
So there's the discovery portion of this.
And then there's the actual execution and checkout.
We have one of the most amazing checkout infrastructures.
I think that has been our bread and butter.
Yeah, some of the largest flash sellers in the world come to Shopify because of what we can handle at scale.
And then also we're the only checkout platform that have millions of merchants on a single platform.
So cool.
I love that I can just go to a random clothing website or something like, oh, let me check out.
And do you want to log into your shop account?
Yes.
I don't want to create 18,000 other accounts.
That's amazing.
I remember the first time I saw it, I was like, this is Yeah, it's so good.
And funny story inside story.
That was a Hack Days project inside a shop a long time ago, which was clearly like a great idea that has then just absolutely just exceeded all of our expectations.
It's been such a great thing for merchants to have that.
And such a great, like everyone that I talked to is, oh, I love like logging with shop.
Shop checkout is the best.
So we have this great checkout, but checkout is so much more than just a payment like when you log in with shop.
You don't need to put in your payment credentials, you have your address field.
But our checkout.
What has been so fascinating is, over the last half decade we've invested in making checkout modular.
So our checkout, if you're a merchant let's say you're someone really large, i have a lot of gmb, i'm all birds, i can use shopify's checkout and i can customize it.
I can say you know what i need to show you.
Uh, like you know, in canada right now we have a strike on our postal service.
I need to tell you, I need to tell all my Canadian customers hey, there might be delays in shipping, or our shipping costs are different, or whatever.
So this is all the way to a furniture store where they need to do a white glove delivery.
And so they're like, hey, Melissa, when do you want this chair delivered?
And so we have all of these extension points, we call extension points in our checkout.
But what's great with us investing in that for the last five years is now we're at this point where our checkout is a single platform that works across millions of businesses.
That in a dime and we launched this in August as a checkout sheet you can integrate into your agent and it can come up and be styled to be, let's say, your background is think in your agent.
You can style it to feel native to your agent and it will work across millions of businesses.
And I think that is just amazing.
Something that is like incredible, that people don't normally think about when they think of agentic is like how do you check out?
And so that's something we've invested in for the last half decade.
So it's like everything from discovery all the way to checkout.
I think is like something that you will start to see more agents adopt over the next couple of years.
And then when you think about AI to commerce, especially with small brands compared to like, large global brands, how do you think it differs?
How does your product philosophy differ to serve the one person entrepreneur, versus the large global brands?
Like we're talking about Allbirds, like huge company, huge sneaker company.
Yeah, I think that there are.
I think the one thing that we always try and do at Shopify is we always try and make sure that we're democratizing access to customers and helping all entrepreneurs succeed, whether it's a small merchant or a really large merchant.
I do see differences in terms of how they reach customers and consumers.
We want to make sure that's pretty uniform.
So we're going to do our job to make sure that even the two-person company trying to build something is going to show up in all the right places, no matter what channels happen.
We had social media on the rise, and now we have agentic.
This is new and not new at the same time for us.
We've seen so many rises in search being the biggest thing when we first came out as a platform in like 2010, being the primary way.
Then social media.
And so this is just the next wave.
And every time we try and make sure that we're at the forefront, we see a large difference in terms of how AI is leveraged, to be honest, by those folks, which has been fascinating.
Yeah, I think that there's just this scrappy resilience that smaller entrepreneurs have, which is fascinating.
And so they actually are some of the most hungry users of our AI tools.
So like they're generating media using Sidekick for their ads.
They're generating media to help them with their product images.
Super into leveraging Sidekick to change all of their SEO tags to be more optimized.
And so you see them use it a lot.
For larger merchants, I think it's a lot more like what that specific person is doing.
So this is like internal to the organization.
A marketer will use Sidekick for their space specific job.
Could tell me about the data of the store?
But we actually see more uptick, I'd say, a lot of uptick in smaller merchants.
And then when it comes to really large merchants, it's really on specific tasks, right?
Like specific tasks are data analysis and trying to understand what should my marketing tactic be coming up?
So they're different.
I think there's more holistic usage coming from smaller merchants.
And for larger merchants.
You just see it more and okay, they have a very particular role that they're using Sidekick for.
That's fascinating.
Yeah.
Do you think that has to do with the?
You were talking about the scrappy nature of the entrepreneurs, but they don't have a whole team around it.
And do you think it's more about like, the specific roles that people play in larger companies, where it's like how do I use this to help me rather than how do I look at it globally?
100%.
100%.
Like that one entrepreneur needs to do all the things.
Yeah.
Right.
Whereas that marketing manager is trying to do their best job in their role.
In marketing.
Yeah.
In marketing.
And the fascinating thing is how many, I'd say like executives at larger merchants they have been using it more holistically.
Oh, interesting.
Yeah, because then they're able to oversee like a larger area.
Okay, like help me brainstorm.
What customer segments should I go after next quarter that we haven't tapped into yet?
Build them for me.
So there is some stuff that we're seeing, but yeah, for sure the early entrepreneurs, they are...
They're a different breed of a person.
They're just so stubborn and just scrappy and resourceful.
And we see that in the way that you sidekick for sure.
That's really cool.
And then you have been partnering with OpenAI's agentic commerce protocol.
Can you talk a little bit about that partnership and how that came about, how you're using it?
Yeah, I mean, we've worked closely with OpenAI for years, not just as a partner, but also as a customer, to be frank, because we use a lot of their models.
We use a lot of models from a lot of different partners, but they for sure are one of them.
And last year actually, we started talking about okay, how do we get customers, how to get product discovery in?
And so earlier this year we started getting together to talk about okay, how do we bring commerce to ChatGPT?
And right, I think this was just announced.
We're super excited to start to roll this out.
But essentially we partner with them to make sure that our merchants products are discoverable and that they're able to check out directly inside of ChatGPT.
And so that's, been, yeah, that's been super fun to work with them on.
And I think that commerce will start to take over, not just, it's going to become ingrained in every agent.
And so I think you will start to see a lot of activity across the board, but we are super pumped to be working on this with OpenAI.
Yeah.
When I saw the launch about this, I got really excited because I was like man, I am using ChatGPT to brainstorm all these gift ideas and stuff.
And then they're just linking out to it.
I'm like, I just want to buy it here.
I
100%.
Yeah.
And I think that's part of why we decided, okay, let's work together on this.
Let's start to work on how do we make that possible for so many consumers?
Do you think that's also going to change the way that that people think about marketing their shops as well?
Because now it's going to be like, how do you show up in these searches?
What do small business or merchants or anybody really large businesses to have to think about now?
About how we talk about our products and what we offer and how they're marketed to different people?
I think that is probably the top question that I get, not just about this particular integration with ChatGPT, but also at large.
I think merchants could smell this having gone through SEO, then social media.
They are very in tune with where consumers are changing behavior.
Yeah, I do think that will change.
So a lot of that has been okay.
How do we help merchants make sure that their product data is correct?
Because a lot of the models currently they're just starting to get awareness on.
Okay, what are these products that I can search for?
Can I search for it accurately?
Am I pulling the feet versus 10 feet?
Because that's the type of nuance you get with agents, right?
You're like, hey, I'm trying to set up a sprinkler system in my front yard.
It's about 10 feet long.
What's a good garden hose for me?
And it'll give you, it'll give you the exact one that you want.
But in order for that to be the case, it needs to know that this product is eight feet.
This product is 10 feet.
And so we really started with the foundations there.
It's like, how do we create a standard product taxonomy?
Now from there, every merchant's going to be like, how do I make sure that I stand out?
And this is where we as a platform have tried to make sure that all of the stuff product descriptions, reviews about products are all within Shopify so that it can then be served to like in one shot, in one integration with all of these partners.
We always say.
I think one of the superpowers of being on Shopify is the fact that we're millions of merchants.
And so we can go and talk about these partnerships with folks like OpenAI Microsoft, all these partnerships that we've brokered over the years because we represent millions of merchants.
And it's one integration that they can build.
And then all of these businesses can get access to it.
And so I do think that we will invest a lot more in.
Okay, how do we make sure that you continue to show up properly?
And how do you want your brand to show up?
We launched something in Quietly called like the knowledge-based app, where you can install that application.
You can see what questions are being asked of your products, of your store, and you can provide grounding data.
So you can start to understand.
Okay, I want to be like I am.
I do not have any tariffs or duties on my product.
Because that's a question that I get asked a lot.
So I want to make sure that's clear.
So we are experimenting, but I think we'll continue to see more of it in the next six months.
When you think about the future of product management, what do you think are some emerging trends that are going to have a significant impact on it and like our whole discipline in the next couple of years?
Oh, I think I am so excited to see where product management will be.
Even inside Shopify, we started...
A vibe coding interview, pairing interview.
Yes.
Because what I want product managers and Shopify to do and I think it's so exciting for the industry at large, all product managers is we can now prototype stuff that we would typically have to rely on the dev team being free.
And we can just tinker with it.
And we can understand how it feels.
And that process, that feedback loop is so tight.
And so I think that that is such a superpower.
Of course, like engineers, not all of this code is actually ready to be actually put into production.
But even that alone, being able to gut check ideas is huge.
And I expect PMs to do this all the time to be able to communicate better.
PMs don't.
It's fascinating.
I would say this. we don't actually do any real work.
Our work is actually in communication and helping to make sure that everyone is building the right thing at the right time.
But that goes building the right thing is not an easy thing to do.
You can't just throw requirements on there are a million ways to solve any problem.
And in being able to do a prototype quickly, you can quickly see Is it working out?
Is it actually how I want it to feel?
Especially for very high interaction products, where you want to understand is this going to feel right?
You think about the swiping gesture that was introduced by Tinder and others.
And then now that's very native in a lot of different UIs.
But you wouldn't come up with that without having this tactile experience and feedback loop.
And I think that PMs now getting the ability to do that independently is, And then now taking okay team here's a more clear, crystallized view of what I'm thinking means that their teams can now be more effective when they're bringing it to production.
So I just expect this to be a huge force and function for us to move faster as a whole discipline.
Yeah, I'm super excited about the whole thing.
Fine coding thing.
I'm worried because i think people are gonna put junk code into their systems.
But yes, you do not merge your prs without someone reviewing it for sure.
But i love the fact that we can now prototype, test with users, test with ourselves, just like.
Communicate things to so much more easily, instead of just waiting and trying to translate it through developers and it's okay.
We can just look at it like you're saying, and then see if it's feeling right.
If it's not feeling right, get in front of people.
Like it's huge to me that we can now do this A hundred percent.
It is magic.
Amazing.
Vanessa, my last question for you.
If you could give one piece of advice to your younger self, what would it be?
I think one of the things that I realized and I wish I realized this earlier is that, not only just as a PM, it's what led me to, I think, PM and staying in this role for this long is instead of chasing titles or like particular companies.
Think about how you want to spend your time and then go find the place where you're going to get to do that.
And so if you really enjoy collaborating, go find the place where you're going to get to collaborate with people the most.
If you enjoy building, then go find the place that you're going to get hands-on building.
I think sometimes I talk to a lot of PMs and PM has grown very quickly into quite, and when I started at PM wasn't that much of a thing.
It was just getting started.
I think a lot of people are coming into PM because it's starting to become something that is known to be a good job.
But I think PM is actually quite difficult for anyone who's been in the industry.
It is a slog.
It is 80% slog, 10% the best, 10% the worst.
And I think you have to know why you love it.
So if you love talking to people, then maybe you'd be in a PM environment where you're having to talk to a lot of customers.
But I think it's how you spend your time, because I think, more than anything, PM is actually being a source of energy and courage to your teams.
And I think to have that and to make sure you have a good career doing it, you have to know which environments you're going to be able to supply that energy, because you get the energy from essentially building.
So they have a really good idea of what gives you energy so that you can give it back to your teams.
I think that's great advice.
And man, great advice for people out there too thinking about do I want to work at a small company?
A large company?
Like what types of company too?
Because I've seen so many people like, let's say, from enterprise companies go oh, I really want to work on a startup.
And then they get in there and they're like, not like that.
So it's honing in on those factors.
I think that's fantastic advice.
Thank you so much, Vanessa, for joining us.
If people want to learn more about you and your work in the go.
You can find me on x.com.
I'm not there all the time, but I do lurk and I do pay attention.
Yeah.
All right.
Thank you for having me.
Yeah.
Thanks for joining us.
So we will put the links to reach out to Vanessa on X and also to Shopify and more information about their agentic commerce on our show notes at productthinkingpodcastcom.
Thank you so much for listening to the Product Thinking Podcast.
We'll be back next week.
And, in the meantime, if you have any questions for me, go to dearmelissacom and let me know what they are.
I answer them every single week and make sure that you like and subscribe so that you never miss an episode.
We'll see you next time.
