When you have a product, it's funny, even though you're very proud of it, you're also very critical of it because you know, all of the flaws you have. We also knew it was magical something new.
I do not come from MLOps or AI.
So if you were to ask me, Mario, you designed a product, you get it out there and 70 % of people do not actually accept.
The suggestion, I'm telling you, that's not a good product, but in this case, it was completely different.
So that surprised me is, Oh my God, only a 30 % acceptance or 20 % acceptance can create something that changes the world on this.
And that was the learning.
Even for me, we just finished a set of AI days in one of the teams. They just took an entire day to learn more and more, even though it's like, we're so busy that sometimes we miss announcements out there and we have to take a series of days just to learn what is in the industry.
And how can we use that appropriately as well.
They go and use a lot of products, including Copilot by the way, and they try to go and use it in novel ways and see what they could learn.
Then share in a channel at the end of the day, everything we learned.
And so I would say AI days inside your company is something like really powerful.
Creating great products isn't just about product managers and their day -to -day interactions with developers.
It's about how an organization supports products as a whole.
The systems, the processes and cultures in place that help companies deliver value to their customers.
With the help of some boundary pushing guests and inspiration from your most pressing product questions.
We'll dive into this system from every angle and help you think like a great product leader.
This is the Product Thinking Podcast. Here's your host, Melissa Perry.
Hello, and welcome to another episode of the Product Thinking Podcast. Today, I am super excited to introduce Mario Rodriguez, the chief product officer at GitHub.
Mario oversees GitHub Co -Pilot, the AI -powered developer platform trusted by more than 150 million developers and 77 ,000 organizations.
I think co -pilot is almost a household name at this point.
I'm thrilled to discuss with Mario how product leaders can adapt to the evolving world of AI and emerging technologies.
But before we talk to Mario, it's time for Dear Melissa.
So this is a segment of the show where you can ask me any of your burning product management questions, and I answer them here every single week.
Go to DearMelissa .com and let me know what's on your mind.
Hey, product people, I have some very exciting news.
Our new Mastering Product Strategy course is now live on Product Institute.
I've been working on this course for years to help product leaders tackle one of the biggest challenges I see every day, creating product strategies that drive real business results.
If you're ready to level up your strategy skills, head over to productinstitute .com and use code LAUNCH for $200 off at checkout.
Here's this week's question.
How would you change your approach to discovery when you're responsible for reducing fraud on a platform.
So I think the key here is to remember that discovery is about understanding what the problem is.
So at this point, you have a business problem, which is reducing fraud.
So we still have to be able to protect the revenue of the business and protect the business itself.
Now you want to go back and figure out what's causing that fraud.
Right? If you have too much fraud, it always will come back to a customer problem too, because you might take very stringent controls and not allow people to transact and fraud will bring down your platform and then you can't provide your service.
So at the end of the day, this becomes a customer problem if you can't keep operating too, both a business and a customer problem.
But in discovery, the idea here is to figure out what is causing this problem.
So I would take it back and start to say, where's our fraud coming from?
Is it coming into from certain areas?
Is it from certain pockets of things?
And then you wanna understand to the user behavior or the things leading up to that fraud actually happening.
right? So this makes you take on a role of people actually transacting with your platform, coming out onto the outside of it, empathizing with that customer, and trying to figure out how to intercept it and figure out how we can actually test and solve those problems in that way.
So when it comes to discoveries, not just about customer interviews, and in this case, I'm sure your fraudulent customers do not want to be interviewed, that's okay.
It's about solving problems, right?
Evaluating the problem, identifying the problem, and in this case, got a business problem.
How do I actually break that down in a systematic way?
Try to figure out what the leading indicators are showing that this fraud might be, and then figure out how to protect the business.
That's what I would do from a discovery perspective and try to get really smart on that, try to understand where the bad actors are coming from, what are their motivations, how do you actually anticipate that, there's some empathy there because you do have to empathize with your fraudsters.
Put yourself in their shoes and figure out how do I intercept this and start to make this place a little bit safer.
So I hope that helps.
And I wish you the best on reducing your fraud.
Now, let's go talk to Mario.
Welcome, Mario, it's great to have you on the podcast. Thank you for having me, Melisa.
I'm really excited on being here and really talk more about product and AI.
Yeah, and I'm so excited to dive into GitHub co -pilot because I think everybody is talking about it.
But first, can you tell us a little bit about how you came to be the chief product officer of GitHub?
Yes, it's a real story.
Maybe I'll start all the way back.
I joined actually Microsoft in 2002 and worked for games.
And that was an amazing experience.
I got to work with many great producers, Peter Molyneux, and people, like, really loved their craft. And I ended up doing this game called Forza Motorsport.
And as I was diving deeper and deeper into that space, I'm like, I'm gonna become a producer of games.
And then after doing it for a little while, I'm like, I really like product, maybe not games overall, but maybe something in the software side.
And I was reflecting about, okay, if I'm wanna go and invest in some place, is it gonna be mainly in a consumer product or is it gonna be on the backend of things?
And I was like, I think my passion really lies for developer tools.
So I was lucky enough to get a job as a PM1, I did a switch from like software engineering to product.
Many people do that.
And I got a job at this group called Team Foundation Server.
And since then for 20 plus years, I've been doing developer tools.
And I love doing developer tools like I think is the way that I describe it to my wife and my kids, it's look, humanity already invented fire, the wheel, And everything we're gonna be creating from this point on where it's gonna be powered by software.
And if I'm gonna be here on earth, on a limited time giving software developers the best tools available in the market, that seems like a life pretty good spend.
So I had this passion for developer tools.
The opportunity came also where Microsoft was acquiring GitHub.
And I moved in into that team then at that moment joined GitHub around 2018, and was helping at the very beginning on enterprise and making more, expanding GitHub from what it was at that moment, which is the home of open -source and having the best version control product into this end -to -end platform that can take you from idea to production.
And that was really exciting for me because if you expand what that amazing product that had a lot of taste, a lot of developer love, and make it into this thing that everyone can use, then you could accelerate human progress because of that.
So I worked a little bit on enterprise, then on collaboration tools, on planning and tracking.
I took what I call inside of GitHub a hiatus for a year and a half.
And I'm like, we need to get issues and project management in GitHub for product managers to be better than what it is today.
I spent a year and a half doing that.
And then I got an amazing opportunity to lead copilot and that product took off and then from there, I transitioned into the SHIELD product officer role.
So that's a little bit of a story behind all of that.
Amazing. And I love your passion for developer tools.
It's definitely coming through.
So let's jump into it.
Let's talk about copilot.
Cause this is the, the thing that everybody is talking about these days.
GitHub super successful with your launch of copilot.
Tell us how it got started.
How did you start to imagine this like AI native product and what AI could do to help developers to fit seamlessly in?
Yes. And yeah, one of the things that I actually also love about co -pilot is the core part.
We have this thing where the human is at the center and then we're augmenting you to do more and to really spend time in the things that us humans are really good at, which is creativity or we have a lot of advantage on, which is creativity.
But yeah, and GitHub has this thing called the GitHub Next team.
And the GitHub Next team is in charge of looking for not only in Horizon 1 initiatives that can make GitHub better, but really more about Horizon 2 think two years from now, Horizon 3 thinks that may never actually happen as well.
And they had been in talks with OpenAI, and OpenAI had this thing, Code GPT -3.
Internally, I think, was called something like CodeX for us at the Codex model.
And they had been collaborating with them on that technology, essentially saying, look, these LLMs are very special at understanding natural language and solving problems, specifically in the coding space.
And we have seen that played out.
These models have gotten significantly better in natural language understanding and significantly better in understanding code and being able to code as well.
And the GitHub Next team created a paper I said, is this science or fiction more or less if you want to think about it that way.
And that was to start with in GitHub on, Oh my god, is this something special or not?
When I first saw it, what really impressed me and why I said, wow, like, this is probably going to change everything.
In our space, we have been able to autocomplete code for a long time.
Then we have this technology called IntelliSense, and there have been a lot of ML models that helped you do those types of things.
But what I never experienced in my life is you being able to go through a comment and describe something in natural language and then have the system or the AI model understand that and then translate that into code.
And that natural language conversion to code was something, again, something that I had to go all -in on this.
Now, the interesting part of this dilemma is the following.
What we were playing with, at least in the research lab, was just a technology.
Like, it could do things like solve Python problems, sometimes will take a hundred shots to do it, which is not something that you could ship in a product.
Like, just imagine to solve anything you have to tell it to do something a hundred times.
That's not something that anyone will buy.
So we have this amazing technology, we have these pre -existing tools that developers use, how do you marry those two things?
And that's easier said than done, sometimes you're gonna be too early with the technology, sometimes you actually gonna go and put it in a product in the wrong way, and then you're gonna not necessarily waste the technology but you're too early with the wrong assumptions as well.
And, yeah, and that's the craft of product in my opinion.
It's that complexity of the problems, it's no matter how good my PRD is, at the end, the product has to actually make it and be good.
So we played a lot with that tech into making it into GitHub compiler.
The simplicity of what we shipped at the beginning was incredible.
There were only three things that make compiler successful in my opinion.
Number one is we actually made it so as you're coding you're not interrupted.
We had this thing called ghost text, and as you were doing your normal coding then it will come in and then tell you, do you wanna take this suggestion or not?
And very beginning, it was a pop up and we had many ways of doing that in the UX and we found one that worked.
The second thing was it had to be fast because we chose the modality on showing you a suggestion.
It had to be really fast. So we got it down to a hundred milliseconds which really meant that we kept you in the flow developing as well.
And the third thing is we needed to bring more context in.
So the suggestion was something that was more personalized to the code base and to you.
And not to what the model actually knew about in the world.
A lot of people get confused.
Like they think that the suggestions come from all of this corpus of code.
And just a little bit of that is true.
The contexts and the LLN together with that, it's what really generates a new thing altogether.
And we just needed to tune that.
And those three things is what made QPalis successful The fact that it was a ghost text, the fact that it was super fast, and the fact that it was contextual and it gave you pretty high quality suggestions because of that, and then we just watched that take off.
We were the first co -pilot in the name and in the product as well, and we're very proud of that.
How did you settle on those three factors?
What was the process to actually figure out what was going to make it successful?
Yes, it's interesting because I still remember it, to a lot of what we do at GitHub, we operate on this set of principles, right?
And we're very much, and you know this very well because of the build trap.
We're very much outcome driven instead of output driven on these things.
And because of that, we also value the learning loop or your iteration velocity through that.
So if there is a week and we could do three experiments, we learned three times.
If in a week we could only do one experiment and want to learn once so we value learning fast. So the way that we got there was through a bunch of failures believe it or not.
It's, you try this thing and you're like, nope, that did not work from a UX modality.
You try this other one, that did not work.
You try this other one.
You're like, this is starting to feel like it.
So you play a lot with the product, you experiment a lot and then just learn or learn.
And if you do that and you have some people that have good taste, also have some expertise in the field because you do need expertise in developer tools.
Then you start getting to a place where you're like, okay now I could actually iterate towards something that is a good product.
So those are the things just through a lot of iteration and then a significant amount of engineering work then to figure out how to make it faster, how to do this thing called feel in the middle which was very important to us because a lot of development doesn't just happen on a new line at times.
So you do want to fill it in the middle as well.
So then you start marrying the technology with the product, but it was through failures, which is something like I don't think the product discipline talks a lot about.
We like to center a lot of successes of it, but I actually think you end up creating a better product through failures than through successes the majority of the time.
Yeah. What were some of the things that you tried that you decided through testing were not what you were gonna go with?
Yeah, so like I said, like sometimes at the very beginning, you would just be asking Cobra Alert a question, a motto, and that was just too disruptive.
We tried kind of other things where the suggestions would appear on the right hand side of it, to another panel.
So it's not a motto, but it's not in your flow.
And what we found out repeatedly is if it's not in the flow, it just does not get used.
So then you're starting to think about, okay, then how do you put it in the flow?
Okay, The best way is to be on top of what IntelliSense already does today, but having a slightly different treatment on it.
Okay, so then do that, does that feel good with no technology limitations?
Yes. Okay, then how do you marry down with the technology?
So it was a lot of DevEx or experiences and that we fail, and we ended up in no, it needs to be in line.
And then the other stuff is through a lot of iteration is the context right?
No. Why is it not right?
Then you do these evals.
Maybe we can talk about that later, but like you do a lot of e -balls and then you play with the product a significant amount as well.
So you have a little bit of your own intuition together with some e -balls then that gets you into a place that you could iterate towards a product you could ship.
Yeah. What were, let's get into e -balls.
Like, what are you running there?
How's it working? Yeah, I think if you ask me, what is the most important thing about creating AI products?
I would hands down tell you, e -balls.
and there's two way that we think about it, there's of course, offline evaluation.
And the key in there is to get the right test. But once you have the right test, you will find out that you can pass your offline evils.
There's somehow in your organization, there will be an incentive system that you could get very good at your offline evaluations.
Okay, so what we started noticing is, okay, we're starting to get really good on offline evils, but we're not seeing that in some parts where we're playing.
So now let's go and evolve from offline evaluations to online evaluations.
Then you have to set up all of that infrastructure because you have privacy in mind, you have to set up flyding.
And to a company like us, that's really easy sometimes to do.
But if you think your average enterprise company, like how do you set up flydings and all of those things to 50 ,000 teller machines or something like that, that's a lot more involved engineering wise for some of the enterprise companies.
For us, you had to set up your online evils.
And then you start getting, oh my God, we're working offline evils.
Doesn't work as well online.
And by the way, in my opinion, in product and AI, that should be your expectation in my opinion.
It's that your auth plan is just gonna make you feel a little bit better about what is happening, but it's not gonna be 100 % representative, especially a scale of what is really gonna happen in your product.
And let me tell you, like, you make very different set of decisions based on the success rate of some of these AI features.
Meaning, if let's say a suggestion only gets accepted 20%, if you get your suggestion rate to be 50 % or your success rate to be 50 % or you get your success rate to be 90%, the actual product will completely change because at 20 % you're gonna have to walk through the user through a lot of failure points and be like, oh, you think it's not that good yet, but for this thing, it works really well.
So you're trying to get them to the 20 % that really works.
When you're at the 90%, your product works well at scale.
And then you're really trying to design this how is the experience gonna differ when it doesn't, and when it does, how are you gonna continue to have that trust overall?
And then when it works 99%, you're just moving fast and you probably don't care almost that if what it does that doesn't do well.
So we set up online evils, we have our offline evils.
And then even after that, you have to have this other thing which is, okay, what are the metrics you're gonna have?
How are those metrics gonna get gained essentially?
And then how are you gonna listen to, this one gets better but this one, I would say suffers because of it, but net -net the actual thing for the user was better or NANET it was significantly worse than what they were experiencing before.
So for us, it was a lot of then trial and learning on the actual metrics.
Once you have your offline evil, your online evils, then you have to choose this metrics that really makes sense overall.
I could give you a couple of examples.
Yeah, what kind of metrics made sense for you?
So like for example, accepted or acceptance rate or AR for us.
And acceptance rate is if you ask a binder for something and we suggest it or maybe you didn't even ask Kovalio, we just show you a suggestion.
Are you accepted or are you not accepted?
It's very binary, right?
Yes or no. We have another one that is accepted and retained characters.
So after you accept it, how much of what you accepted did you change?
And then we have other ones that have to do with, okay, based on the suggestion, if we suggested one line what is then the arc or and return car insurance.
And if we suggested 20 lines, how much of that do you ended up changing?
It's very different.
So as an example, I could write accepted rates if what I'm suggesting to you is smaller.
So when you're in Gmail or when you're in Google products and they give you a suggestion you can see for the most part, it's very short and hence you're probably gonna accept it.
But if they gonna suggest you a whole paragraph you're probably gonna accept it and then change a bunch of things on it.
So we started measuring a lot more things to have to do, not only with accepted, but how much would you accept?
Why did you ended up accepting English languages?
And then how much do you ended up changing?
And how did it change as well?
And the combination of all of those things together with latency, how fast we ended up giving you that, puts a story behind the quality of the product overall.
And then we have a lot of what I call must -have tests that always need to pass no matter what, because we know if we're regressing that and the quality of the experience is just not gonna be good.
And then we have other things that we're like, are the models getting better?
Things still didn't work in the last iteration of the model and we're gonna try the new models and try to see how good they're doing on this one.
And there's a lot of benchmarks out there, the software engineer benchmark or suite, verified is one of those, as an example.
How are you using those metrics or the feedback to help improve the models?
Is it automatically learning about them Are you going through and looking at what's happening and then taking that and deciding how to train the model?
What's the feedback mechanism there in the learning?
So what I would say is a lot of what we do is we do flights of AB, and then we run them all the way to, we get some statistically significant or some style on that that we're like, okay, we can trust this, and we usually go ahead and do the old one and the new one.
So there those IP tests, and there are times where we were like, we don't even care about the old one.
We just gonna run two versions of DDoS.
So we actually doing like B and C, no A, no actually control group of what exists today, and there's good reasons to do that at times when you're feeling pretty good about some tag, then actually looking at A it's not gonna give you anything and all you're gonna do is being slower in making a set of decisions, in my opinion.
So sometimes we just do B and C, but what we do is we fly them either A and B or some type of B and C, and then we take a look at them.
Usually for us, nowadays it takes us around seven days to get something back that we could trust, sometimes depending on where we put it, it might take us hours only, but for the most part we try to do three days, so looking at the last three days, or look at the last seven days and then we maybe meet on Monday or meet on Tuesday, we take a look at those evolves and then we're like, okay, where's it failing and where's it doing good?
So sometimes, an example, we do something in our life, hey, AR, it's shooting up through the roof, people are accepting things like crazy.
And then, by then we're seeing ARC, which is our accepted and retained characters, go down significantly.
We're like, that's not what we wanna do.
If all the time you're accepting a bunch of things and then changing it, you're probably gonna be pretty upset at copilot as an example.
And then we have other metrics that we try to control as well.
But it's think about it on a weekly basis, We're trying to flight something and learn from it.
I think Kevin Scott, who's the CTO of Microsoft, always said, if you're not flighting or learning something every week in AI, you're probably doing it wrong.
And sometimes in Christmas stuff, we don't do that.
But for the most part, we try to be always learning every week.
When you launched Copilot and you started watching people use it, what surprised you?
What kind of things did you observe and trends and how developers are interacting with it?
Yeah, when you have a product, it's funny, even though you're very proud of it, you're also very critical of it because you know all of the flaws you have. And we knew look, we developed for a living, GitHub is a developer company, right?
So we knew the places that it was not doing well, whenever we were using it on a daily basis.
We also knew it was magical, something new.
And when you get innovation out there, what really made it for is how many people used it and how many people absolutely loved it.
And even though it, maybe when we first released it, acceptance rates were like in the 20%, 30%.
So just think about this, 70 % of the time that we suggested something to you, you did not accept it.
So you would think that makes a horrible product, but no, because of all of the value when you did, people just absolutely loved it.
And that was a learning even for me.
I do not come from MLOps or AI.
So if you were to ask me, Mario, you design a product, you get it out there, and 70 % of the people do not actually accept the suggestion.
I'm telling you, that's not a good product, but in this case, it was completely different.
So that surprised me is, oh my God, only a 30 % acceptance or 20 % acceptance can create something that changes the world on And I could see it.
Then the second thing it was, this one didn't surprise me, but there were so many other cool things that people did with it.
How many people ended up programming just by giving comments to Call Pilot instead of having to write the entirety of that code.
And how many people did that across many programming languages and natural language.
So I could be programming Spanish as an example, or at least learning how to program in Spanish as another example.
So I think that was one of the things that also surprised me, is the ability to use that model in multiple languages, not only programming languages, but also the way we speak.
That's really cool.
When you came into GitHub too, you mentioned you started working on bringing it into more enterprises, expanding it there.
How have you seen the adoption of co -pilot in larger companies versus startups and scale ups?
What's been the trend of adopting AI across that?
Yeah. Yeah, it's exponential and you're having a lot of fun, but it feels like a tornado.
I think there's a book out there called Inside the Tornado and it was pre -content how Intel grew and all of those type of things.
That's how it felt because you're having this product that essentially people are taking orders trying to get it.
But at the same time you have to, it's a brand new way of programming.
So you do have to do a lot of skilling.
And there's a lot of fear of AI, at that moment, there was a lot of fear of AI too.
So you have to do a lot of education and then you have to do a lot of things like look, we don't train on none of this code that belongs to our users, right?
So you have to also make sure that you have all these controls that you can show them, your Enterprise customers, that their data is safe, that their data is not trained on and all those things.
So we created a trust center.
We did tours with them to show them how even the flow and the diagram works.
How do, how does LLMs work?
That was another thing that we needed to get a lot of education out there.
But it just took off and it was constant going to customers, doing skilling and doing enablement, supporting our revenue teams, supporting our go -to -market teams, supporting our customer success teams. And it's pretty fun when you're working on something that is growing exponentially, that you get PM product market fit within a period of months.
and then you continuously work on the quality of it and you're seeing it improve and you having people program like my nine year old and my seven year old cannot program mainly through natural language.
And that was possible because of the beginning of the AdWords Copilot.
That's so cool. And it's so fun to watch people have that be more accessible to them.
Yeah 100%. We sometimes talk about why is it that professional developers are the only developers in my opinion.
And it's like saying, hey, like only Michelin star chefs can be cold chefs, or only the people that go toward the France can be cold that they know how to bike.
No, like we need to help us this dream of creating 1 billion developers.
That's 10 % of the world population by, I don't know, 2050 or so.
Imagine if we have 1 billion people that through natural language can create software to better the world.
That would be special.
And we should call those people developers too.
Why is it that it's only professional developers?
They're creating something, and software is at the core of that too.
They are doing it through natural language, it's slightly different.
But I think it could be called developers too.
So I think that's one of the key things.
I think we still have to go and achieve, I give up, is to realize that vision of making software to be accessible to a billion people.
I love that idea. of I think it also touches on some of the fear behind AI, right, that it's just going to replace all the developers.
And before we started recording, I told you that I've heard a lot of investors out there and a lot of people say, hey, how many developers do we think we can have replaced if we just bring in AI?
How much more efficiency should we get out of here?
And that's been the talk out there a lot.
How do you see AI either replacing or enhancing developers?
How do you think it's gonna impact the role?
We view it, there's Co -pilot, we view the human at the center and we view it as augmenting the developer, augmenting this product teams or this engineering product and design teams as well.
And the reason why I say that, and you're always going to have either investors or places in the industry where they're trying to find efficiencies or all.
But the way that I think about it is, Every time I go to one of my customers, their backlog exceeds their operational budget more, like their OpEx budget overall, meaning they want to achieve more things that they have people for.
That has been, I think, a constant through my career in the enterprise space.
I never go to a customer and they're like, look, we did everything.
We're giving everyone a vacation.
Like they don't have any other good ideas that we need to implement this quota.
In fact, it's always like we're behind on this transformation, we have these other 50 things that this specific customer wants, and we'd only do one this quarter.
Let's apply a Monte Carlo sequence on how our plan is going to end up, and all of those stuff.
So for me, it's that, shouldn't we be using technology more to create more output overall and achieve more outcomes overall?
I think Satya talks sometimes about the beauty of AI, if we realize the vision is, we're gonna be able to increase the world's GDP by 10%.
Wouldn't that be great?
We've been stuck with very low percentage increase, maybe 1 % or 2 % in developer nations, but imagine if the whole world can increase GDP by 10 % and what that means to society on it.
So we think like the way to do that, if we believe in the vision that, hey, everything's gonna be powered by software.
Let's get to our backlogs a little bit faster.
Let's make sure that our technical debt gets reduced, let's make sure we're producing higher quality software.
Let's make sure we're doing that with less tickets of bugs and defects that we have to deal in production, let's get there.
And then from there, we take a look at places that have efficiency gains or non -efficiency gains.
But I think for me is that, so where I spend a lot of my time in conversations with our customers, it's mainly on that is, how can you get through your backlog faster, achieve then a set of outcomes because of that and create this AI leverage and then hopefully have your company grow because of that.
Yeah, I think that's a great way to think about it too.
And I just hear so much fear about this is all gonna go away.
And then there's the product manager one too, where people are saying like, do we even need product managers anymore?
Is product management dead?
How have you seen AI shaping the way that we think about product management and the work that we do as product managers?
Yeah, 100%. I need product for an organization, so it's in design.
And we had these conversations.
Like, what is the future of product management, even inside of GitHub?
If I reflect a little bit on it, product is like a new discipline out there.
If you think about it, like, humanity knows how to do agriculture very well.
But creating software, we haven't been doing that for hundreds and hundreds of years, right?
I guess, it's been something in the last year or 70 or so.
And then just recently, this kind of role of product management really gave birth and started gaining momentum.
I don't even know if we as the industry really understand the role of product management across all of the organizations.
Some people really look at that role as program management.
Some people look at that role as project management.
And for me, it's not about those things.
Although there's a fair amount that you have to do in product to make sure that we have the right programs. There's a lot fair amount that you have to do to make sure that projects end up being delivered.
But for me, like the craft in product is about achieving those outcomes.
And I think I was telling you, if our role was just to create great PRDs, Oh my God, life would be so much easier and again, I would probably retire and be on a beach. Like I could do those things right now very well at times.
A new model came out 4 .1.
We just released the GPT 4 .1.
And actually, as we speak, just, I don't know, three hours ago we released also 0 .3 and 0 .4 minutes.
So these models are getting really good at creating PRDs in my opinion and taking something that I have in my head and I could rubber talk with it and be like, Is this a right strategy?
What are your arguments against this?
How should I make it better?
But I feel like that's just like one small part of what product really is.
For me, product is, look, there's these customers.
They want to hire your product for something.
And people talk about JTBDs or jobs to be done.
There's many ways that, I think, Ryan Singer uses, ShapeUp as both doing some of these as well.
So there's many ways that you would think about what is the problem that you're solving.
But there's a problem that your product is gonna be higher on.
And there's a set of business outcomes that you wanna also achieve when you create that product.
And marrying that, that's where the beauty comes.
And a lot of doing great product management is the learning loop on figuring out how you get the problem a user has and your problem to solve that, and your product to solve that problem.
And that is incredibly hard at times to do well.
It's like investing, like Warren Buffett and Charlie Munger are great investors because they make great investments.
Like it's that plan.
So great product managers create products that achieve the outcomes of the business and get hired by their customers to achieve the job the customer hired that product to do.
And if you marry those things, like, then you have something special.
So for us at GitHub, going back into your question, it's how do we not create PRDs?
Yes, we have to do that.
Not create amazing change.
Yes, we have to do that.
Not hey, let's make sure that we're paying attention to all our customer feedback.
Yes, we have to do that.
And AI admins us to do more of that better.
So instead of taking 24 hours, maybe I'll take two hours right now.
But the rest of the time that we're saving, right, it's all spent on creating a great product by figuring out what is the outcome that we wanna achieve, and then how do we measure that, and how to make sure that our customers find that, and then that we're gonna end up marrying that with the business, meaning, how do you price it, how do you package it, how do you make it available, all of those hard things.
Well, I think it's neat too, about your story, and especially with Co -Pilot here and AI, is that you basically reimagine the developer experience around this, right?
GitHub, did that from the beginning, but now you're doing it again with AI.
And when I see companies that are hitting it out of the park with AI, it's because it's so ingrained into their product strategy, but it flips what we're used to on its head.
And it says just because we've done it this way forever doesn't mean we have to keep doing it this way.
And now that we have a technology that allows us to recreate the experience or make the experience better, we should be harnessing that at its core.
And what I think is so cool about Co -Pilot and some of the other AI stuff that we're seeing out there is that if you think about it for developers too, so many people have tried to optimize, and I know developers hate this, but how fast their fingers go on keyboard for velocity.
But you don't hire a developer for that.
You hire them so that they can think strategically.
And this, to me, it frees them up to be able to do that.
It frees them up to focus on the harder problems so that they are not just typing, typing, typing away.
And that's where I think there's so much potential in this because now we're going to get so much more brain power around solving problems, rather than just trying to figure out how to write perfect code.
You got it, 100%. I think the people are being successful today with AI is because they use the AI to free up time to work on the creativity items, because those problems are incredibly hard. And let me tell you, no AI today can solve that.
The world today as it exists, and we're having this conversation, didn't exist 500 ,000 years ago.
And there was no blueprint to create it either.
Like no one out there went and said, oh, this is the way that we actually create podcasts in the world, and this is when this is going to exist. That didn't happen, but it all got created through us and humanity, right?
That's what we have. So you're right.
The people who are using AI today, very successful, they understand what is the AI leverage that they want to get.
And at the end, the AI leverage that you end up getting is the creativity, right?
Like, for example, I have a roadmap and am I a hundred percent certain of that roadmap 12 months from now?
No, right now the planning is amazing.
That's why we create roadmaps because it allows us to actually go through a planning phase and all of those types of things.
But to create a product, you have to have a lot of iteration on it.
And that necessitates a lot of creativity and a lot of time spent on reflection and understanding of what exactly is gonna make it great and a lot of intuition, et cetera, et cetera.
So like for me, product is all about that.
And the people are doing great today in using AI, are using it to solve problems from a creative side of the human by removing all the toil that human usually had to do.
So instead of spending 40 hours in total, that human spends 10 or five.
And the rest of the time is really on that creative genius.
Yeah, and it's so powerful to think about it.
But I also see people struggling right now, especially product managers or leaders at organizations as well is there is so much changing, there's so much AI coming out there to adopt.
How do you know where to start, right?
Or where to encourage your people to start.
And I think this comes into two camps, right?
Some people are thinking about just productivity on the back end.
What can you use AI to do your job better?
But then there's also the ones who are thinking about how do I bake AI into my strategy?
So when you're thinking about leading the organization and encourage your team to adopt new AI, try to understand it, try to figure out how to bake it in.
How are you approaching those two things?
How are you looking at it for productivity, but also how are you looking at it to help your strategy of making GitHub better?
Yes. We just finished a set of AI days in one of the teams. It's called the Platform and Enterprise Team where they just took our entire day to learn more and more.
Even us, like, we're so busy that sometimes we miss announcements out there And we have to take a series of days just to learn what is in the industry, and how can we use that appropriately as well?
So we just finished, I think it was day two yesterday, they might do one more day overall, where they go and use a lot of products, including compiler by the way, and they try to go and use it in novel ways and see what they could learn or they use it for things like, Oh my God, you know what?
I'm spending a lot of time creating XYZ status report, and then I'm just going to try to automate this with that or get a new perspective on how we could do that better.
So what I would say is to anyone studying in this space, you have to use it.
It's like learning how to bike, you have to bike.
There's no other way for you.
You can watch 20 ,000 YouTube channels, you're not going to learn how to do it.
So you have to do it.
So I would start with AI days where now the tricky thing is, okay, do you have procurement, allow you to play with those things?
So you have to do a lot of things before that.
Even us, we We need to go and make sure that the tools are approved.
There's a lot of privacy implications with many of these things.
So we go in, make sure that we have these tools available and then we just use them.
And then we share in a channel at the end of the day, everything we learned and then people can be like, oh, I like that.
I'm gonna start using that as part of my everyday or my weekly overall.
So I would say AI days inside your company is something like really powerful.
The other thing that I would say just on a personal level, So I use it constantly to learn new things and to really go back and forth.
The way that I learn is mainly visual and to go back and forth from some concepts.
And I don't like to click 20 ,000 links to do that.
And I would say LLMs and many, chat, you'll be t, Microsoft compiler, even ours as well allow you to do that.
So if it's anything programming related I go to our GitHub compiler and then I just try to learn something related either to Rust as an example, like I'm not very familiar with Rust and there might be something that I want to understand a lot better because I'm coming to a team.
I'm coming to a conversation with my Code Search team and all of Code Search is written in Rust and I may want to understand a little bit more how we do an indexing as an example.
So I used it to augment myself very quickly on it.
But I would say AI days, use it every day, mainly just to learn at the very beginning then to reduce toil that you have over all.
and they don't have to bake it into the strategy of your company.
You have to, again, don't think about it as efficiency and think about it as a multiplier.
And once you make that transition, by the way, almost all the apps from now on are going to have intelligence baked into it.
The key is how you're going to bake it.
If you think about it as AI's omnipresence, that's, in my opinion, AI native.
when you're trying to just bolt it as a UX panel or something like that, all you're doing for the most part, in my opinion, it's infusing AI into your application.
So that people are very successful with AI at the product, not in the organization, but at the product level on AI, is they think about it as it's just there.
The product is the feature, AI is not, AI is just omnipresence across everything the user is doing.
and those products are special in my opinion.
Yeah, and I definitely think copilot is among them.
Thank you so much for being with us on the Product Thank You podcast. If people want to learn more about you and GitHub, where can they go?
Yes, so I'm on X and I'm on LinkedIn, Varyerad1, so you can ping me there at any point in time.
And about GitHub, just github .com.
Great, and as individuals, you could sign up and try it as well.
Yes, it is free actually.
So if you get a GitHub account, you're going to start using it right away.
You get 2000 completions and 50 shots.
And we have a pro plan and a pro plus plan as well.
Amazing. Thank you so much again for being on the product thinking podcast, and thank you to our listeners out there.
We will put all of those links as well on our show notes at productthinkingpodcast .com.
We'll be back next week with another amazing guest. And in the meantime, if you have any questions for me, go to dearmelissa .com, and let me know what they are.
We'll see you next week.
