Welcome to the NVIDIA AI Podcast, where we explore the cutting edge of artificial intelligence and its potential to transform our world.
In this episode, we are joined by the co-authors of a recent essay entitled Generative AI, A Creative New World.
These experts will discuss their thoughts on the exciting potential of generative AI and how it can enable new forms of creativity and expression.
They will also delve into the challenges and ethical considerations of this technology and offer their insights into the future of generative AI.
Join us as we explore this fascinating topic with two leading voices in the field.
Usually I start our podcasts off by introducing myself.
I'm your host, Noah Kravitz. and then reading a short introduction written by me, a human being.
But that first bit you just heard was actually written by ChatGPT. an AI model that's also one of the co-authors of the essay at the heart of today's conversation.
What can I tell you? I couldn't resist the chance to get as meta as possible, even the topic at hand.
That being said, joining us today are Sonya Huang and Pat Grady, the two human co-authors of Generative AI, A Creative New World.
They're both partners at Sequoia Capital, where they focus on growth stage companies.
And I am super excited to dig into their work.
So, Pat and Sonia, thanks so much for taking the time to come on the NVIDIA AI podcast and welcome.
Thank you for having us. So we're taping this in early December.
And like most of the Internet, I have been obsessed with Chad GPT since his public preview went live about a week ago.
There are things about it that to me, I think may signal something of a watershed moment for generative AI, but there's a lot more to the space than just this one example.
So can I ask you, Pat or Sonia, to start us off by telling the listeners about your essay and maybe beginning with an explanation of what generative AI is.
Of course, happy to do it. So thank you for asking.
We put out the landscape because we think that we're in the moment, in the middle of a pretty profound moment for technology.
And we view it as really the dawn of a new class of applications.
And we call this class of applications generative AI.
And what do we mean by that? Why did we choose the term in the first place?
If you look at historically the role that AI played in software, It was really good at analytical tasks.
So sifting through a mountain of historical data, finding patterns in it for a multitude of use cases.
You can detect I think that was one of the biggest first use cases you can predict when your Uber is going to arrive.
These are all kind of these analytical tasks that are used to optimize an existing thing.
And I think the technological kind of category is like these are all encoder models.
The thing that's different now is decoder models, which is like you're not taking text as an input and converting it to machine representation.
You're taking text as an input And actually producing an output that's consumable by a human being and that would have actually been created by a human being.
And so the difference as I see it is like machines are not only good now at encoding, so they're not only good at finding patterns and doing machine-like things, they're starting to get really good at doing human-like things.
And what are humans good at? We're good at knowledge work.
We're good at creating. So we can... We can write poetry, we can write code, we can design video games, we can do a lot of these creative and knowledge tasks.
And for a long time, machines had no way of competing with us in doing that really well.
And I think that's what's so different about this class of models, which is that they're these decoder models. that are large enough to produce coherent, interesting, and sometimes superhuman results.
And I think everyone has been just incredibly impressed by what OpenAI has been putting out with GPT-3 at first and now chat GPT, I think it doesn't stop there.
OpenAI is not just going to be an end application for people to chat with.
It is going to be the platform that enables a whole host of applications that are being built on top of it.
And that was really the critical insight that we tried to draw in our landscape.
So in your essay, essay blog post, I referred to it in different ways, so forgive me.
There are a couple of things that it's a great overview of, you know, it's a relatively short history, but the history of space and you're kind of broken it into waves of generative AI, say technological advancements and then applications built on these things.
And then obviously in the work that you both do in investing in and managing portfolios of companies who are doing kind of early stage, I think mostly tech work.
You've seen a lot, you've seen a lot of technology, you've seen a lot of applications built on tech and companies. grown out of that tech.
One of the things, well, there are a couple of things, I guess, but one of the questions that maybe it can be kind of an avenue to get into this is that You use GPT-3 to help do some of the writing, actually some of the words and then some of the ideas. that you sort of took and reshaped, added a human touch to.
Because as your essay shows, you know, these applications, some of them are closer to kind of prime time being ready to spit out final drafts of creative products and some aren't quite there yet.
And then you also use some image. I think you use mid-journey for the images.
And over the past year or two years, we've seen... sort of in the popular consciousness, I guess, a lot of interest in these image applications, Dali, Crayon, Eterni, I think Lenzo, as we're talking now, is kind of the the hot one du jour, it's hard for me to keep up with my kids know more about this than I do.
And then And on the text side, I do a lot of work writing.
And so I've been kind of following even in something like Google Docs. auto-complete has kind of turned into auto-suggest and it's getting easier to see, oh, these words come up that will finish my sentence for me. wow, they're getting more and more accurate.
As you put this piece together, and then again, maybe even digging into that kind of more broadly to your work, What things have stuck out to you most as being kind of like the really big aha moments or things that have surprised you? just about the evolution of the space and kind of, you know, the moment we're at now at where you see things headed.
Maybe we could zoom out before we zoom in and try to put it in a bit more historical context before going to some of those specific trigger points that we've experienced over the last handful of months or years.
We'll go very broad. Our objective at Sequoia is to help the daring about legendary companies, and legendary is often defined over a period of decades. tend to take a pretty long-term perspective on these things.
If we do a kind of quick rehash of the history of technology, 1960s, we got semiconductors, Seventies, you started to build systems on top of those.
Eighties, we connected them up with networks and had the advent of packaged software.
90s the pc comes to the fore 2000s we've got the internet doing useful things 2010, the Internet is now delivered over mobile devices and the Cloud serves as the back end.
We've had these different waves of technology that led us to this present moment.
And there are two things that are very interesting about them.
One, each success at Wave was an order of magnitude increase in the accessibility of compute for people.
There are now 8 billion people on the planet.
More than 7 billion of them are on the internet.
And so that's done. The way that the ball will be moved forward for the technology industry as a whole is not by providing access to more and more people, Everybody has access.
So that's one important takeaway. The other one is, each of those success at-wage was an order of magnitude or more increase in the amount of data that was being produced.
And so we're now sitting at a moment in history where everybody in the world almost has access to technology of some kind, and there is more data available than ever before.
So what is the way for technology to move the ball forward now?
What's to make better use of that data? and so the the broader theme here is not generating ai specifically or even ai more broadly The broader thing here is data, and how do we use data to do more interesting stuff?
How do we go from dumb software that's just capturing information to smart software that's producing information?
That's the thing that this really sits at the heart of.
And some of the inflection points would be 2012 when deep learning became a thing, 2017 when the first transformer-based models came out, thanks to, you know, attention is all you need.
Earlier this year, the stable diffusion moment was a huge moment. possibly akin to the Netscape moment in the generation of the internets.
Not necessarily because it was a technical achievement.
There was a technical achievement involved.
But because it was a watershed moment in accessibility, it showed everybody that these models were available for you to grab and run on your own machine. which hadn't been true previously.
It's similar to what Hunting Face did years ago when they released one of the first bird models.
And so this idea of these models becoming accessible to the masses which is sort of what the stable diffusion moment provided, sort of what the Chad GPT moment has provided, sort of what Hummingface has provided.
That draws people in. That draws in the creators.
That draws in the hobbyists. That draws in the people that may not understand the technology itself to the greatest degree.
But they understand different applications of it.
They understand different possible use cases.
And that's what leads to the Cambrian explosion of applications that's going to make this really interesting.
We are super early in that transition at this moment.
It is still mostly the hobbyists and the creators and and folks who have not necessarily turned things into full-fledged end-to-end businesses Just yet, but the creativity we've seen out of these developer types has been pretty phenomenal.
I appreciate your zooming out. I have a tendency to get very excited about the moment zoom way in.
And to what you said, you know, for me, the chat GPT, and not to say that this will be the one looking back necessarily, but that watershed moment was a moment for me of of, oh, I can show this to my mom.
I could show this to my son. I could talk about this with my software engineer buddies. and everyone that access is kind of universal and the things that can be outputted from it.
Let me dig in actually to what you said about the stable diffusion moment.
Maybe can you speak a little bit about how we got there and what the open sourcing of AI and machine learning models and the availability of these larger datasets to folks, both in terms of Getting to that moment and then also, you know, where we've come, where we're going in the short time since then.
I can take that. So I would say, and then Pat referenced the Google's attention is all you need paper.
I think that was really the catalyst to a lot of this. because it set off the race to scale, right?
And so just, I mean, quick recap of the paper, it introduced the transformer architecture And with these large models, you actually can demonstrate that as they scale, they get more and more performance.
And so it set off the race to scale in the language category.
And for, I would say for wave one, it was the large research labs that were really doing a lot of really interesting stuff. but keeping it very close to the best, right?
So for example, Google would expose some of these capabilities and actually use it to power, you know, for example, translation.
But it wasn't accessible to anybody. It was all very much hidden inside these large research orcs.
I think Wave 2, OpenAI deserves a lot of credit for actually opening up access to these large language models as an API.
And so if you think of intelligence as an API, I think OpenAI was really the first one to make that accessible because until then it had only been like very narrow things that were accessible.
And so I think they deserve a ton of credit for making it possible for anyone to access a cognition engine over API.
The stable diffusion moment is intriguing to me because it was the first moment where At first, I think everyone thought you had to have billions of dollars to build these large models.
And stable diffusion was fascinating because, and Pat mentioned, it was on the back of a technological breakthrough as well.
Right. Like with diffusion models, you can actually you can actually make this stuff happen with an order of magnitude less compute than needed before.
And so as a result of that breakthrough, it was possible for a lot of these models to be kind of trained and released. in the open source, which just wasn't possible pre-diffusion.
And so the thing that stable diffusion did was it just made these models Completely open source.
The weights are on Hugging Face. You can just download them.
And that was, you know, the next moment of usability.
And then I'd say the final wave that we're going through right now is I actually, I compare it to the iPhone where you're starting to see If you remember back to when the App Store first came out, it was some pretty gimmicky stuff.
Yes. And but like that was also the moment that sowed the seeds for, you know, companies like Uber, for example, that relied on, you know, having connectivity everywhere and maps like Right now, a lot of what we're seeing in the generative AI stuff, I would compare to the early days of the App Store where you've got a very gimmicky, maybe a thin layer, on top of what these models are capable of doing.
So for example, there is a lot of people that are just doing text to image on top of these existing foundation models.
I think where we get really excited for this wave of applications is what happens when you just completely rethink when applications are possible. now that you have intelligence as an API, now that you have generative AI.
And I think the shape of those applications is going to look very different from a lot of what's taking off in the App Store already.
The example I'll point to is actually a company in their own portfolio, Notion. that just launched Notion AI.
Like, you know, if document editing historically was all about kind of manual thinking, writing, writer's block, etc.
When you kind of When you flip that on its head and take an AI-native approach, how does the actual act of writing change?
How does the act of organizing your thoughts change?
And so for us, we're just we're way more excited for the people who are not just kind of throwing a thin skin on top of foundational models, but like using them to rethink you know, delivery of their core products and services.
And so that's what I would call the sort of iPhone app store moment that we are just at the brink of right now.
If I could double-click on that and add to that real quick, please.
One of the examples I like to use is the Flashlight.
So in 2007, iPhone came out in 2007, App Store came out in 2008.
People were using their phone as a flashlight before the phone actually had a flashlight.
So there were apps that you could buy where you pay five bucks and all the app does is it turns your whole screen white.
And you can use it as a flashlight. There was no flashlight.
Well, of course, that doesn't make sense to be a third-party app.
Of course, that makes sense to be a utility that's part of a platform. and eventually it did become that.
So that's both an example of maybe a dividing line between what will eventually become first-party apps that are just part of the foundation models or the platforms offering the foundation models versus third-party apps.
An example I like to use on the third-party apps is DoorDash.
The enabler for DoorDash is the smartphone.
But even though the App Store came out in 2008, DoorDash didn't get founded until 2013.
So it was a full five years later. We would suspect that something similar will happen with jittered AI where there's a lot of excitement today, a lot of the apps are still pretty rough, And it might be five or even 10 years before people really figure out what the killer applications in this technology are, because it's such a mind breaking paradigm shift. it's hard to just immediately or intuitively conceive of what those could be.
So when I hear you talk about it, one of the things that pops to mind is...
I'm more of a think in words as opposed to visuals kind of person.
And so for me, again, and there's recency bias, I know.
But the ChatGPT moment for me was a little bit bigger in my own head than visual tools that preceded it.
And part of that is because kind of quickly it was like, oh, I can prompt this to just answer a question, have a conversation.
It's optimized for dialogue. I can prompt it to, you know, write a podcast introduction or, you know, my 13 year old immediately, you know, he first said and be like, hey, I heard about this thing that can write essays or homework assignments.
And then I showed it to him and he started playing around and he had it generating songs with chord progressions and lyrics and getting deep into, you know, write a song about Gus Fring from Breaking Bad in a minor blues scale or whatever.
And we had a generating code and ASCII art and all these different things. paradigm that you mentioned of, you know, it's not just about how does it change the role of being a creative writer, but What's the thing that's, you know, maybe five years hence that kind of breaks the mold or pokes something open to a new way of using intelligence as an API?
I mean, that's fascinating. Are there, not to put you on the spot to predict the future, Are there any trends, any examples that kind of point towards... you know, a sort of paradigm shift you see coming in the way that people approach Writing or, you know, getting getting food or whatever it might be.
I would say in the majority of board meetings that I'm attending right now, What are we doing with generative AI?
What are we doing with AI is one of the key topics that's getting discussed.
And, you know, it is no longer a research pet project.
It is no longer a, you know, oh, you VCs just want this to happen.
I think... every company that is on top of its game right now realizes there is something incredibly important happening and they're actually developing, they're actually dedicating real products and engineering resources towards making this happen.
Like to go back to the Notion example, they turned around that AI experience very quickly because they just realized how essential it was to the future of Notion.
If you look at what Neva just launched with large language model search, That was, again, turned around very quickly.
And so to me, the thing that... The thing that gives me confidence that we're going to see a lot of these more interesting applications come out in the coming years is like the input is, you know, our companies tend to be you know, our company sends me thinking several years out and a lot of these products won't really reach full maturity for several years.
But the fact that they're all thinking about it right now and dedicating resources to me is the best leading indicator.
Do you see a particular advantage or disadvantage for startups and sort of smaller nimble companies as opposed to some of the larger incumbents who have been working even throughout all the waves you mentioned, or most of them.
But by virtue of being larger established incumbents, you know, it takes a little longer to steer a ship that big and in your direction.
Yeah, I mean, the war in technology is always a war between the little companies who have good products and the big companies who have good distribution.
Can little companies get distribution for their products before big companies copy their products and shove it into their distribution channel, right?
That's always sort of a battle. The case that says the big companies have an advantage is one that, in our opinion, overvalues or overestimates the benefit of the data that they have.
That data in a lot of cases is not nearly as accessible as people suspect it to be and probably doesn't provide as much of an advantage as people suspected would.
And so we think there is some data advantage to being a big company, but a far greater advantage is that of agility. where the small companies win by a lot.
And so you mentioned the Niva and Notion case studies.
What's common about these two case studies, number one, it was driven by the founder personally.
It was Ivan in the case of Notion, and it was Chudar in the case of Niebuhr.
Number two, they did it extremely quickly in a matter of days. because they personally got excited about it, had the expertise to go dig in and came back with a working prototype in literally days.
And of course, then there was a lot of polishing and maturing and stuff like that that happened before it got into in a production, but that's not the sort of thing public companies tend to be capable of.
As a point of comparison, we talked with a well-known public company and the person who runs product there.
Their response when asked what they were doing with JRDevAI was, We're all over it.
We're going to try to fund a team of 100 to work on this next year.
That year, it'll be relevant by next year.
You need to fund a team of one to work on it right now.
The cycle time and the way they approach things is very, very different.
And we think that sort of bodes in favor of the startups.
So that makes me think of there's a quote in the piece on sequoiacap.com.
The best generative AI companies can generate a sustainable competitive advantage by executing relentlessly on the flywheel. between user engagement data and model performance ask you to kind of unpack that a little bit And I don't know if maybe my brain just kind of made a leap to that because it was in my head to ask you about it.
But it makes me think about a range of things.
But this idea of data and what data sets are and how much data humans are generating, I think we're outpacing ourselves day by day with the amount of data being creating it and a lot of it pushed out onto the internet where conceivably it could be artist in some way to be fed into a model to then be used to generate more data, if you will.
Can you talk about that idea of executing relentlessly on the flywheel between user engagement data and model performance? and how that kind of plays into where these companies are, what they might be thinking about, or where you see things headed in the near term.
Absolutely, happy to take that. I think of autonomous vehicles as a comparison point where Getting the basics right, getting 60% of the way there is pretty easy.
Getting the next 10% is a little harder.
Getting the next 10% is a little harder and like, you know, once you're on the last 1% of use cases, it is like excruciatingly hard and you need You need data on all the coroner cases.
And that's what's so expensive about making and so impossible about making AVs work.
I think that really applies with these generative AI applications as well, because, you know, Just from the get-go, just from a really good foundational model, you can get, depending on the domain, you may be 40% of the way there already.
But if you're really able to somehow capture, somehow to build this really nice mousetrap of users coming to your product, and you're capturing all that data in the interaction, right?
Maybe you're generating five different possible outputs and people are choosing which one to use. maybe you're generating output and people are upvoting or downvoting.
If you actually own the entire interface, you actually even have like mouse click data.
There's a ton of data actually in learning how humans are interacting with these model outputs.
And if you feed that back into the model, I would say that is the mechanism and that's the feedback loop that gets you from the 40% to the 60% and eventually to the 99, 100%.
And so, I mean, this concept that's taking fire within the industry of kind of RLHF, reinforcement learning with human feedback, I think that is very much the sustainable moat for a lot of companies in the generative AI space.
I would say one of the first things we ask the company that's building the generative AI space is like, can you show us what is your North Star metric? on performance and can you show us that performance is getting better over time?
Can you show us that engagement, cohorts, retention, et cetera, is getting better over time.
Because to us, that is definitive proof that these companies are actually building that flywheel in the right way as opposed to a lot of companies right now are just slapping a ui on top of a model and not actually learning from the data and to us there's less sustainable of a moats there.
We talk a lot on the podcast and four or five years that I've been lucky enough to host it.
Talk to a lot of folks who have brought up how this, it's a little less recent or maybe two, three years ago, but this kind of explosion in compute availability and the price performance and you know, home hobbyists with GPUs they may have bought to play games. were able to kind of leverage that to do AI-related tasks, and that kind of really democratizing the accessibility to train models and do things with machine learning and AI.
Is compute a factor at the moment in generative AI and in what some of these businesses are or will be able to do or might do? to, you know, raise funding and gather the resource to do, or is it kind of still the case that the compute advance, we have this kind of explosion And now it's the software and tools and the human beings and the creativity and the ideas that are kind of playing catch up to harness. everything that compute can already do.
Is that kind of still the case for generative AI, or is there actually... you know, sort of a next level of compute power that's really going to maybe spring all of this to a new level or introduce a new wave.
I would say we are operating on the brink of available computes.
And so, for example, if you look at some of the things that Midjourney has said publicly, Like there just aren't enough GPUs in the world for them to power all the user growth they want to have.
And so we are operating on the brink of available compute.
There are some really interesting studies that, you know, we are like AI and what's happening with like AI Moore's law is outpacing what you would have predicted with Moore's law proper.
And that's coming off the back of a lot of very smart people who are realizing that, you know, compute still is a fundamental constraint, right?
If you believe that large models are going to get us to the promised land, then compute is the constraint and they are pouring a ton of talent into algorithmic speedups, optimizing how you're making use of that compute.
So I would say, We're absolutely operating at the brink of what compute can give us today.
And as the compute gets better and as we get better at making more efficient use of it, That's almost kind of what creates this AI Moore's law that opens up, you know, models that improve more than an order of magnitude each year.
Just thought that. I think to Sonny's part of the AIs, they had Moore's Law.
I think parameter size has doubled 13 times in the last five years i think it's something to that order of magnitude which obviously can keep going.
So the answer is probably both. To your question, both compute that is governing progress and the other side of the equation, people connecting what exists today with compelling end-user applications.
We see a lot of companies who start with one of the foundation models, like an open AI model, and they realize they can get better cost trade-offs If they scrap together open source models, they can point on Hugging Face as one example.
And so as companies start to scale things that they're building with the foundation model, sometimes they realize that There are cheaper alternatives that are good enough and actually deliver the right performance trade-offs for whatever their application is.
So there are ways to back away from that frontier of compute, so to speak, depending on what the app is.
It does feel also at the moment like... Most of the applications are not applications, they're utilities.
To have an application, you really need to complete an end-to-end workflow.
And typically, these are just getting inserted in the middle of a workflow versus completing workflow.
Even with the current technology, if it were to progress no further, there's a huge range of compelling applications you can build with no further technological progress.
So it's a little bit of both. We both need more compute for more technological progress and we need better applications for what we have today.
Makes sense. I'm speaking today with Sonia Huang and Pat Grady.
Sonia and Pat are partners at Sequoia Capital and The co-authors or co-authors along with an AI model, a blog post called Generative AI, A Creative New World.
I encourage you to go check that out. on the sequoiacap.com website.
It's a great introduction to the whole idea of generative AI.
If you're not familiar or if you are familiar, it gives a really good overview both in words and images that I also created hand-in-hand, human and machine, human and AI models. to give an overview of these four waves of generative AI we've been talking about. where things stand now, where they might be headed across different areas, different industries, I should say, different areas of output.
I want to pivot a little bit before we wrap up and ask about issues related to trust.
We'll use trust as kind of a blanket for a few things here.
When I first started playing around a few days ago with ChatGPT, one of the first things I did was to ask it to write a blog post that I was working on for a different project.
And it spat something out that, you know, was pretty good and pretty reasonable.
And my first thought was, well, I wonder if I could pass this off as my work.
And then my second thought had to do with that phrase, passing this off with my work.
Not long after that, I read a post on a blog called Stratechery written by... tech business analyst, Ben Thompson.
And I believe it was called AI homework.
And this dove into the first thing that I think I mentioned one of my own kids brought up, which was, oh, I heard about this new thing that can write essays for you.
And the post on Stratechery goes into him working with his own child Who needed to write an essay for school and asking Chad GPT to write an essay.
And it spat out a pretty well-written passable essay that actually was factually incorrect.
And it dove into this idea of sort of the black box of AI.
And so for me, one of the things was, wow, like this tool is so slick and it kind of functions like Siri sort of mashed up with Google search, except it can also like output and all these other ways that I ask it to, but there's no sourcing.
It's not telling me where it's grabbing the information from.
So I can't click a link and see that, oh, you know, This thing didn't come from a source that I trust.
It came from a blog that I've never heard of.
And then I dig into the blog and I see that these other posts are conspiracy theories or just factually incorrect.
And so I notice or dismiss that. And then I did a little poking around, looking into Lenza as we record.
This is kind of popular this week. And I saw some threads on Twitter from artists that were actually asking folks, hey, I know it's really fun and cool, but please stop using these different AI image generation tools. because they're stealing from human artists like me.
And so I want to ask the two of you, How do we, how do you as VCs working with companies, building businesses, how do we as a society I do I as somebody who thinks this stuff is really awesome, but also wants to make sure that my kids and the generations to come are being ethical.
How do we sort of deal with this idea of AI is a black box.
We can't see all of the whinnies that it gets to its final output. applied to something like, if I ask who was George Washington, it will give me an answer that sounds pretty authoritative, but I don't know where it's grabbing its sources from.
Noah, you bring up some really important points there, and there's actually a few that are tangled together.
So I would say the first one is hallucinations, right?
All those can make things up. The second is cheating, right?
How do we know if something is human generated or machine generated?
And the third point that you bring up is just fairness and copyright.
And so we've thought a lot about each of those things.
And I think, you know, these models are an incredible provocation for, you know, what are we going to do to combat these thorny issues?
Because I agree that we need answers to them.
And I think it's going to be an evolving thing as the space moves.
On the topic of hallucinations, if you compare what chat GPT does to GPT-3, I would say at least qualitatively in terms of experience, I've experienced that chat GPT hallucinates a lot less.
Like it almost knows when you're trying to trick it.
And the thing is actually built into the model, if you kind of ask it to re-examine what it just said, It has enough of an appearance of reasoning that it can end up admitting, oh, I was wrong before.
And so one thing I think that's going to happen is that as you kind of use these models almost recursively on themselves, and kind of encourage them to divulge a true level of uncertainty, confidence, how close are they to the ground truth.
I think that the models will get better and better.
You're starting to see some of that happen in chat, GPT, but they have to get better at this concept of hallucinating.
And the second thing related to hallucinations is I think you are gonna see a lot more transparency and auditability over results.
If you look at what Neva announced in terms of, you know, LLM powered surge.
Real quick, what does Neva do? Neva is a search engine that we backed.
And they've been doing a lot of efforts around large language model based search, just reimagining what can the search experience be when, you know, we're not simply returning a list of links and ads, but, you know, we want to be ad-free and really rethink how can we return results that truly fit the shape of what a user wants to query when they're searching something.
And if you look at, Neva actually just announced some of what they're doing, you actually can trace back where they're getting different snippets from in their outputs.
And so I would say that kind of transparency and auditsibility also engenders trust.
And the final thing I'll say is right now, ChatGPT is not exposed to the internet.
As these models get exposed to the internet, if you view the internet as kind of the reflection of ground truth, You can compare their outputs to the ground truth, and you can also get a truthfulness score there.
And so I'd say there's a lot of research being done on the hallucinations topic.
I agree, it's incredibly important to solve.
On the topic of, you know, cheating, a few answers here.
One is, I do think as these models get better and better at, for example, writing a base essay, I no longer think it's as critical of a skill to be able to write that three paragraph essay on Abraham Lincoln.
Like I think there's so much of skills that we have to be good at are going to be how do we prompt these machines, right?
Like Toby from Spotify calls it spell casting.
And so one is like the nature of critical thinking, knowledge tests that we do will change as we work with these machines.
And so I actually think it's a good thing that people are learning how to operate in an LLM world.
And the second is, I think researchers are kind of thinking about ways to make it If you look at OpenAI, for example, they're looking at ways to basically embed cryptographically whether an output was generated by open AI or not.
And like they were the only ones that have that key, but it's kind of embedded in the output.
Which I think is fascinating. I think eventually you will have a feed this back to the model company.
Was this generated by the machine or how much was generated by a human?
And so I think that will help solve the kind of cheating question.
And then the final question over kind of copyright and ethics.
I think this one is the thorniest, you know, and there were answers to everything from, you know, all art is derivative.
You know, if you've created something that's really on the back of somebody else's work, like, how does that work?
I'm guessing you know from a technological perspective you can see you know one how close was the prompt to really just piggybacking off somebody else's work and how close is the output to really just piggybacking off somebody's work.
So I would say from a technology perspective, it's possible to solve some of these problems.
And the bigger question is, I would say like, societal kind of cooperation, like what is fair when you have these derivative works you know, both what is legal and what is fair.
And I think that's something that, you know, the community and regulators, everybody, like all the stakeholders kind of have to get together and figure out together.
Yeah. All right. In the time that we have left, it's too big of a question.
So one thing, hopefully, but... What did we miss?
What did I not think to ask about that is kind of at the forefront when you're thinking about and you're grappling with generative AI and what it might mean going forward.
Something to bring out that the listeners should go away thinking about.
I'll offer two things and I'm sure Sonya will create.
One is the difference between computation and communication.
People who want to view this as analogies to the internet are conflating those two concepts.
The internet was a revolution in communication.
It brought people closer, it connected people, it made things more efficient.
This is a revolution in computation. So the shape of it will be fundamentally different.
It will still produce killer applications.
It will still lead to applications across business and consumer applications. but it's not going to follow the same pattern.
We don't have a crystal ball or know exactly what that means, but we do think that's an important thing to keep in mind.
The second thing, which will maybe sound a bit juvenile or silly, one framework that we find useful, which kind of comes from this question around, can people use this to cheat?
If you look at the SAT, college entrance exam here in the United States, There are three sections.
There's math, there's reading, and there's writing. computers have always been able to do that math section, or at least pieces of it, fairly well. computers can now do the reading and writing sections fairly well.
This is not a comment on cheating. This is a comment on if those are the three basic skill sets, that go into human labor, and computers can now basically cover them.
And even a couple years ago, they could barely cover one of them.
What does that imply about all the different things computers are going to be able to do?
And our suspicion is that the answer will surprise us to the upside in a dramatic fashion.
I would not bet against your suspicion on that one.
Sonia, anything you want to add in closing?
Yeah, I would just say a lot of the energy in the ecosystem so far has been around text and image models.
As we look at some of the foundational research happening in other domains, I think it's incredibly exciting to think about what's going to happen when When you're using AI to think about the structure of proteins, what's going to happen when you're using AI to actually you know, generate entire gaming storylines, assets, worlds, like on demand.
And so, I would say so far, it feels like people are thinking about text and image applications still.
And I'm personally very excited about what's going to happen to gaming, what's going to happen to biology, what's going to happen to all these different domains as these different models come online.
Because even though it feels like you know, chat GPT and stability where these watershed moments, like there's so many more of them that are in the hopper if you look at the model research pipeline.
And so I think it just means the entire application space can be way broader than anybody thinks.
Yeah, it's easy and I'm guilty of it. be the first to admit it's easy to focus on the big shiny thing right in front of me i can't stop logging into but stepping back there are wild times ahead for sure and Hopefully, fingers crossed, I think so, for the better of all of us.
Sonia and Pat, again, thank you so much for taking the time to come on. fascinating conversation, obviously just scratching the surface of things to come.
We mentioned the essay. on the Sequoia website for folks who would like to learn more whether generative AI or some of the companies you've mentioned or your own other work that you do, other thoughts you might have, where are some of the best places to go online?
AI Twitter is incredible. And so if you're following the right people on Twitter, And it's incredible, like, I mean, research is happening and being published on top of Twitter before it's even published on archive.
Like it feels like very much even the academic world and the practitioner world is all converging on Twitter right now.
And so that's the source of a lot of you know, my learning and when we talk to people in the ecosystem where others are learning as well.
All right. Good enough. Again, thank you both.
And all the best going forward. And, you know, maybe a year or two, maybe it'll be two months down the line when the next big thing comes in.
Perhaps we can check in again to get an update.
Best to both of you and thanks again for joining the podcast.
Thanks for having me. Thank you.
