And so we've been through all of these tech waves and we've learned how to have this discussion in a way that, for the United States, interest balances these two things.
And if we're gonna make a departure from a posture that was developed from 40 years, we better have a pretty damn good reason.
Today, a new frontier of scientific discovery lies before us.
You can sometimes judge a book by its cover, and I think this is a strong start.
The conversation around AI regulation in the U.S. has changed dramatically.
Just a year ago, the loudest voices were calling to pause or shut down open source AI.
Today, the U.S. is pushing to lead the global race.
So what changed?
And what does it mean for innovation, competition, and the future of open source?
I'm joined by A16Z General Partners Martin Casado and Anjane Mitha to unpack the new AI Action Plan, the politics behind it and the implications for builders and policymakers alike.
Let's get into it.
As a reminder, the content here is for informational purposes only, should not be taken as legal business tax or investment advice or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund.
Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast.
For more details, including a link to our investments, please see a16zcom forward slash disclosures.
We're talking a week or two after the action plan has been announced.
Looks like we've come a long way.
Yeah.
You guys have been on the front lines for years now in this discourse, fighting to make this possible.
Why don't we trace where we've been, so that we could then understand how we got here and where we're going?
I mean under the Biden administration, we had the executive order, which was basically the opposite of what we're seeing today.
I mean, it was trying to limit innovation.
It was doing a bunch of fear mongering.
But to me, what was even more striking was not regulators being regulators.
You'd expect that.
But if you remember, Anj and this is why we got involved is you'd have these politicians, you know, making recommendations, which is fine.
You'd expect that.
But nobody was saying anything.
You know, it was like academia was silent.
The startups were silent.
And if anything, like the technologists were kind of supporting it.
So we were in the super backwards world where it was like innovation is bad or dangerous and we regulate it.
We should pause it.
You know, there was this discourse and it was like somewhat fueled by tech as opposed, you know, and then nobody was going against it.
And so I think today we should definitely talk about the action plan is great, but we should also talk about how the entire industry has kind of come around to say like listen, we need to keep these things in check.
We need to be sensible.
Yeah.
I mean, pause AI.
That was two years ago.
Remember the big sort of, you know, all the CEOs signed this petition.
Oh, yeah.
I think that was the last AI Action Summit, right?
The one before Paris.
Guys, there's been so many of these.
Yeah, I must try.
No, no.
Remember, like, what was the...
Dan Hendrix's CAIS, what was California AI?
The Center for AI Safety.
Center for AI Safety, that's it.
The nonprofit, yeah, yeah, yeah.
Yeah, and then they got like all of these, like people, to sign this list.
You know, like when you need to worry about the existential risk of AI.
And like that was the mood.
It was almost like...
Can I just do something by contrast?
So I was there during the early days of the web and the internet.
And at that time, you actually had examples of the stuff being dangerous.
Like Robert Morris let out the Morris worm.
It took down critical infrastructure.
So we had new types of attacks.
We had viruses, we had worms, we had critical infrastructure.
We actually had a different doctrine for the nation.
We said, you know, the more we get on the internet, the more vulnerable we are.
So instead of like mutually assured destruction, we have this notion of asymmetry.
So there's all of these great examples of why should we be concerned?
And what did everybody else do?
Pedal to the metal, invest more technology.
This is great.
And so like, you know, we were still at the time, like we wanted the internet.
We wanted to be the best.
We wanted to build it out.
You know, the startups were all over it.
And coming into this AI stuff.
Two years ago it was the opposite, which is like there were the concerns with new technology which you always have.
But like there are very few voices that were like, actually it's really important we invest in this stuff.
And so that's kind of, to me, the bigger change is this more cultural change.
I think that's right.
There was a moment in I think it was last summer, where somebody sent you and me a link to the SB 1047 bill.
And I remember Marty and I reacting like...
There's no way this is going to get any steam.
What was absurd to us, I think, was that it made it through the House and the Senate and it was on its way to a final vote and would have become law.
One signature from the governor later.
And I think there was this escalation where I realized I think my view is that technologists like to technology and politicians like to policy.
And you pretend like these two things are in different worlds.
And as long as these two worlds don't collide and the engineers get to build interesting tech and there's no sort of like self-own too early.
We generally trust in our policymakers.
A lot of the policymakers who actually, I think were quite open about the fact they didn't know much about the technology because it was moving so fast still felt like something had to be done.
Therefore, this is something.
Therefore, it must be good.
And the this was this, I think, the most egregious example of this being adversarial was SB 1047.
I totally agree.
But that culture shift was one from let's let the tech mature and then decide how to regulate it later to, like before, let's try to regulate it in its infancy was like a massive, I think, shift in my head.
But let's just talk about how bad it was.
You had VCs.
Their entire job is investing in tech, talking against open source.
You know, like Vinode, Founders Fund, they're like, open source AI is dangerous.
It gives China the advantage.
And there was just some sort of prognostication that if we didn't do open AI, I think the Chinese would somehow forget math and not be able to create models.
And then you forward by a year and they've got the best models by far and we're way behind.
So it was like the people that are supposed to be protecting the US innovation, brain trust were somehow on the side of the.
Let's slow it down.
And I think that now there's this realization of I see China's really good at creating models and they've done a great job.
We've kind of hamstrung ourselves from whatever discussion we were having.
And I think you're right.
I think it's good to be concerned about the dangers and job risks, but it has to be a fulsome discussion.
You need both sides.
And when you and I jumped in, it just didn't feel fulsome at all.
It was like one side was dominant and there was almost no one on kind of the pro-tech pro-innovation, pro-open source side.
I just think it didn't feel grounded in empirics.
Well, certainly not that.
It came from, you know.
So what is the steel man of the critique of open source that they were making a couple years ago?
That, you know, this is like a nuclear weapon.
Would you open source your nuclear weapon plans?
Would you open source your F-16 plans?
So the idea was that somehow, like, this was, like, you know...
Nuclear weapons are not dual use.
Nuclear energy is dual use, right?
An F-16 is not dual use like a jet engine is dual use.
But a lot of the analogies that were used at the time were something that you know.
If you squint one way, parts of it are dual use.
They could be used for good or for bad.
But like the examples were clearly the weapons.
And that's what they would say.
They would say, listen, these things are incredibly dangerous.
Would you open source like whatever the plans for an F-16?
Yeah.
And then, you know, the other side would slowly decided like this conversation is ridiculous.
We got to go ahead and set up.
It says, you know, no, you would not do this for an F-16 because that is a fighter, you know, jet.
However, like a lot of the technologies used to build it.
Yes, this is, you know, fundamental technology.
It's not like people aren't going to figure out any ways and we need to be the leader, just like we were the leader in nuclear.
And we were –, by the way, in nuclear like, if you go historically.
When that came out, we invested incredibly heavily in it.
The things that we thought were proximal to weapons, of course, we made sensitive.
But this – all the universities were involved.
Like the entire country had the discourse and that just wasn't what was happening.
Yeah.
I think that's true.
They were basically like there was a substantive argument against open source and there was an atmospheric one.
And the substantive one was like the one Martin mentioned, that the technology was being confused for the applications right.
And all the worst case outcomes of the application or misuses were then being confused.
But they were also theoretical too.
It's even worse than that.
It was like you're right in what you're saying, but it was like This could potentially create bioweapons.
It's funny, we got a bioweapon expert, and he's like, well, not really.
I mean, the difference between a model and Google is almost nothing.
But that was used as this straw person argument, and then it could hack into a whole bunch of stuff.
Nobody had ever done it before, but it was theoretical.
So it was like these theoretical arguments that were very specific versus a broad technology.
That was one.
And then the atmospherics.
There was a famous former CEO who went up in front of Congress and literally in a testimony said The US is years ahead of China.
And so, since these are nuclear weapons and the misuses, we're being confused with the technology and we're so far ahead, let's lock it down so we can maintain that lead and therefore our adversaries will never get their hands on it.
Which we're both just fundamentally wrong for the reason Martin said like substantively, AI was not introducing new marginal risks.
So if you did an eval on how much easier... Well, at least not identified at the time.
I mean you would go to Don Song, who is like a safety researcher McCarthy Genius Fellow at Berkeley, and you'd say what are the marginal risks of AI?
She'd say, great question.
We should research that.
That should be a good research problem, yeah.
The world expert on this question was like this is a very important, but it's an open research statement.
Yeah, so no empirical evidence at the time that AI was creating net new marginal risks and just factual inaccuracies that we were ahead of China.
Because, if you just paid attention to what was happening, DeepSeq had already started to publish a fantastic set of papers, including DeepSeq Maths V2, which came out last summer.
And you're like, okay, obviously these guys are clearly close to the frontier.
They're not years behind.
And so when R1 DeepSeek R1 came out earlier this year, you know a lot of Washington was like shocked.
Oh my God, like how did these folks catch it?
They must have stolen our weights.
No, actually it's not that hard to distill on the outputs of our labs.
Have you actually looked at the author list of any paper in AI?
Like where do you think these people come from?
Right.
So I think those two things were.
It felt like we were being gaslit constantly because both the content and the atmosphere were just wrong.
Maybe one question for the smartest people or the most sober people who are against.
It is like maybe they were asking where should the burden of proof be?
Because it's hard to prove that there is risk, but it's also hard to prove that there isn't risk.
And so this question of what's risky, is it riskier to just go full steam ahead?
Or is it riskier to kind of slow down until we better understand sort of these models, you know interpretability, etc.
?
I mean, I think it's really important to ground these hypothetical discussions on what we've learned as an industry.
I mean, the discourse around tech safety has been around for 40 years.
And we went through it with compute.
Like remember when we're like okay, Saddam Hussein shouldn't have PlayStations, because you can use.
GPUs to simulate nuclear weapons.
That was actually a pretty robust and real discussion, but that did not stop us from having other people create chips for video games, right?
I mean, we went through the internet, we went through cloud, we went through mobile.
For the United States, interest balances these two things.
And, you know, listen, we've had kind of areas that were very sensitive to national governance.
Think about like Huawei and Cisco, for example.
And we as a nation did start to put in kind of imported-export restrictions as a result.
And so I just feel these almost platonic – you know polemic questions like the one that you just posed aren't rooted in 40 years of learning.
So all I ask is, if we're going to make a departure from a posture that was developed from 40 years, we better have a pretty damn good reason, And if we don't have a good reason, then I think we should probably learn from that experience.
Yeah, I think extraordinary claims require extraordinary evidence.
And so the burden of proof should be on the party making the extraordinary claims.
And if there's a party who's going to show up and say you know, AI models are like nukes and California should start imposing downstream liability on open source developers for open sourcing the weights.
That's a pretty high claim to make.
And so you should have like exceptional proof if you want to change the status quo.
And the status quo is you do not hold scientists liable for downstream uses of their technology.
That's absurd.
That's a great way to shut down the entire innovation ecosystem and start throwing literally like researchers in jail.
We don't want that.
We want them to be trying to push the frontier forward.
And I just don't think that the tall claims were not being followed up by tall proof.
When we're talking about open source, are we all talking about the same thing meaning, are there degrees of open source or is it kind of just like a binary?
Open weights, I think was the primary contention, which is that if somebody put out the weights of a model and a bad guy took those weights fine-tuned, it did something really terrible.
Two years later, the SB 1047 regime proposed that the original developer of the weights that they put out basically is free information should be held liable, which was absurd.
Right.
So I think Yeah, I just want to make sure we're very clear, because people jump on top of these things.
What he's saying is clear.
So basically, if...
The weights were over a certain size and there was a mass casualty event.
I think catastrophic harm was the word used, but there were no real.
No, it was mass casualty.
There were so many versions.
Actually, I don't know which version you're talking about.
But I remember we actually looked it up.
The legal definition was three or more people were killed or the medical system was overwhelmed, which there was actually precedence of this, including like a car crash.
Right.
Right.
And there was actually precedence of this happening in a rural area which basically doesn't have any sort of capacity.
And so, you know, basically it would move the conversation to the courts and outside of policy, which is again historically, we've taken a policy position on these things which follows precedence, that we understand, you know, to make sure that we don't introduce externalities like, for example allowing, you know, China to race ahead with open source, which is, you know, which has happened.
And the key thing is, by moving it to the court, even if you don't.
One could argue oh Ange, but like sure it's moving to the courts.
That means it's open for debate.
It's not clear that open weights are going to be regulated with liability.
The point is that that creates a chilling effect.
The chilling effect is the idea that when our best talent is considered... I could be sued.
Like, I'm a random kid in Arkansas developing something.
Like, I don't want to be in a world where it can be resolved in the cars.
Right.
Hey, I can't even afford, you know, whatever.
And in a situation where you have an entire nation-state-backed entity like China... actually doing the opposite of a chilling effect right encouraging a race to the frontier why on earth would we want you know that there's this meme of a guy on a bike and he picks up a stick and puts it into his front wheel and topples forward that's the effect of a chilling that is what chilling effect is right at a time when your primary adversary is racing So let's trace how the conversation has changed because we don't see Vinod tweeting about open source anymore.
Obviously, it's changed your tune, especially right now.
Is it really just deep seek?
Or how do you trace kind of how the sentiment shifted on open source?
Let's go through a few theories.
I'm not really sure what happened.
I almost felt like it was almost culturally in vogue to be a thought leader on the negative externalities of tech.
And it kind of started with Bo Strump, but it was picked up by Elon.
It was picked up by...
I mean, a bunch of like these intellectuals that like we all respect and still do.
I mean, they're just really the titans of our industry and our era.
They were asking these very interesting intellectual questions around like do we live in a simulation?
What happens if AI can recursively self-improve?
And then actually, you know, they created whole kind of cultures and online social discourse around this stuff.
And so, I think, to no small part That became a bit of a runaway train and it's just catnip to policymakers.
And so I think part of it is people didn't really realize that this had become so real, because of course GPT-2 comes out and then 3 comes out, and all this stuff's amazing and somehow it got conflated.
So I think part of it is just a path dependency on where we came from, which is kind of the legacy of Bostrom.
I think that was part of it.
I think the ungenerous approach would be that there was a lot of discourse is awesome, but a lot of the people pushing the discourse were first order thinkers.
They weren't doing the math on.
Wait a minute.
If policymakers who have no background in frontier AI which, by the way, nobody does, because this space is only three, four years old start to take discourse as canon, which is a big difference, then what happens?
What are the second and third order effects?
And the second and third order effects are that you start making laws that are really hard to undo and start mistaking interesting thought experiments as the basis for policy.
And once that happens, those of us who've, look, law is basically code.
Code is hard to refactor.
Law is like impossible to refactor.
And so I think the second and third order effects was that of a lot of well-intentioned folks, for example in the existential risk community, saying look, if you're intellectually honest about the rate of progress of AI, it's not crazy to say that there are some existential risks on the technology.
It's non-zero.
Sure, yes, that is true.
But then to then say that that threshold is high enough to start introducing sweeping changes in regulation to the way we create technology, that leap I don't think a lot of the early proponents of that technology realized they would do that.
In fact, I think Jack Clark, who runs Policy for Anthropic, literally tweeted towards the end of the SB 1047 saga.
He was like I guess we didn't realize the impact of how far this could have gone.
And then I think what DeepSeek did was just made it super legible to everybody else.
So I think there were already like.
I think DeepSeek was the catalyst, but it wasn't like there was a step.
It didn't change the reality that the second and third order effects of policymakers confusing sort of like discourse for fact were always going to be terrible.
I just think it brought to light something a lot of us were already saying, which is we're in a race with adversaries, and that should be the calculus we should be working backwards from.
There was always this prevailing view, which has turned out to be so wrong, from really well-intentioned people, which was like it's going to be regulated anyways.
If it looks like we're self-policing, we can dictate how that happens, right?
Right.
And unfortunately that just turned out not to be true because, you know, whatever self-policing we seem to be doing scared the shit out of people.
And then, of course, I would say very opportunistic elements in tech decided to use that for whatever agenda that they had.
And so it kind of got away from us.
Yeah.
Mark had this sort of the Baptist bootlegger.
Yes, I was just going to say, exactly.
True believers and then sort of people who use the sort of that thinking to support their own ends.
And it seems like that's changed, even just on the company.
But the reality is, I think it was driven, I think the majority of people are neither.
The majority of people are pragmatists that are not trying to take advantage of the system.
That think well, maybe if we have this discourse, it's an honest discourse and then we'll self-police.
And then I just feel like the silent majority was not –
Part of the discussion, maybe the biggest change now is like those people are there.
Like the founders are there.
Academia is there.
VCs are there.
Now the people that are not either Baptists or bootleggers are driving the discussion, which is independent of the action plan itself.
I feel much in a better position now.
Like, for example, there's still a bunch of stupid regulation that's popping up, but I'm not calling Ange at night and saying we have to do something now because I feel like okay, there's actually representation that's sensible where at the time there was none.
And I think to move to the action plan.
I think this is a great like if you read the first page.
What a marked shift the fact that the co-authors include technologists.
And I think that was the core problem is, DC is a system like a self-contained system and the Valley is a self-contained system.
And I think a lot of the people here were assuming best intentions over here and vice versa.
And what happened is a few bad actors essentially use that arbitrage opportunity to represent Silicon Valley's views incorrectly in DC.
And when we saw some of the legislation, we had policymakers calling us up and saying wait, you guys aren't happy with 1047.
But the guys, the other tech people, were calling us and saying you'd love more of this kind of regulation.
We said, what other tech people?
And it turns out we are not one homogenous group.
Little tech is extraordinarily different from big tech, which is extraordinarily different from the academic communities.
And I think one of the things we had to contend with was like, we used to be one shared culture.
And then, when tech grew, we actually.
There are some major differences in the Valley, at least between parties.
We're not one tech ecosystem anymore.
We have different interests.
And DC hadn't updated that.
And I think what's amazing about the action plan is it's written by people who have bridged both with enough representation across, like the four or five different subcultures within tech, who have different interests.
I think that's new.
Going back to open source, why don't you talk a little bit about just sort of the how different companies help us make sense of how different companies have have thought about it, or from a sort of uh, business strategy perspective?
You know, maybe we saw meta with maybe the first big open source push um.
You know, open ai has sort of evolved their tune i've seen even anthropic seems to be evolving their dialogue um, a little bit.
Um, how should we think about open sources as a business strategy in terms of what, what's changed here and why?
Oh look, I don't think this is.
This part is actually playing out beautifully along the same trend lines of all previous computing infrastructure.
Databases, analytics, operating systems, like Linux.
The way it works is the closed source pioneers the frontier of capabilities.
It introduces new use cases.
And then the enterprises never know how to consume that technology.
And when they do figure out eventually that they want cheaper faster, more control, they need somebody like a Red Hat to then introduce them and provide solutions and services and packaging and forward deployed engineering and all of that around it.
And which is why the arc generally in enterprise infrastructure has been closed source wins applications and open source tends to do really well in infrastructure, especially in large government customers, regulated industries, where there's a bunch of security requirements.
Things need to run on-prem.
The customer needs total control over it.
Broadly, you could call that the sovereign AI market right now.
Lots of governments and lots of legacy industries are going wait.
This open source thing is really critical to us.
So I think, whereas two, three years ago it was, open source was viewed as like this, like largely philosophical endeavor, which it is, open source has always been political and philosophical by definition, but now there's an extraordinary business case for it, which is why i think you're seeing a lot of startups and companies also changing their posture because they're going wait a minute, some of the largest customers in the world enterprise customers happen to be governments and happen to be legacy industries and fortune 50 companies and they want stuff on-prem.
And that's when you go adopt open source, i say i think there's been a business shift as well.
I don't know if you do.
Yeah, this is So.
I totally agree.
I do think it's interesting to have the conversation where it's the same and where it's different.
Like everything Anshut is exactly right, which is we have a very long history with open source.
And it's a very useful tool for businesses, but also for research and academia, etc.
But let's just talk about businesses and startups, right?
It's a great way to get a distribution advantage.
It's a great way to enter a market where you're not an incumbent and you're a startup.
So it's just kind of one of the tools for building in software that's been used.
And open source has been used in a very similar way, right?
I mean, you can use it for recruiting.
You can use it for brand.
You can use it to get distribution.
And we see all of that.
There's something that's unique about AI that software doesn't have.
And, like we're seeing, very viable business models come out of it that don't have the limitations of traditional software.
And this is for two reasons.
One of them is like open weights is not the ability to produce the weights.
But open software is the ability to produce the software.
Like if you give me open software, I can compile it, I can modify it, whatever.
But giving open weights, you don't have it.
You don't have the data pipeline when you're talking about open weights.
So you don't actually enable your competitors in the same way open software enables it.
So that's one.
The second one is this is very nice business model.
That's kind of a piece dividend to the rest of the industry, which is you – open ways to your smaller models that anybody can use, but the larger model you keep internally, which is actually also more difficult to operationalize for inference right.
I mean, there's kind of good reasons to do this.
And then you charge for the largest model.
And then, you know, the smaller open models you use for band or distribution or whatever.
And so I feel like it's actually almost an evolved from a business strategy and an industry perspective version of open source.
I think it's the AI flavor of open core, which was supposed to be a theoretically sort of sustainable model for open source software development, which was really hard to implement because of the reasons Martine said, where once you gave away the code, it was really hard for you to protect your IP.
But with weights, you can contribute something to the research community.
You can give developers control.
You can allow the world to red team it and make it more secure, while you're still able to.
Actually, because of the way distillation works and some of the ways like post-training works you can still actually hold on to some of the core IP, which then allows you to build a viable, sustainable business.
And that is unique about open space.
But also, you have the data pipelines.
You have the data.
I mean nobody else could – just because I give you the weights doesn't mean you can recreate the model.
Like you could distill it to a subset model.
There's a bunch of stuff you can do but not necessarily recreate it.
And so listen, having been kind of a student of open source business models for 20 years and have watched – it shaped the way that the – the industry has adopted and built software.
I actually think that the AI one is more beneficial to the companies doing it for sure.
But as a result of that, we're going to continue to see a lot of it.
And so I think we should just kind of assume that open source is part of it and every country is going to do it.
And One of the best things about this current AI action plan is it acknowledges that and it wants to incent the United States to be the leader in it, which is such a dramatic shift from where we were this time last year.
Yeah, there's sort of an ecosystem mindset that people who, if you've worked in any kind of developer business which Marty and I unfortunately, have spent way too long doing working on dev infrastructure and dev tools but you sort of internalize this idea that you have to often sort of trade off short-term revenue for long-term ecosystem value right.
And I think what the action plan shows is that yes, in the short term, it may seem like we're giving away IP to the rest of the world by open sourcing weights and showing the rest of the world how to create reasoning models and all of this stuff.
But in the long term, if every other major nation is running their entire AI ecosystem on the back of American chips and American models and American post-training pipelines and American RL techniques, then that ecosystem win is orders of magnitude more valuable than any short-term sort of give of IP, which anyway, as we saw with DeepSeq, that marginal head start is minimal.
Okay, so just to close the loop on open source over the next several years, how do you predict open source and closed source will intersect?
Like what will the industry look like?
Well, I think these are two different markets.
Yeah.
I mean, literally the requirements of the customers are completely different, right?
So if you're a developer, you're building an application and you happen to need the latest and greatest frontier capabilities.
Today you have a different set of requirements than if you're a nation state deploying like a chat companion for your entire employee base of like 7000 government employees.
And the product requirements, the shape of how you provide those, do you deploy them, the infra, the service, the support and then the revenue models are completely different.
And so often I think people don't realize that closed source and open source are not just differences in technology but completely different markets altogether.
They serve different types of customers.
And I think if you believe AI is this sort of explosive new platform shift, then there'll be winners in both.
I do think what we need to contend with is that it seems like it's getting harder and harder to be a category leader if you don't enter fast.
Like the speed at which a new startup is able to enter the open source or the closed source market and create a lead is absurd, right.
We both have the chance to work with founders who are, I mean, literally 20 something year olds, out of college, two years out of college, building revenue run rate businesses in the tens to hundreds of millions of dollars, serving both of these markets expanding like this.
And so I think the biggest mistake is to confuse these two markets as one and to do the classic like oh, let's wait to see how they evolve, because the pace at which a new entrant is able to actually create a lead in the category is quite stunning.
Let's go into the action plan.
What are our biggest reflections from it?
Where are we most excited?
If you look at the quote that they start with.
I wanted to read it out because I thought it was pretty poignant.
It was, today, a new frontier of scientific discovery lies before us.
And I thought that first opening line was fantastic out of all the things they could have said.
You know, they could have said we're in an arms race, which sure the first page, the title, says winning the race.
But if you actually start reading the document, the first sentence is a quote from the president that says today, a new frontier of scientific discovery lies before us.
And I love that they led with something inspirational.
Yeah.
Because ultimately the technology has to confer some benefits on humanity.
And I personally, I just love the fact that We are just starting to explore what these frontier models mean for scientific discovery in physics, in chemistry, in material science.
And we need to inspire the next generation to want to go into those areas because it's hard.
It's really hard to do AI in the physical world.
You have to literally hook up wet labs and start doing experiments in an entirely new way.
And you need people who are excited not only about wanting to do machine learning work, but also the hard work of of being lab technicians and running experiments and literally pipetting new materials and chemistry right.
And that, I think, was missing in a lot of the discourse under the previous administration.
So you can sometimes judge a book by its cover, and I think this was a strong start.
And now I think we should actually dive into some of the bullets.
Okay, so the other one that I thought was a huge omission is there's basically no real mention of academia, investing in academia.
There's some oblique references to it, but it's just been such a mainstay of innovation, computer science of the last 40 years not having a major part of it.
I think it's a shame.
And I understand that right now there's kind of a standoff between higher ed and the administration.
I get it.
And I actually think that both sides actually have fairly reasonable points.
But to have a major tech initiative Without including academia.
It just feels like we're.
You know what is it?
Fighting a battle with a hand tied behind her back?
Like some aphorism.
This is a good problem to have, which is that I think it's extremely ambitious.
It's a little bit light on execution details. right, which is what happens next.
So a good example of that is, I do think, directionally.
I love that because it acknowledges that.
Hey, before we start actually passing grand proclamations of what these models are risky or whether these models are dangerous or not, let's first even agree on how to measure the risk in these models before jumping the gun.
That part, I think, in addition to the open source bullet, was probably, I thought, the most sophisticated thinking I've seen in any policy document.
And look, the reality is America leads the way.
And so every other.
Within 24 hours of this dropping, Marty and I were getting texts and messages from folks in many other governments around the world going.
What do you guys think?
And it was not hard for me to endorse it and tell them like, look at it as a reference document, because there are things here that arguably are more sophisticated than policy experts, even in Silicon Valley, would recommend.
Because building an AI evaluation ecosystem is not easy.
And I think they had a pretty thoughtful proposal on the fact that that's important.
Now the question is, how?
And I think that's what we have to help DC with, the hard work of like implementing this stuff.
But the vibe shift going from let's not jump the gun on saying these models are dangerous.
Let's first talk about building a scientific grounded framework on how to assess the risk in these models.
To me was not at all a given.
And I was really excited about that.
Yeah.
There's been a lot of focus in the last few years by several companies, but also by the broader industry, around this idea of alignment.
Have we made any progress on alignment?
What is your perspective of what are they trying to do?
Is that a feasible goal?
Help us understand what they're trying to solve for.
So at an almost tautological level, alignment is an obvious thing you'd want to do.
I have a purpose.
I want to align the AI to this purpose.
And it turns out these models are problematic, generally unruly chaotic, whatever adjectives you want to use.
And so, like understanding how to better align them to any sort of stated goal is very obviously a good thing.
And so I think we'd all agree that alignment to whatever the goal is to make it more effective that goal and do that thing is good, especially given these models who tend to have a mind of their own.
The subtext that certainly I bristle to is, that is, that the people doing the alignment are somehow protecting the rest of us from whatever they think their ideal is, as far as dangers to me or thoughts I shouldn't have or information I shouldn't be exposed to, which is why I think we need to be – even when we come up with policy we need to be very careful not to impose like a different set of ideological – rules on top of these.
I just think like alignment is something we should all understand.
Actually aligning them to me is kind of where I take issue from any sort of kind of top-down mandate.
I agree.
And I think...
There's a quote from a researcher which I think is very accurate, which is you've got to think about these AI systems as almost biological systems that are grown, not coded, up.
Because sure, they express as software, but in many ways when you're training a model, you are actually growing in this environment of coding a bunch of prior history and training data, et cetera.
And often empirically.
You actually don't know what the capabilities of the model are until it's actually done training.
So I think that's a useful analogy.
Where I think that falls down is when people go oh well if we can't align it, because we actually don't know.
It's a biological mechanism.
Until it's grown up, you don't know what its risks are and so on.
Then we can't deploy these AI models in mission-critical places until we've solved, let's say, the black box problem, the mechanistic interpretability problem, which is can you trace deterministically why a model did something?
We've made a lot of advances as a space in the last few years, but it still remains a research problem.
But that doesn't mean just because you don't understand the true mechanism of the system doesn't mean you don't unlock its useful value.
If you look at most general purpose technologies in history, electricity, nuclear fusion.
Like we.
There are many examples of technologies where we knew they were complex systems and we didn't truly understand at an atomistic level or mechanistic level how they work.
But we still use them.
And we don't understand how the internet works.
I mean like there's a whole research of network measurements trying to find out what the heck the internet was going to do.
Is it going to have congestion collapse?
I mean, like...
Any complex system has states that you just don't understand.
Now let's not say these models more so than many, and the implications are very real.
But we know how to deal with ambiguity.
We don't even know how our own brains work.
Her consciousness.
But we don't stop working with other human beings.
Unfortunately, we're stuck with them.
We have no option on that one.
I think to extend that analogy, what do you do?
You're like, okay, I don't know how a brain works.
It's got a bunch of risks.
This person may be crazy, but I still want to unlock all the beautiful benefits of the big, beautiful brains that humans have.
And so you develop education.
You send kids to school and you teach them values, and then you send them off to college and then they get to learn something specific and then you get to test them in the real world environment.
They get a resume and they get work experience and they get to prove that they actually are, within a risk-based framework manageable, and so on.
And that as a society is unlocked, Human capital, right?
Arguably the greatest technology we've had in 500 years of modern industrial innovation.
So I think what I hate about the alignment discourse is it sometimes confuses the fact that we don't understand the system for the fact that then we can't use it.
And I think I don't think we've like for a long time.
I think mechanistic interpretability, which is kind of like some folks would say is the holy grail, is like being able to reverse engineer.
Why a model does something is still a research problem.
But that doesn't mean we haven't made progress on how to use unaligned models or to improve alignment to a point where they're useful in massive ways, like software engineering.
And I think what the smartest might say is it's not that, it's really just what's the rush.
Like you know, maybe let's focus on, like you know, integrating all the capabilities we already have before.
You know, pushing the frontier, to which then it ends well, but the arms raise, et cetera.
Like there's a risk of slowing down too that maybe isn't fully appreciated.
Until we've solved cancer.
Every month that we're not rushing to the frontier of accelerating biological discovery or scientific progress is a month that millions of people are suffering from disease that we could be solving with AI.
We don't talk about the opportunity cost of slowing down the frontier.
I mean, this is the thing with all of these, like there's always this kind of reverse question on innovation and they say well okay, it's like the Bostrom urn experiment.
You know, his kind of whole urn hypothesis, like there's an urn of innovation and you pull up balls.
One of them is a black ball that destroys everything, right?
Like, so...
Eventually, you'll draw that ball, so why would you ever do an evasion?
That is the thought experiment.
And the answer is so simple, which is?
It just turns out that it's much more dangerous not to pull out balls than pull out balls.
That's always the answer.
So when people ask PDoom, so what is the PDoom?
The answer is not like 0.1 or 0 or 100.
The answer is the PDoom without AI is actually quite a bit greater than the PDoom with AI.
And the what's the rush?
The answer is the same thing, which is clearly if you ignore – exactly what I'm saying.
If you ignore the benefits of technology, then you would say if it's all negative, no rush at all, right.
The reality is the benefits are – they're so dramatic and they're so obviously dramatic now.
Thank God we've got a year's worth of – of data on this stuff.
Like they're clearly economically beneficial.
They're clearly beneficial in expanding a number of areas of like core science, that the rush is getting to the next set of solutions, You know, as opposed to being afraid of a set of problems that we still can't clearly articulate.
And listen.
As soon as we do understand marginal risk and we do have these we absolutely should address those directly.
Which, again, the Action Plan does a great job of penciling this out.
I mean it does want to explore implications on jobs, implications on defense, implications on, you know, alignment.
And that's exactly where we should be in the exploration base.
Do we have a definition of marginal risk or a perspective of how to think about that idea?
Well, let's just be clear what we mean by marginal risk, which is computer systems are risky.
Network systems are risky.
Stochastic systems are risky.
We've got decades of...
You know ways of thinking about measuring regulating, changing common behavior based on this type of risk.
And so the question is, can you take all of that apparatus that's been hard won and apply it to AI?
If so, like A.
We know it's effective because we've used it before and we've got a lot of experience with it.
And B, it's ready to be done.
Or is there a different type of risk that's not endemic on those systems?
In which case, we'll have to come up with something that new, which you go down that exploration.
Maybe it works, maybe it doesn't work, et cetera.
So that's what marginal risk is.
And I just think that the problem is if you don't know what it is, how are you going to define a solution?
I think that's right.
I mean philosophically.
The idea is, if you're going to say we need new solutions, then you need to articulate why the problem is new and why are solutions that work really great are no longer sufficient, right.
And I think it's almost obvious when you state it, but this was the state of the world a year ago, that we were having to look around the room and say can I raise my hand?
Why are we introducing net, new liabilities and new laws that we've never had to do before if you can't articulate why there are no new problems to solve?
If it ain't broken, why are you trying to fix it?
And so marginal risk is, I think, a slightly just more technical way to say we have the tools to manage risk, we don't need new ones.
And if you think we need new ones, then hey, just take a minute to articulate to us why.
Is there anything else you wanted to make sure we got to?
Otherwise, that's great.
Time to put the action plan into action.
Excellent.
Martin, Ange, thanks so much for coming to the podcast.
Thank you.
Thanks for having us.
Thanks for listening to the A16Z podcast.
If you enjoyed the episode, let us know by leaving a review at ratethispodcast.com slash A16Z.
We've got more great conversations coming your way.
See you next time.
