English 箭头
Podcast Cover

[The Era of Evals: How Mercor Became the Fastest-Growing Company in History]-[Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor)]

Lenny's Podcast: Product | Career | Growth · B2 · 2025-09-18

Technology
Or study on the web version

📋 Summary

The Era of Evals: Unlocking AI Through Human Expertise

We are currently witnessing a fundamental shift in the artificial intelligence landscape. As Brendan Foody, CEO and co-founder of Mercor, explains, the industry has entered the "era of evals." While much of the public discourse surrounding AI has focused on job displacement, a new, massive category of work is emerging: the creation of reinforcement learning (RL) training data and evaluation rubrics designed to guide AI development.

The Bottleneck: Why Evals Matter

For top AI labs, the primary challenge is no longer just computing power or raw data; it is the ability to measure model success. Foody notes that "if the model is the product, then the eval is the product requirement document." Labs need to know exactly how to measure progress, whether it is for software engineering, investment banking analysis, or legal contract redlining.

Mercor has positioned itself as the labor marketplace for this necessity. By sourcing extraordinary professionals—ranging from FANG software engineers to Emmy award-winning screenwriters—Mercor provides the human intelligence required to create high-quality rubrics and verifiers. These humans define what "good" looks like, allowing AI models to learn through reinforcement learning from AI feedback (RLAIF) and human-in-the-loop assessments.

The Rise of Mercor: A Record-Breaking Ascent

Mercor’s growth is unprecedented, moving from $1 to $400 million in revenue in just 16 months. Foody attributes this success to several core tenets:

  • Market Pull and Leading Indicators: Rather than forcing a product into a stagnant market, Mercor identified a massive vacuum. They realized that the world’s wealthiest AI labs were desperate for high-caliber human experts and were willing to pay for them. By focusing on these leading indicators, they scaled rapidly.
  • Customer Obsession: For the first 18 months, Mercor operated without a traditional sales or marketing team. Their growth was driven entirely by word-of-mouth and the delivery of "six-star experiences" to their customers.
  • Talent Density: Foody emphasizes that hiring the right people—often former founders or top-tier industry experts—was the key to maintaining a high standard. He notes that while they were patient with their first ten hires, once they confirmed the market fit, they shifted gears to scale rapidly.

The Future of Work and the "Elastic" Economy

Foody challenges the narrative of AI-driven job loss by highlighting the concept of "elastic demand." He argues that in industries like software development, where AI can make humans ten times more productive, the demand for output will not shrink—it will explode. We will simply build 100 times more software.

He encourages individuals to focus on skills that leverage AI rather than competing against it. "Don't fight against them using the models," he advises. Instead, professionals should learn how to integrate these tools into their workflows to achieve greater results. He believes that the most successful people in the future will be those who lean into this "narrative of abundance."

Conclusion: The Long Road to AGI

Despite the hype surrounding superintelligence, Foody remains pragmatic. He believes we are on a long road that will be paved with millions of specific evals. The path to automating knowledge work is not just about more pre-training data; it is about the thoughtful, data-efficient post-training work that aligns models with human intent. For those looking to participate in this future, his advice remains simple: "You can just do things." Whether it is building a startup or mastering a new AI workflow, taking the initiative is the most critical step in an era where the barriers to creation are lower than ever.

🎯Key Sentences

1
The wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities.
2
We're entering the era of evals.
3
Why is this so valuable?
4
The market is bound by the amount of things where humans can do something that models can't.
5
If you really think about it, we were put on Earth to create reinforcement learning training data for labs.
Expand All

📝Key Phrases

1
whatever it takes
2
revenue run rate
3
product market fit
4
bottleneck
5
job displacement
Expand All

📖 Transcript

The wealthiest companies in the world are willing to spend whatever it takes to improve model capabilities.
We're entering the era of evals.
We started working with all of the top AI labs.
What the labs need is a labor marketplace.
They actually need extraordinary professionals that can measure model capabilities.
You found this pocket, maybe the biggest business opportunity in history.

ListenLeap Brings You Into Real Context Learning

🎨 Interesting Content
🌍 Real Materials
📱 Listen Anytime
Or study on the web version