Hello, and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. Ever since the internet went mainstream, people have been struggling with perhaps the ultimate question.
What do I call my website? What domain name do I register?
But almost as quickly as that became a question in people's mind, the secondary question sprang up. what's my domain name going to be worth a week, a month, a year, or whenever I'm done with it and might want to pass it on.
Well, thanks to AI, we have a better answer to that question than ever.
Joining us today is Jason Ansell, Senior Principal Engineer with GoDaddy.
And Jason's here to talk about how they're using AI to help you figure out what a domain name might actually be worth down the line.
Jason, thanks for joining us. Thank you for having me.
So first things first, what do you do at GoDaddy?
I'm a senior principal engineer here at GoDaddy and I use machine learning and deep learning to try to help small businesses compete in an increasingly technological world.
So how does your work play into not only helping me as a small business person pick the best domain name, but also let's say I want to speculate on the domain name market.
How does your work play into that? Yeah, so the domain name aftermarket is a really interesting thing, and it's been growing a lot in recent years.
If you have a domain name and you want to sell it, you can go to GoDaddy or a website that GoDaddy offers called Afternick. and you can list your domain name for sale.
In the current experience, there'll be this box where you have to enter in what the value of the domain name is.
If you're a expert in domain names, you've been in the domain name industry for a while, you have a pretty good sense of what to enter into that box. but it can be very hard for a small player or an individual who's maybe selling their first domain name to know what a good value is.
So in the last year and a half, we've been using deep learning to try to model this market for domain names and predict what a domain name will sell for.
And our deep neural network is currently about 25% better than a human expert.
So we can give this small player in the market advice about what to ask for and or if they're buying, we can give them confidence that they're getting a good value. and help these sort of little players compete with these more experienced domainers.
So you said your system currently can predict about 25% better than a human can.
Can you break that down a little bit in terms of...
I'm going to get a 25% better value for my domain name or just kind of walk me through the experience.
I'm registering, uh, Noah's amazing widgets.com.
What happens? So the way to think about the value of a domain name is what would it sell for?
At GoDaddy, we operate the largest aftermarket for domain names.
So we have lots of data. We have more data about what names sold for in the past than anyone else.
How big is the market roughly? So if you go to GoDaddy today, you'll see tens of millions of domain names listed for sale.
And then not all of those domain names sell.
So looking at historical sales, there's millions of sales and you can divide those sales into different types.
So there's. There's sale models where you auction a domain name in a short period of time, and there's sales models where You list a domain name for sale and perhaps have a haggling process to arrive at a price with a buyer.
Gotcha. Okay, so I'm typing my domain name into the box and what happens?
We will take that domain name. Domain names are an interesting problem area because...
There's aspects of natural language processing in there, like domain names consist of words, but they're not really sentences.
And there's also lots of pieces of a domain name that aren't words that are important to take into account.
You know, for example, If you're looking at some domain name .com, if you're using a housing analogy, your neighbors are the .net, the .org, the .info.
So just like in housing, like if your neighbor is a vacant house, you know, that's probably bad for your value.
And if your neighbor is a really nice house, it's good for your value.
Right. We can look at how each of these neighboring TLDs is being used.
We build a vector embedding to model every use of those things.
And we also use natural language processing techniques.
So we use an RNN, a recurrent neural network, with a vector embedding of words.
And then we take the output of that RNN and we feed it into a fully connected neural network where we incorporate these other features that aren't associated with the individual words in the domain name.
So when you're talking about natural language processing for something like domains, and you gave the example of, you know, to use your real estate analogy, location is everything and you've got these different dot extensions.
Mm-hmm. I've noticed in the past five years, let's say, there have been trends where there were a lot of websites that used .us at the end, for instance. and would use those to create a word, an English language word that people recognized.
Delicious was the one that comes to mind, right, with the .us at the end.
Do these kinds of things, these little plays on word and language and domains that don't use vowels, for instance, that kind of thing, do those throw a wrench into your... modeling process and how you approach natural language.
It almost seems like you're looking at a colloquial offshoot of English, for instance.
Yeah, and it could be challenging in that the language that one uses for domain names It's like English or whatever language you're using, but there's definitely the way that words are used in domains is a little bit different than how they're used in natural language like books.
And so what we do is we build our own vector embedding to create a language model for the way language is used in domain And we do that using a different data set where we actually take the list of every single domain name that's registered, which is hundreds of millions of domain names,
And we use sort of a word-to-vec, Skip Graham-type model to train a neural network to start guessing domain names, where you'll give partial domain names and the neural network will have to guess the rest of the domain name.
And over time you build a language model That represents how things are used in domain names where, you know, in, in sentences like the letter I has one meeting, but if you put I at the beginning of a domain name, it has a little bit of a different meaning than in English.
Different connotation, totally. That's fascinating.
So let's go back a little bit and talk about your background.
When you were a kid, did you know you wanted to be a... machine learning expert working with the different connotations of the letter I and domain names.
How did you get into this stuff? Yeah, so I was interested in computer science very early on.
And so I... was teaching myself C++ when I was in high school and was definitely excited about it.
I got a PhD from MIT about four and a half years ago, and my PhD was on the boundary between compilers and programming languages and machine learning.
So I was creating new programming languages that allowed programs to adapt.
So it's something called program auto-tuning, where we found that the fastest program you could create was different on different arc So if you were on an AMD machine, you might want to run a slightly different algorithm than if you were on an Intel machine to get the best performance.
And so we created programs that could change and adapt to get the best performance on their specific architectures.
And so from MIT, did you go straight to GoDaddy or how has your professional career since then evolved?
Yeah, so towards the end of my doctorate, my office mate at MIT founded this company called Locu.
And so I was working part-time for Locu and then doing machine learning at Locu.
So yeah, The machine learning I was doing there was trying to model web pages and extract content.
So let's take this sort of semi-structured process web content and assign semantic meaning to it.
And then GoDaddy acquired Loku, and that's how I joined GoDaddy.
And for a number of years at GoDaddy, I was continuing that work of trying to model semantic meanings on the web.
And then about a year and a half ago, I switched to a different team in GoDaddy and started working on this domain valuation problem.
Gotcha. Very cool. So as you mentioned, the domain market doesn't seem to be slowing down.
There are tens of millions of domains registered and in use, millions more that are are for sale right now, being auctioned off and what have you. see the domain market in general or even the specific work you're doing with predicting the value for customers, where's that all headed?
Yeah. So I see the domain name aftermarket only growing.
You know, the number of dot coms available is is fixed.
And and so There's not going to be a new growth of four or five Letter.com domain names.
And so I think we're seeing many million dollar or greater than a million dollar domain name sales.
And I think we're only going to see more in the future.
And one thing that I really hope is that by using deep learning to predict the value of domains, we can democratize the market and really just grow it in general where. if we make it easier for people to participate and act intelligently in the aftermarket, then you can start seeing just regular people start investing in domain names and viewing domain names as an asset class that they might add to their portfolio.
And without either giving away any of the AI's secret sauce or putting you in the role of a stock analyst, so to speak, any tips for the first time? domain speculator.
Yeah, so I think it's the model for a domain name investor is you buy a lot of domain names and you expect to sell some small percentage of them.
So sort of a more successful domain investor might sell 5% of their domains in any one year.
So I think it's about finding domain names with good value and making a lot of bets just because you need... to make enough purchases such that sort of the law of averages can cause you to do well in the market.
Right. So GoDaddy obviously has a wealth of data about domain registrations, domain sales available to you.
And I understand you're using it in kind of a unique way with a tiered approach to neural networks.
Can you speak a little bit about that? Yeah.
So we have lots of different types of data.
And so, as I said before, there's Looking at all sales combined, you're looking at millions of data points, and then if looking at listings, this person decided to list this domain for sale and it didn't sell.
You're talking about tens of millions of data points.
Right. And then for all domain names, hundreds of millions.
And within each of these categories, there's different subtypes.
So there's, you know, You want to model an auction a different way than you'd model a thing where you listed it for sale and waited for three years.
And similarly for listings, it matters who's listing it because people have different strategies and also the time.
And so... The way we use all these different data sources is we actually have a tiered approach with a total of 11 neural networks.
So we have neural networks that model the different markets.
And those neural networks will use vector embeddings to represent different what we call contexts.
So the idea is that a sale that involved negotiation and haggling over the phone will have one context, which will create a vector embedding for And we'll also create vector embeddings for different years because the markets change over time.
And then we also have models that predict... list prices.
So what would this person list for versus what this another person would list for?
And we model the probability of sale. So if you look back in the history of a domain name, you might have something that was listed for $4,000 for one month.
And then the person decreased the price to $3,000.
Then they took it off the market for a week.
Then they put it back on the market for... something so you have these sort of complex histories of different list prices and We train a neural network to basically say okay after this complex history.
Did it sell yes or no? And each of these different models can be very good at their specific problem.
But if you start combining them, and taking the output of some of these models, then feeding it into the others, You can create approach that can have higher accuracy than models trained on just one data set.
So the way we do it is we have sort of a first tier of neural networks that train on just one data source each.
And then we have a second tier above that that takes the outputs from that first tier and retrains on each of these data sets.
So you're training, trying to break the probability of sale, But you have the outputs of all these other models that are trained in different contexts.
And then we have a third layer on top of that, which we call the calibration layer.
And the reason why we need that is that our models are context-aware in their predictions.
So we'll predict a different price for a negotiation that goes over the phone with haggling versus sort of a 30 day auction versus a, you know, what do we call a buy it now where someone just clicked a button and bought it for the list price without haggling at all.
And so the overall market is some combination of all these different ways a sale could happen.
So what we do is we take all of the raw predictions, basically every possible context of sale could happen, and we feed those into a third layer which is also a neural network, which comes up with a single prediction, which is as representative of the overall market as possible.
And so when you're offering advice, so to speak, to your customers, to would-be sellers, is it including the type of listing they should have, buy it now versus auction versus...
Buy it now versus auction. And it's also including the length that the listing should be up there.
Yeah. So looking forward, one thing we might be doing next year is creating tools that for domain investors to help model the portfolio.
We're basically saying like, well, if you adopted this pricing strategy, here's how we think you're going to do.
If you adopted this other pricing strategy, Here's another thing you could do.
There's also just general advice that's really obvious when you look at the data.
And so that stuff like you should put a reasonable price on your domain names in that a large number of people either put extremely high prices on their domain names, which turns off a lot of buyers or they don't put a price.
They basically say like, hey, as a potential buyer, you have to make an offer.
And so you should as a seller of domain names, you should price all your domain names reasonably.
You should have them configured so that a buyer can with just one click get that domain name transferred to their account.
And then looking forward to next year, we might have some interesting modeling tools where we use these probability of sale predictions, where we can basically say, you know, here's the graph where on one axis, you have all the possible prices you could ask for.
And then on the Y axis, you have, here's the chance that you're going to get a sale in one year.
So then in your tools that you offer to customers when you're advising them, so to speak, on domain investing, Do you offer up specifics in terms of, oh, for this domain or for your portfolio, you should use... buy it now versus auction.
You should use a low listing price versus a higher one.
Do you get into that level of detail with the predictions?
Yeah, so the current offering that we're showing our customers is just showing one price, and that's the price that it's most likely to sell at.
Internally, we have these probability of sale models, so we can generate graphs where the at every possible list price, here's the probability that that domain name will sell.
And from that you can get, you know, one-year revenue maximizing prices, and you can create all sorts of interesting tools where you you can model a domain investor's portfolio and basically show them the trade-offs basically saying, if you price this way, here's how we think you're going to do.
If you price this other way, you're going to do differently.
And in, And in the future, we may be exposing different different tools like this to our customers to help them price their portfolios optionally.
But there's just general advice that we're giving our customers today.
And then that general advice is you should put a reasonable price on all your domain names.
You know, there's, there's many people who either price far too high or they don't put a price at all where they they're forcing their customers to make an offer. then enter a haggling process which is slow and and can turn off a lot of potential buyers and so our general recommendation would be put to put reasonable prices on other domains and configure them so that they can be bought instantly so that someone can click a button and get the domain.
So all of this leads me to a kind of a general question about the ability to use the models and all the work you've been doing.
And if you put a different set of data into it, what might happen?
I'm thinking about the psychology of buying and selling and where you price your offering in terms of the housing market, which is a hot topic where we are right now in the Bay Area.
Yeah, there's definitely a lot of similarities to the housing market in how... I think if you look at the Zillow's estimate...
Increasingly, the Zillow estimate is starting to impact the sale prices of houses where Initially, the directional flow of information was sale prices informed how the Zillow's estimate was built.
But today, it's bidirectional, where if the Zillow's estimate is high, then you're house is likely to sell for more, and if the Zillow's estimate is low, it's likely to sell for less.
One thing we're hoping will happen in the future is we'll start to bring more consistency to the domain name market, where if you look at... sales in the past, there's just a lot of randomness and a lot of noise.
And, you know, some buyers overpay and some sellers, you know, for too little.
And as we just put more information in both parties' hands, we can make the market more efficient and more more fair to everyone.
Great. So I know you've been hard at work on a technical paper submission to a conference, and I know some of our more technical leaning listeners are going to want to dig in on this.
What can you tell us about? Sure. So we've been working on a novel technique for generating what's called prediction intervals.
A prediction interval is a lower and upper bound, and the idea is that the true value will fall within that prediction interval some target percentage of the time. you know, say 90%.
So because we have so much data, what we found is that a lot of the prior published techniques for generating prediction intervals don't work that well. and one of the reasons they don't work that well on our data is many of them assume normally distributed error, and...
Others assume symmetric error. So basically the idea of being off by negative 100 is the probability of that is the same as being off by positive 100.
And so when we're looking at errors, we don't see symmetric or normal data.
And so we've come up with a new technique that utilizes our large amount of data to generate these prediction intervals.
And the basic idea is that we train a deep neural network that outputs this lower and upper bound directly.
So rather than trying to model the error, we directly output the lower and upper bound And we use a loss function where we take the bounds that are outputted by that neural network.
We then automatically scale them so they cover the target percentage of a mini batch.
And we use larger mini-batches for our other null networks so that we have a better estimate of the coverage.
And then the objective function is to minimize the width of that error bound.
And so what we found is that our technique, depending on the target, so depending on whether you're trying to get 90% prediction intervals or 90%, or 70% confidence intervals, our prediction intervals are 20 to 2x tighter than the prior published techniques for generating these.
Very interesting. So how's that going to help you or our listeners investing in domains to make more money?
Yeah. So our current public tool just probably puts a single number, but this will help people understand the uncertainty.
And so the idea is that there's some domain names where the price is pretty certain.
So there's a pretty narrow range. for what the price could be.
And there's other domain names that are, there's a lot more uncertainty where maybe the model just is like, it could be a thousand or it could be, you know, 10,000.
And yeah, By outputting prediction intervals, we're able to help communicate the level of certainty that our model has to our customers.
And so this also has many implications in other fields.
So if you're looking at stocks, you could use this to model the risk in a way that isn't assuming that symmetric errors so that you know the chances of a stock doubling are the same as the chances of a stock having um which is you know sort of a more sophisticated way of modeling risk, you can use it for modeling things like climate where the risk of some runaway scenario that gets way worse, might not be symmetric in both directions.
I love the idea that Domain investing is actually the key to solving climate change problems.
Totally. And I think it's really the revolution here is the data and the size of the data.
So you look at... The papers that have been published in prediction intervals 10 years ago, and they're using data sets that are 1,000 samples or 10,000 samples.
And the type of techniques that we're using just don't scale down that much.
And so in this era of big data, all of a sudden, you can use really more powerful and different techniques that just don't work for the data sets of a few decades ago.
And is that still accelerating in terms of your ability to... get better output, crunch more data, tweak your models as the tech is allowing you to do so?
Is that still going to ramp up in the next few years?
Oh, definitely. I think that we're looking forward.
We're only going to have more data. And the computing power behind it is growing exponentially. at a really fascinating rate.
And so I'm hopeful that we'll find sort of even better techniques in the future.
Awesome. Jason Ansell, thank you so much for coming on.
This has been a fascinating look at everything from the psychology of free markets to how you guys are using the data in ways that literally couldn't be done just a few short years ago.
Listeners know GoDaddy.com, but If they want to find out or even get their eyes and hands on your domain appraisal tools, where can they go?
Yeah, thank you for having me. This was a lot of fun.
And you can go and try out our domain appraisal tool.
By going to godaddy.com and clicking the domain appraisals, I think the direct link is godaddy.com slash domain dash value dash appraisal.
But, you know, if you search for that, you'll find our tool pretty easily.
Perfect. And the beauty of podcasting is listeners can just keep hitting that 15 seconds back button until they get it right.
Exactly. Awesome. Thank you so much. Best of luck with all you guys are doing.
And thanks for being here. Great. Thank you.
