Hello and welcome to the NVIDIA AI Podcast.
I'm your host Noah Kravitz. How do galaxies and cosmological structures form and evolve?
What is the nature of dark matter? And can we understand the reionization of the intergalactic medium?
No, we haven't changed our name to the NVIDIA Astrophysics Podcast.
Instead, these are the questions that greet visitors to Brent Robertson's website. an associate professor at the University of California, Santa Cruz, and the Maureen and John Hendricks Visiting Professor at the Institute for Advanced Study in Princeton, New Jersey.
AKA the place where Einstein hung out when he was in the States.
Brandt's areas of research read like a who's who and what what of things I don't understand, but I'm pretty psyched to learn about over the next 20 minutes. theoretical topics related to galaxy formation, dark matter, hydrodynamics, and numerical simulation methodologies.
Brent's session here at GTC 2019 is titled AI and Astrophysics, Applying Artificial Intelligence and Deep Learning to Astronomical Research.
And that's what we're here to talk about.
AI and astrophysics. Professor Robertson, thanks for making the time to join the AI podcast.
Hey, thanks so much. Thanks for having me today.
So I feel like I'm done here because I read all that stuff.
I thought it'd make kind of an amusing introduction, but you're deep in it.
And I'm going to start with the basic question.
What is astrophysics? Astrophysics is the study of physical processes that occur in the universe around us.
So astronomy, which is part of astrophysics, is the observational tools and methods that we use to look at things in the universe, to look at galaxies, to look at stars. to look at larger scales in the universe.
Astrophysics is trying to understand the detailed physics of what's going on in the universe.
It combines together astronomy and physics into a single discipline.
So you worked on the Hubble telescope project, a name that folks like me are familiar with, a while back.
And then for about five years, you were co-chair of an organization using data from something called the Large Synoptic Survey Telescope. with the aim of better understanding how galaxies form and evolve.
So in lay terms, if you can, tell us what these types of instruments do And when deep learning became a part of your toolbox and important to using the data you collect from them?
Yeah, that's a great thing to talk about.
So the Hubble Space Telescope has been kind of the workhorse instrument in astronomy since it was launched.
It's a bit old now, and so now there's some new facilities that are starting to come online, one of which you mentioned is the large surveys, Synoptic Survey Telescope, LSST.
Right. We always go by acronyms. We don't care too much.
I should hope so. So LSST, which is still under development and will be coming online in the next few years.
We'll take an image of the entire southern sky every three days, a digital image.
And so this is a huge amount of information, terabytes of information. every single day are generated by this facility.
We're interested in learning about the galaxies and the stars that are in these images, what they tell us about how galaxies evolve with time, what the large-scale properties of the universe are, what's in the universe, dark matter, which is this kind of mysterious material that doesn't interact with light and the dark energy, which is another mysterious material. that's causing the universe to expand more rapidly.
So we want to see what's in these images.
And today, when people go and take a picture of a galaxy, often they know what they're pointing at.
They point at a specific galaxy. They're not doing a survey of the whole sky. like we will be doing.
So they already know what they're looking at before they take the picture.
Now we're going to take huge picture of the sky.
This is building from the Sloan Digital Sky Survey, which was the first digital sky survey LSST is very much bigger telescope, has a very much larger camera.
So the data rate is just enormous. So if you want to see what's in the you know, in these pictures that we're taking, how can a human do that?
Or how can even the entire internet of people out there do that?
So there are There are existing projects like Galaxy Zoo is a famous one where people go online and they open up a browser and they say, oh, I think this is a disk galaxy.
This is a spheroid galaxy. But now with this enormous digital data set, we can use AI to do that for us at a rate and at a scale that just is not possible for individuals to do.
And furthermore, we can actually get additional information out that humans aren't very good So we can actually go at the pixel level in these images, and there are billions and billions of pixels in these images that we'll be taking. and go to every single individual pixel and try to categorize what we think that pixel What information is there?
So is that pixel part of the background sky?
It doesn't have anything in it. Or is it part of a disk galaxy, that pixel itself, not just the whole galaxy?
But what is the probability that that pixel is in a disk galaxy or a spheroidal galaxy.
And why is this important? Galaxies are complex.
They have different properties. So a disk galaxy might have a bulge at the center.
You might want to tell those things apart.
We call this semantic segmentation in the AI world.
This is exactly what we're doing. So it's applying techniques that are well-trod already within AI, but to data sets that are astrophysical in their nature.
So I don't know if this is a fair question or not, but it's my brain's attempt to start wrapping itself around what you're talking about.
When you're talking about these images with billions of pixels and getting down to the individual pixel level to extract information, what does one pixel represent in terms of square miles? or whatever the right metric is.
Oh, they're enormous. Yeah. You know, so astronomers use ridiculous units because... because the scales are so incredibly massive.
So a single pixel might represent a scale that's, say, three or 5,000 light years across in some of these galaxies.
So there's lots of stars and gas in every single pixel.
And what's neat is that this changes the colors of the pixels.
So if you look at an image, of course, when you see one of these beautiful Hubble Space Telescope images, those have been massaged.
So they take the raw data that comes out, they do some further processing to make it scientific quality.
And then they take those scientific images, and then they assign colors to those individual bands.
Okay, so this is actually the way that many people have already applied AI to astronomy. just a JPEG picture of a galaxy and fed it into Inception or one of these other AI frameworks that have been trained on ImageNet.
And that actually tells you, oh, this is a galaxy.
It's a type of galaxy. But what it doesn't do is this semantic segmentation And so my student, Ryan Housen, who is a PhD student at UC Santa Cruz in the computer science department, has developed this new framework called Morpheus, which is basically a unit for those that know what that is.
And what it does is it takes in the raw images, well, the science grade images off the telescope in their original formats, You don't have to process it into a JPEG.
You just feed it into the unit. And it does this per pixel classification based on both the features that the AI framework has learned in the image, but also the colors of the pixels because we have multi-wavelength, the color of the light, the wavelength of the light.
We have multiple of those images from HST or from these future facilities.
And then we can use that to have even more information for Pixel that we're getting out.
So two quite well. There are dozens of questions in my head, but two pertinent.
How long does it take to process one of these science grade images through this framework?
And how do you know either what you're looking for or maybe a better question is how do you know, how do you determine accuracy?
Yeah, these are excellent questions. And actually, that's what is really important about what we're doing, I think, addressing both of these issues. we have to be able to analyze these entire datasets.
However, we also want to maintain a high level of accuracy.
And what we learned was if we just supply, say, a single image to the framework, the framework at the per-pixel level was not necessarily very accurate.
So what instead we do is we reprocess every image many, many times by shifting it basically unit stride by pixel by pixel.
So this gives us a distribution of the classifications of any given pixel in the image.
And this allows us to make some quality cuts to figure out when the model is working well, when it's not.
And we can get more reliable information out that way.
We can test that accuracy by comparing with the way that humans have classified the galaxies.
But we can actually do more specific tests where we can look at data sets where people have taken Hubble Space Telescope data and identified all the galaxies. down to some brightness level of the galaxy.
And then we can see how well our model that, how it detects all of those galaxies or not, the completeness, we call it, of that model.
And that's something that most people don't pay attention to And what Ryan Housen has been able to show with his work is that the completeness of this deep learning framework is actually as good as the standard techniques.
And it works well beyond where it's been trained.
So let's say we only look at bright galaxies and we train on the brightest, biggest galaxies, and we'll actually still find many of the smallest Venus galaxies we know of.
Yeah. Very cool. Yeah. How much And maybe it's not how much compute power do you need to do this, but is crunching the amount of data you're working with right now, would that have been possible two, three, five years ago?
No. So the largest sized images we've been doing to date are 100 million or close to a billion pixels.
And we're going to have to scale up to... very much larger but we're not using that many resources so right now so i have a dgx1 nvidia dgx1 system That's where we do a lot of our processing.
So we do both our training and our processing of images there.
The training took a few weeks to do. And then the processing, because we do this reprocessing of the images many times, it's slower than... just getting a single number out, but we actually get a huge amount of more information out of the process too.
That will continue to scale, and we've shown that we can scale to these big images and do them. you know, the whole sky in a month or something, you know, completely tractable.
So that's exciting. Yeah. Dig in a little bit to your story.
And we're talking with Brent Robertson. He's an astrophysicist.
He serves as associate professor at UC Santa Cruz.
And he's also the Maureen and John Hendricks visiting professor at the Institute for Advanced Study in Princeton, New Jersey.
So before coming to UC Santa Cruz and this fellowship, you're a professor at University of Arizona and you've held a bunch of fellowships at top universities and research organizations before that.
So you've been doing this kind of work for a while in a bunch of places.
Were you always, and I always go to the, as a kid, and it's interesting how many times people say no, But were you interested in this stuff as a kid?
Were you drawn to the skies? Did you know this is a path you wanted to follow?
I was always kind of romantic. So when I was young, I wanted to be a poet or a writer.
I lived in Seattle at the time. I could not get money to... to go to school to do that.
So I thought, well, instead what I'll do is I'll get a scholarship to study physics and astronomy at at University of Washington, and then I'll fool everybody and I'll go back and be a writer after the fact.
But it didn't work out that way. I started liking astronomy.
So then when it was time to apply to grad schools, I applied a bunch of places.
I went to Harvard for grad school. And that's where I started really getting interested in computation, a lot of different problem simulation work.
And when I was at Caltech as a postdoc before I went to To Arizona, I started getting interested in GPUs and GPU programming, and then at Arizona, I really started taking that work kind of to the next level with one of my former students, Evan Schneider, who is now a postdoc fellow, Dr. Evan Schneider at Princeton University in the astrophysics department.
And we do a lot of large-scale simulations on Summit and Titan, these big, large projects.
GPU-enabled supercomputers, but I'm also working in AI with Ryan Housen.
I've got to ask, and this is kind of a, it was a two part question, but it's sort of a third part.
Is there intelligent life out there beyond earth?
Do we have it? I see you laughing at me.
It's okay. Or, and here's kind of the awful twist.
Is there any chance, and I'm not trying to go truther, I'm just, I've always, you know, any chance that we're actually in a simulation? being controlled by something else out there.
And does my just asking that question make you just want to get up and leave?
No, not at all. I think it's completely natural.
And you should be asking the questions that you care about.
You know, astronomy is one of these fields where, you know, we're not curing cancer.
We know that. What we can do, though, is address the big questions that no one else can answer.
You know, I think it's entirely possible, in fact, likely that there's intelligent life elsewhere in the universe.
We don't have any great evidence for that right now.
That does not mean that it's not out there and that we can't look for it.
I'm supportive of things like SETI, the actual search for extraterrestrial intelligence.
You know, we think a bit differently. That's one of the good things that we bring to the table.
We're used to thinking about the universe as this, you know, enormous place.
We're versed in the richness of the universe and the diversity of other worlds.
And it just seems unlikely at this stage that, you know, that things here are unique.
Right. Ballpark figures, how many other galaxies are out there?
How do you quantify these things? How do you talk to somebody like me? who says wow you're this is what you do so you know a billion pixels and they're each this many light years like how much stuff's out there Oh, well, there's an infinite amount of stuff out there.
That's the difficulty. So the universe itself and our models, our way of thinking of the universe is effectively infinite, if not infinite.
But that doesn't mean that we can see out to infinite distances.
And the reason is that light has a finite travel time.
So we can only see so far back into the universe.
We say back because since light has a finite travel time, when you see far away, you're looking into the distant past.
Now the universe is expanding. So the universe that we can see that we've been able to see is a little bit larger than you might otherwise expect.
If you just said the universe is 13.7 billion years old.
And light can travel so far, but since the universe has been expanding, it's actually further than that that we can see out.
And so there's a huge number of galaxies, you know, I would not say countless within that volume, but very, very large, hundreds of billions of galaxies within that volume.
You know, when we look at a part of the sky, like the whole ultra deep field that you mentioned, which is the deepest ever image that we've been able to take that I hope take.
There are only tens of thousands of galaxies in that little tiny patch of the sky because it's a very, very small fraction of the whole sky that we can see.
Right. So one of the great things that LSST will do, and actually there's a new telescope NASA's developing called the Wide Field Infrared Survey Telescope.
Okay. I'm on the science working group for that, working with NASA to help develop that project.
That's a lot like LSST. It'll take wide area images of the sky, but it's as powerful as HST for looking back into the distant past.
And so that will launch sometime in the mid-2020s.
So it'll be a digital space-based image of a large patch of the southern sky.
Wow. And so it's very complimentary to LSST.
It'll go deep over a somewhat smaller area, but still quite wide.
And then we'll also have this huge kind of shallower but still fairly deep image from the ground with LSST.
So why is this interesting and important?
Well, W for since it's in space, the atmosphere doesn't blur out the light.
So you actually get these razor sharp pictures just like HST has.
But now you have it over thousands of square degrees.
The moon is like a square degree in the sky.
So we'll have thousands of times the size of that with the resolution of Hubble.
And that's a huge amount of information for us.
Are you planning for advances in GPU and other compute power that you're going to need them to be able to crunch this data?
Yeah, so that's why we're starting today.
You know, we're building Morpheus with the idea that we're going to apply it to LSST and WFIRST.
And when I go to WFIRST, meetings. I talk about this work, and they're aware of it, and they know NASA's very forward thinking.
That's what NASA's all about. And they understand that advances in computation are actually going to help them leverage this huge investment of billions of dollars into these facilities.
So they're excited about that. How big are these telescopes physically?
So LSST, which is on the ground, is about eight meters across, which is like 24 feet across, a little bit more than that.
WFIRST is a converted spy satellite, and it is about the same size as Hubble Space Telescope has the same size mirror.
But its field of view is very large. And you might wonder, oh, well, you have a large mirror and a large field of view.
If you point those telescopes toward Earth, you can map out a large area of the Earth.
We're just turning one of them around, basically a surplus instrument that the government gave NASA. back at the sky rather than at, you know, our adversaries.
So they're big telescopes. And WFIRST, you know, is the size of Hubble, but has a wider field of view.
So I have to point out, and for the listeners, we're recording this in a- hotel conference room on the, I think second to top floor of this hotel.
And I'm, you're facing me, I'm facing you.
There's a window behind you. As you were speaking, I noticed this beautiful rainbow coming out over the San Jose sky.
I like to look for these signs and things, but as you were talking, I was kind of like, Oh yeah, this is a good stuff.
Well, it's science. I mean, the rainbows are science.
Exactly. So if you look back... on your work to date, work you've done, work you've seen in the field.
Is there anything about the application of AI, deep learning, that kind of stuff in particular that stands out either as surprising to you and or kind of milestone moments that kind of pushed your work forward?
Well, yeah. So GANs are very important. The idea that you could boil down an image to a list of numbers and then regenerate an image.
Yeah. You know, from that list of numbers that actually builds on a technique that we PCA technique that we use in astronomy a lot.
But with these nonlinear functions in the neural net, it's a bit different, but it's similar in this way.
And this is going to be important because one of the things we don't know how to do yet very well that we're actually very interested in doing is applying GANs to the sky.
If we want to take an image of deep space over the whole sky, what does that look like?
Well, we know what small patches of it look like.
And we could actually use, in principle, things like GANs to generate a fake image of the whole sky and use that to train ourselves. on what the real data is going to look like.
So today we could be engineering the kind of analysis tools that we'll need to address W-first data or LSST data by using simulated images that have been created using GANs.
That's something I'm I'm interested in doing soon.
I still need to learn a lot to get there, but I think that the tool set is there today.
Yeah. So we know you're working on the W-First, the LSST, which is plenty as far as I'm concerned, but any other...
Any other big things you're working on? Well, I forgot to mention the James Webb Space Telescope, which is another fantastic instrument.
So James Webb in some ways is the direct successor of Hubble.
So James Webb's a 6.5 meter. So, you know, 20 feet across roughly space telescope. that will peer back to the very earliest times in the universe.
You mentioned reionization, which is at an event that we don't have the time to go into detail.
But it happened during the first billion years of cosmic history.
Okay. JWST, James Webb Space Telescope, will be able to image galaxies at the very earliest times.
Has that been done before? Well, so it's building from the Hubble Ultra Deep Field that we did before, which... found some of the most distant galaxies.
Now we'll be able to find them in abundance with JWST.
So I'm a part of a team that will execute a program that, in the first year that JWST launches to map out a fairly large area of the northern and southern sky, looking for galaxies that existed Only a few hundred million years after the Big Bang.
So this is something that will happen in 2021, roughly.
Okay. And we're tremendously excited. It's a huge national investment of time.
You know, $8 or $9 billion have been used to build JWST.
This is what it's designed to do is this kind of science.
So I'm really excited about that. That's fantastic.
For folks who want to find out a little bit more about these many amazing things you're working on, there are places online you can direct them.
Oh, sure. Well, they can get me on Twitter.
My name's Brant underscore Robertson is my tag on Twitter.
Okay. Also, my website at ucsantacruzrobertson.sites.ucse.edu.
That's my research group's website. I'm the head of the computational astrophysics research group at UC Santa Cruz.
Fantastic. I wish we had more time, frankly.
But, Brant, we appreciate you coming on the show.
Hope you have a wonderful GTC. And obviously, best of luck with finding out more about what's out there.
Hey, thanks so much for having me. I appreciate it.
