Hello and welcome to the NVIDIA AI Podcast.
I'm your host, Noah Kravitz. What's in NASA's Mars rovers have in common with the quest to cure cancer?
A lot, as it turns out. Some of the same technology used in the Mars rover missions is now being used to help detect, predict, and treat cancerous cells.
Which technology and how does it work? Here to talk about it is Thomas Fuchs.
Thomas is the Dean of Artificial Intelligence and Human Health at Mount Sinai in New York City.
And he's also the co-founder and chief scientific officer at Page, the first company with an FDA-approved AI tool for cancer diagnosis.
Thomas, welcome to the NVIDIA AI Podcast and thanks so much for joining us.
Thank you so much for having me, Noel. Very much looking forward to the conversation.
Likewise, there's a lot to get into. Let's start with Paige.
What is the company all about and how did it get started?
So PAGE is the leading AI company in pathology.
It was founded in 2017, so we spun it out of Memorial's Lancaster.
And at its core, it does AI in cancer research and especially clinical care. and they are within pathology.
So we look at these digitized pathology slides, we find cancer, classify cancer, predict response to treat, predict outcome, help pathologists to do their job not just faster and better, but especially help them to do things they can't do yet. and then help oncologists to find better treatments for their patients.
PAGE is actually an acronym, it stands for Pathology Artificial Intelligence Guidance Engine.
And that's what we do. And these days, PAGE is used across the globe on four continents.
And we had thousands of patients treated last year already based on a diagnosis that were rendered with the help of PAGE.
So applications of machine learning, deep learning, AI, to use the umbrella term, in medicine and health and wellness have really been exploding over the past few years in particular, and for my money are one of the most interesting and useful. applications of the technology.
We've had some different folks, some folks who are working to try to battle cancer and some other folks doing other things with machine learning on the podcast before.
Let's dive into how Page works a little bit.
My understanding is there's sort of a combination of the page tools and then also third-party applications that are used throughout the diagnosis process.
Maybe you can take us in a little bit deeper.
Yeah, of course. I think it's always good to start with the patient.
So Suppose you have some pain in the chest or somewhere else.
First you get the radiology, and then if the radiologist sees something, the first thing that's usually done is to take a biopsy, so a needle biopsy. at the location.
And then what comes out of there is some tissue And in the last 150 years in which pathology didn't change much, A pathologist looked at the tissue through a microscope and then decided the diagnosis.
So it's very subjective, of course. in many ways.
So what Page does is really help to digitize the whole process and then build AI on top of it from its beginning.
That whole thing is part of the field of computational pathology.
We coined the term over 15 years ago. And since then, the field exploded.
But of course, back then it was very niche.
Right. So... After you have the biopsy, the tissue is then put on microscope slides and these are digitized and end up being enormously large images.
So 100,000 pixels times 100,000 pixels with millions and millions of cells.
So you could fit all your holiday snaps on one of these single flights.
And large institutions produce a lot of them.
So at Bonsaena, we produce over a million of these slides per year.
Then pathologists really have to look for the needle in the haystack If you have, for example, a mastectomies and breast cancer, you could have end up with hundreds of these slides and you're really looking for a few cells or a larger lesion that actually is cancer or not cancer.
And that's a very long and cumbersome process.
And that's exactly where our AI comes in.
So we trained AI at scale to actually find cancer.
And that also led to the first and only FDA approval in pathology you mentioned.
So to do that, you train these very large computer vision models first.
These, of course, are transformer models these days.
And to do so, we digitized enormous amount of slides over the years linked to all kinds of clinical data and pathology report data. to then train models directly from the image against these reports.
To do so, in 2017, we actually built a dedicated compute cluster with NVIDIA DG Access, so the first ones that came out back in the day.
Fantastic. It was, of course, great. And that allowed us to build a model based on 60,000 of these slides for the FDA. approval and it's used clinically.
But these days, of course, that's not enough.
And we build now a very large foundation models from Millions of these slides, and that's done in partnership with Microsoft and Microsoft Research, so we can actually use thousands of GPUs to build this very large computer vision. months.
Since you brought it up, let's talk about that foundation model.
My understanding is it's one of, or perhaps the largest image-based foundation model in the world.
So it's by far the largest in pathology and what we are training now based on 4 million slides is for sure one of the largest publicly announced computer vision models.
What you have to do is actually you have to separate these very large slides into images of normal size, like you would have in ImageNet or other databases, And in that space, we have 50 billion images.
So again, ImageNet is only 14 million, so that's 50 billion images I take a lot of photos as a dad, but to your point earlier, I think that's plenty of room for my holiday snaps.
It depends how many kids you have. Fair point.
But that's remarkable, 50 billion images.
Yeah, so if you count actually the pixels, it's 10 times more data than all of Netflix.
So think of all the shows you watch, all the movies, if you count the pixels throughout.
That's just 10 percent of the image data we pipe through these models.
The whole point of that is, you end up with a foundation model that truly understands microscopic morphology of tissue very well. not only of cancer, but also of normal tissue.
And then like in language models, you get these embeddings of these images And you can use them for all kinds of tasks.
Again, cancer detection, classification, segmentation, We also predict molecular changes, for example, if you have a specific mutation that can be targeted by some therapy. or sometimes we predict outcome to a therapy or to a treatment machine.
This is making me think of a few episodes we've done in the past.
One was to bring up NASA, some astronomers. who were taking – looking at images – or using AI.
I shouldn't say they were looking. They were processing images taken.
I think this was with the James Webb – One of the very powerful telescopes, right?
And another one more recently talking about cardiac care and early detection of cardiac disease using...
Using AI to power the techniques and something that stuck with me was this notion that not only can you know these these models. you know, churn through data with the speed and scale that humans just couldn't do and detect things that humans couldn't with the naked eye But that in some cases, and I think this is specific to the cardiac care episode now that I think about it, they were actually able to detect things that, and forgive my lack of medical here that were sort of in areas and types of things that researchers and medical professionals hadn't even thought to look for before. they were almost hidden in the cells, in the walls of the the heart or what have you.
Is that the kind of thing that you're finding with your research?
And can you speak to that a little bit? Yes, so you're pointing actually at a very, very interesting development.
So some of the models we build, especially also the clinical ones that are FDA approved, that more or less mimic what the human does.
It does visual pattern recognition to, for example, do Gleason grading in prostate or subclassify breast cancer.
And that can be replicated based on the training data of, for example, the pathology reports.
But it gets really interesting when you go beyond that.
Right. I mentioned that example for mutations.
So we have also sequence data of 100,000 patients where we have the slides and then the somatic sequencing of the tumor.
And for some of the mutations, you can actually predict them based on the image.
So the mutations lead to a change of pattern in the image.
And most of these are not recognizable by humans.
So we were not trained to do that. And pathologists are not trained to do that.
Some of them are very obvious, also humans can see, but others not.
There, of course, it also gets more into basic science because then it becomes very interesting in reverse.
Can we find out what the model looked like?
So was it, for example, as you said, some subcellular morphology at the nucleus, or was it larger? vessel structures or a combination of different kinds of cells.
For example, for the immuno-oncology work we do, You want to detect this tumor infiltrating lymphocytes.
Is there inflammation close to the cancer or within the cancer?
And then you have very high level or complicated or non-linear interactions that are actually quite difficult for us humans to understand.
So model introspection becomes something very important to know what's going on.
But philosophically, if you actually spin it a little bit further into the future, the time will come when these AI models might be able to perform very, very well for one of these tasks.
For example, predict if that drug will work for you as a patient or not. we as humans might not be able to understand the intrinsic combinations these models look at.
And then the question is, how can you then still assure that these AIs are safe and effective and equitable?
And that's where the whole FDA framework and all the efforts into testing these models comes in.
I do think that's, I hope actually we're gonna get to that place because I want to go beyond the state of the art.
And there's so much more, of course, in biology and physics which are underlying these processes that produce the tissue than we humans can grasp today, but that can really, really help patients to get better outcomes. full disclosure, I come from a liberal arts background, not a science background per se, but all of the talk about all the scientific progress that these technologies are helping helping further and talk of new discoveries is what really gets me excited and interested in where this stuff is headed.
I want to come back to the technical aspects of this in a moment, but I want to ask you, Where is Page at now in terms of results?
Like what are you able to do? What sort of, I don't know if success rate is the proper way to phrase it, But what kinds of actual on-the-ground results are you seeing in using the tech?
I think a good place to start is actually the FDA study we ran.
And there we could show that the error rate was reduced by 70%.
So pathologists find much more cancer than they would have before without increase of false positives.
So that's very important. Okay, yeah, yeah.
And we also showed across the globe on 14,000 patients out of 800 different institutions out of 45 countries, that the AI really works everywhere across the globe.
That's because at that scale that we train, It really focuses on the morphology.
It's really on the growth patterns that it's not thrown off by all kinds of nuisance variables or bubbles or air or whatever. all kinds of things you see on these slides.
And that has been replicated at Yale, at Cornell, in Brazil, in Europe.
Everywhere across the world. That's great.
So what's, of course, interesting here is that if you think about FDA approvals of AI, you have hundreds of AIs approved in radio.
I think they're up to 500 by now and in pathology you still have only one There's just one that is actually safe and effective.
And that's, of course, a huge difference.
And one reason is that the FDA looks differently at the two domains, radiology to a large degree is a screening step.
You can always go back and get another x-ray or another mammography, but in pathology, it analyzed the diagnosis and you don't have cancer until the pathologist says so.
So it's much more dramatic in the treatment pathway.
But it also shows that there's that huge chasm between All that hype we have in AI, 15,000 biotech AI companies and all that fluff and everything. and then the reality in the ground.
So if you have a session in the trenches and you want to use something that's safe and effective, you have the choice of one.
And that's, of course, something we have to change.
So Paige does that drastically and actually expanding to all kinds of different cancer types.
So we have, for example, a breakthrough designation for the breast system.
We can do bladder, lymph nodes, and so forth.
And then we use these very large foundation models we discussed, especially the new one, It's actually named after the founder of pathology, Rudolf Virchow.
So it's the Virchow 1 model, the V1 model. and V2 will follow soon, and we can actually use it for pan-cancer applications.
So instead of just doing what we just described for prostate cancer of breast, you can address nearly all cancers, and 50% of cancers are rare cancers.
So if you inflict a cancer, it's a coin toss.
If you have a big one where there are a lot of treatment options or you have one of these rare ones where there's nothing available.
And the foundation models allow us, because they understand morphology in the body of tissue so well, that you can hit the ground running with few-shot learning on rarer cancers and rarer conditions, and then also produce AI for these.
And that's very, very promising. Absolutely.
You sort of touched on this, mentioning the few-shot learning, but are there specific... types of cancers, and please replace types of cancers with a more precise term, but are there specific...
You know, variants that are more difficult to detect because of technical problems.
And how are you trying to get around some of those?
Yes, so there are usually two reasons. First off, they might be rare.
So you have the big ones like breast, prostate, lung, derm and so forth.
And then rarer ones like cholangiocarcinoma, so that's bile duct, which is...
Terrible. And they usually didn't have the numbers to actually build robust systems that generalize.
And that's what the foundation model is changing.
And then you also have within the large cancers, you have subtypes that can be very rare and difficult to diagnose.
We just published results in breast cancer that the foundation model now can actually really find these super rare forms as well. which you wouldn't do with a normal breast AI.
And that actually shows that it's worthwhile, building these very large systems, although of course they are tremendously expensive, just the production of the data that go into these systems is very cumbersome. but it really leads to the benefits for the patients at the end of the day.
Amazing. And a question that popped into my mind earlier, are your system, your techniques sort of backwards compatible with older you know lower resolution or not necessarily older but just kind of lower resolution images or is it all built atop kind of a um I don't know, proprietary, but sort of current, very high-res way of scanning for cancer cells.
So that's a good question. So most scanners that are out there, the high throughput scanners, either have a 200 times magnification with 20x or 440x and we can work with both.
On the cancer detection side, so we could show that it's equivalent.
Sometimes if you have very specific tasks that are really important tailored towards, I don't know, the membrane of the nucleus, then of course you want to have higher resolution.
So it depends on the task. I'm speaking with Thomas Fuchs.
Thomas is the Dean of Artificial Intelligence and Human Health at Mount Sinai Hospital in New York City.
And he's also the co-founder and chief scientific officer at PAGE, as we've been discussing.
Page is an AI company focused on pathology, and they are the first company with an FDA-approved AI tool for cancer diagnosis.
Thomas, I'm going to shift gears a little bit, kind of jog your memory back to something we teased in the opener.
You helped out with some of the technologies in the Mars rover project.
I think, was it Perseverance in particular?
Yeah, so the techniques I used today actually for Percy and Curiosity before, the images we worked with were on the previous rovers.
And it was the Athena class, so Spirit and Opportunity. and then also from orbit.
So I had the enormous privilege to work for JPL for NASA a few years.
JPL, the Jet Propulsion Lab? Yes. Jet Propulsion Lab, who is of course handling the mass rovers.
And what we did is we, for example, used the imagery from the navigation cameras to differentiate sand from gravel and from other terrain types so they don't get stuck, right?
So spirit got stuck in this powdery side.
Oh, right, right. Yeah, yeah. And you want to, for example, omit that and they shouldn't hit obstacles and so forth.
And We also had imagery from the Mars Reconnaissance Orbiter, which is the satellite around Mars.
And the images it takes off the ground actually have a quite good resolution, so 30 centimeters per pixel in the best case.
And they are nearly exactly the same size as these histology slides, so they're really enormously large.
And you also try to classify different terrain type to, for example, look for good landing spots for the next mission and so forth.
And yeah, the beauty of machine learning and the eyes is it from the microscopic to the macroscopic level.
And so with the rovers, how does it work sort of training and updating the model and then running inference from that distance?
Were these, you know, the systems all on the rovers and you would push an update out or i'm getting a little over my head here but how did that work kind of you know doing all this work to help real-time navigation but from so far away Right, so most of the research we did back in the time was, of course, more basic research. show that capability or again do it from orbit for all kinds of purposes like looking for landing sites.
One of the big problems of course in space exploration is that as soon as you send something out there, it has to be radiation hardened.
And that means the machines are usually 10 years behind what you actually have currently.
So there's no GPU out there. I worked on another project where we did actually computer vision for flyby missions at asteroids or, for example, Pluto. where the light gap is so big that you can't send meaningful messages comments out there and then it really has to, for example, find the asteroid to take pictures.
And there also JPL had worked on building specific FPGAs and ASICs that could do that in the future.
But that's certainly one limitation that you can't use the...
Maybe that's something for NVIDIA, right?
Have radiation-hardened GPUs for the next generation.
No, you read my mind. I was thinking about the...
Special edition 40 series radiation hardened.
So in addition to the work you're doing now with PAGE and Mount Sinai and the JPL stuff, You've had a hand in at least a couple of other really interesting and sort of large-profile, high-profile projects.
The Memorial Sloan Kettering AI research department.
And then also there's a supercomputer in New Jersey, I believe.
Yes. Tell us a little bit about those. Yeah, sure.
So that was the one I was referring to we built in 2017, 18, specifically for Page.
So it's still the largest standalone supercomputer only for pathology.
And we extended it over the years. But it's, of course, not big enough to train this humongous foundation model.
That's why we needed a relationship with Microsoft, not only for compute, because of course they're fabulous engineers and scientists, and it's really a great partnership to drive that forward on modern hardware to build that.
Because of course, we are always limited with compute.
Yeah. but also at Mount Sinai, of course.
There's so much fantastic research that could be drastically accelerated if the research community had access to more compute.
But it was, of course, a very interesting exercise.
We wrote papers and they were published in Nature Medicine and well cited and so forth.
But the secret behind all of that is, of course, engineering.
It's MLOps. How do you actually build a stack?
So usually in that case, the problem are not even the GPUs, but it's IO.
How can you pipe petabytes of data? quickly enough to the GPUs to actually train these very large models.
And so there's an enormous software stack to build computer vision models at scale. and the enormous experience that Page has in doing these applications that are, of course, broadly applicable in all kinds of computer vision tasks now, of course, we do multimodal approaches with text and radiology and so forth.
Your own background is originally in engineering.
Yeah. So I had my undergrad was in mathematics and in Austria. which you can hear from my Arnold Schwarzenegger accent.
Sure. After that, I did a PhD in machine learning at a time where it wasn't cool yet.
Yeah, yeah. NeurIPS was 400 people, and it was very niche, and...
Combining of course that AI with pathology was even more niche back then.
And I was going to ask, how did you find your way into working in pathology?
Yeah, so that was at the beginning of my PhD one day. pathologists of the hospital we worked with came into the lab and just wanted some high level statistics based on the Excel sheets, but they showed us a few images.
And as a naive student, I saw these and said, oh, these are just roundish objects.
These cells should be easy to detect. And now we are 20 years later and we still can't do it.
So it shows how to do it. But at least I stuck with it and we can do it quite well now.
Amazing. These origin stories are the best.
So many of them are so similar to that. As a student working on this, somebody wandered into the lab, I was the only person there.
Twenty years later, I'm working with Microsoft to build the world's largest foundational image model to fight cancer.
Amazing. A lot of times we like to end these shows in kind of a forward-looking note.
I'm going to give you a two-part thing here.
The second part, I want to kind of open it up, given your background and you know, your hands-on engineering prowess, as well as obviously all the things you're doing in the real world to ask you about AI more broadly.
But before that... Paige, your work at Mount Sinai, you talked about this a little bit, but to kind of put a point on it, Where do you see the future of AI in medicine and healthcare?
And we can narrow it down specifically to using AI in pathology.
Where do you see things headed? What are the big obstacles beyond, of course, everybody needs more compute? the big hurdles that you're looking to get over near-term and take it out as far as you want, as far as what you see in the future.
No, of course. Let's start with pathology.
So pathology has a very unique challenge in that for the last 150 years, it really didn't change.
So firstly, has to go digital before it's completely AI-based.
And so part of what AI does, sorry, what Page does, it also helps large labs and academic institutions to actually go digital.
So get these scanners, link it to your LIS, make sure everything flows and then it's a SaaS service in the cloud so you can access it from everywhere and you can diagnose from everywhere and so forth.
So that's something that has to be driven in pathology.
Another part is, of course, it's... important that the FDA also, as they want to do now, take oversight.
There are some loopholes, some other companies who don't validate that level of scrutiny I mentioned can use and these LDT loopholes should be closed.
So in healthcare much broader, and that's what brings us to Mount Sinai, which is, of course, a very unique place.
It's the largest health system in New York City. with eight hospitals and 200 locations and so forth.
There we are trying to more or less replicate what we did in pathology for all these different areas. really level up in cardiology, in radiology, in psychiatry.
And AI now plays a role in all these areas. from basic research in drug discovery and drug development to the bad side.
So Mount Sinai has more than 20 models that are actually used in the hospital So for example, direct the care teams to the right patients who are at risk for seizure or malnutrition or a full of other things.
And then on the basic research side, there's work in cardiology based on EKGs and ECGs to actually early predict all kinds of cardiology issues or an ophthalmology where we look at fundus images to try to even predict neurodegenerative diseases from that.
One is also building a neuropathology AI center where 10,000 brains are going to be scanned.
That's for work in Parkinson's and in Alzheimer's.
Amazing. In all these areas, AI is a key to provide capabilities we don't have yet, be it care or be it research.
Besides all these Doomsday scares that AI is an existential threat for us and so forth.
I think especially in healthcare, we have the moral obligation to develop AI as fast and as safe as we can.
Because patients are dying not because of AI, but because of the lack of AI.
In the US alone, 800 patients die or are permanently disabled because of diagnostic errors.
And as we just discussed, for example, in pathology, as we proved out, these are things that can be addressed to a large degree with AI.
You have to build the tools for the physicians to give better treatment.
Any slowdown there is certainly not responsible.
Now I'm going to ask the old school... engineering PhD and you looking at the current, looking ahead a little bit when it comes to the field. and the technical everything going on technically and it's been a What's the word?
It's been a whiz bag. There's a better word.
But it's been a whiz bag a couple of years in particular. at least as far as public exposure to everything going on.
But I've had the chance to talk to a bunch of engineering leaders who've just been saying, I've Never seen this kind of rate of acceleration in tech in my career anyway.
What are you excited about? What kinds of, you know, things on the technical ground level are you looking forward to, you know, in the next, I don't know, year, two years, five years?
On the very mundane side, it's just to get these tools into the clinic.
That's the first thing, huge hurdle. And usually it's not capabilities.
It's not regulation. Usually it's reimbursement.
Sure. very petty monetary issues where health systems are hard to change.
So on the AI side, of course, a lot of the current excitement is driven by language models.
And that's certainly for some reason, of course, for good reasons, we humans over-index language.
Yes. Everybody who can string five sentences together in a coherent way seems to be intelligent.
And that's why we certainly assign a lot of of capability or properties to large language models that might or might not be there.
But at the end of the day, language is of course a human technology.
It's produced by our brain. So our brain, our relatively feeble, wet brain with its few neurons can produce language and reason about language.
But if you go beyond that, for example, in biology and cancer research, and if you look at the issues there, just at a single cell, all these processes are produced by physics and by biology.
Just if you think of all the proteins that are at play at a single moment in a single cell that is in complexity so far beyond language.
That there's a whole universe out there, literally, our physical universe and then the biological one. that goes far beyond language.
You see it even now in pathology, the large models we built, we touched upon that, We humans are missing the vocabulary to even describe it in language.
And then usually we come up with all kinds of comparisons where these cells look like people in single file or these cells look like this.
But there's so much more going on. We start to capture with AI, be it image-based, be it genomic-based, and so forth, that goes beyond our capabilities with language.
And I think that space is going to be dramatically exciting because that will deliver new drugs better care, better outcome.
It's an exciting time to be alive to see the transformation.
Amazing. Thomas, for listeners, While they ponder all of this, who want to learn more about what Paige is doing, maybe what's going on at Mount Sinai and the other things we touched upon, Are there some resources on the web where you would direct folks to go?
Of course, for page, just go to page.ai, P-A-I-G-E.ai, And at Mount Sinai, if you look for Mount Sinai AI, you see our initiatives there and the AI centers.
And if you want to go beyond pathology, that's, of course, the place to go.
Excellent. Thomas, this has been an absolute pleasure.
Thanks so much for making the time. There's a lot to chew on here and a lot to be optimistic about.
And so We appreciate all the work you and your teams are doing and for taking a few minutes to tell us about it.
Thank you so much, Noah. It was really a pleasure.
Thank you for the great questions. Thank you.
