Raghu Varadhan, Founder of the Modern Theory of Large Deviations, Discusses His Lifelong Interest in the Probability of Rare Events

Born in Chennai (previously Madras), India, Sathamangalam Ranga Iyengar Srinivasa Varadhan FRS, universally known as Raghu, arrived in New York from Calcutta (now Kolkata) in 1963 as a postdoctoral fellow at the Courant Institute of Mathematical Sciences. Remarkably, he has been there ever since. In fact, he and his wife, Vasu, an adjunct professor of media studies at Gallatin School of Individualized Study, have lived in the same university-owned building, less than two blocks away, since 1966.

A towering yet humble figure in the field of mathematics, Dr. Varadhan was recently appointed Interim Director of NYU’s Center for Data Science. Twice, he has served as Director of the Courant Institute, from 1980-84 and again from 1992-94. An expert in and founder of large deviations theory, he was awarded the Abel Prize in 2007, widely regarded as the Nobel Prize of mathematics. In addition, he has been awarded the National Medal of Science (2011), NYU’s Margaret and Herman Sokol Award of the Faculty of Arts and Sciences (1995) and the Leroy P. Steele Prize for Seminal Contribution to Research (1996), among others.

Dr. Varadhan is a member of the National Academy of Sciences, the Norwegian Academy of Science and Letters, the American Academy of Arts and Sciences, the Third World Academy of Sciences, the Institute of Mathematical Statistics, the Royal Society, the Indian Academy of Sciences, the Society for Industrial and Applied Mathematics and the American Mathematical Society.

Quietly possessing an unquenchable enthusiasm for mathematics, Raghu continues to walk to his office in Warren Weaver Hall on Mercer Street daily, eager to work on and hopefully one day solve the many problems that still pique his curiosity.

An interest in large deviations theory was sparked by being in the right place at the right time

From the time of his Ph.D. onward, Raghu Varadhan says, he has been interested in probability, and large deviations is an important aspect of probability. “In 1963 when I came to Courant, a colleague of mine, Monroe Donsker, was doing some work in large deviations, so I got interested in it. That was my first exposure. The subject is fairly large; there are various areas in which the theory can be applied and the techniques are very useful.”

Fortuitously, Professor Donsker had a graduate student named Michael Schilder, who put forth a theory which Raghu found “really novel. I started looking at it, and had a strikingly different point of view that ended up being more useful. That’s what got me started initially in large deviations. And you go to seminars and you listen to people lecture and you talk to others. And then you come across problems that somebody else has worked on and you see avenues that open up that seem very interesting to pursue, and so you pursue that. Sometimes the choices you make are good choices, so it’s a matter of luck. It’s partly chance and being at the right time at the right place.”

The probability of rare events—how likely or unlikely something will happen—is the basis of large deviations theory

According to Dr. Varadhan, large deviations theory tries to understand the causes of strange behavior and interpret them in probabilistic terms. “If something rare happens,” he patiently explains, “something which is not likely to happen but that does occasionally happen, it causes you to ask the question, ‘What could have been the cause of that?’ Also, the same thing that caused that rare event would have caused lots of other strange things to happen. You can’t say precisely what the cause was, but you can say what most likely the cause was.”

Simple, yes? Don’t be so sure. “I started on large deviations when I came here in 1963 and I’m still doing it, so it’s gradual progress, not one moment,” he says.

The interesting problems are the ones you don’t know how to solve at the start

As any mathematician knows, failure is encountered much more often than success, requiring endless amounts of resilience to keep plugging away. But, as Raghu’s achievements attest, perseverance can reap deeply satisfying rewards.

“Very often what happens when you’re working on a large problem is that you reach a dead end—in fact, most of the time,” he says. “If you can see the path already, then the problem is not very interesting. The only problems that are really interesting are those where you don’t know how to get there. Chances are, the first few attempts you make lead to dead ends. But dead ends are not really dead because along the way, you learn things.”

Although you reach many dead ends, Raghu describes it as being in a maze. “Finally when you reach the opening, you’ve not only reached the opening but now you have a full understanding of the maze,” he says. “Sometimes you will look at a problem for years and then one day, the last piece fits in, and the whole picture emerges, and that is wonderful.”

The difference between statistics and probability  

In defining how statistics and probability differ, Dr. Varadhan explains that they are inversions of each other. “In probability,” he says, “you’re rigorously computing your mathematical model, determining what the computations are. In statistics, you have data. You want to see if the data is compatible with the model and how to use the data to predict consequences beyond what you observe. So it goes both ways. Through the data, you say something about the model, and because now you can say something about the model, you can predict what’s going to happen in the future.”

Whereas with probability theory, “you’re calculating probabilities of various events and models—unlike statistics, where you’re testing the model,” he says. “In probability, you take the model for granted. That’s why probability is more mathematics because you’re calculating things based on the model. Although you’re interested in events whose probability is significant, you also want to be able to compare probabilities of events that are relatively small. As it turns out, there is a universal mechanism for measuring how small probabilities are, for estimating them, which works in various contexts. That was basically what I was able to exploit.”

The connection between data and large deviations? Through the model.

Given that large deviations theory estimates the probability of certain rare events, even though these events are very rare, they have not been observed, Raghu asserts. And that’s where data plays an important role. “Basically, the theory extrapolates the model that’s been observed beyond it. That means if you want to really use large deviations in practice from the data, you have to guess what the model is. Then from the model, you calculate these large deviations problems. So data is important in determining the model, because without the model, you can’t do large deviations.”

Why do math? Because it’s fun.

A living testament to the enduring influence of teachers, Dr. Varadhan says that he developed a passion for math at a young age. “We had a very good math teacher in high school who instilled in us the idea that math didn’t have to be work,” he recalls. “You could do it for fun.”

Thank you, Raghu, for having such fun all these years. The Courant Institute, NYU and the world at large are all the richer for it.

 

By ML Ball