Alumnus Profile: Bill Dally (PhD '86)

Dr. William Dally (PhD '86) became hooked on computers the first time he got to play with one in the 1970s. After studying engineering at Virginia Tech and Stanford, Dally came to Caltech and worked with Professor Charles Seitz on multiprocessors. Currently, Dally is the Chief Scientist and Senior Vice President of Research at NVIDIA.

After earning his PhD at Caltech, Dally joined the faculty of MIT, where he and his team built experimental parallel computer systems and built the foundation for the theory of interconnection networks. Dally later joined the faculty of Stanford, where he developed stream processing, which led to graphics processing unit (GPU) computing. In 2019, Dally was recognized as a Caltech Distinguished Alumnus for his "significant contributions to the architecture of interconnection networks."

ENGenuity: How would you describe your professional contributions?

Bill Dally: There are two main things that I've done professionally in my career. The first is developing the theory of modern interconnection networks, topologies, routing algorithms, and flow control methods. This started with my PhD work at Caltech, where I developed a device called the torus routing chip and wrote a lot about optimal topologies and how to avoid deadlock in them. I've gone on through the years and have expanded that through the many PhD students I've supervised. Now, anybody who builds a supercomputer basically builds it the way that my series of papers tells them to build it. Also, most of the big data center companies do it as well, even though they build it on top of an ethernet substrate. That's been a big contribution to defining how computers communicate with one another.

I also started a project in the late 1990s called stream processing. It was this idea about how to make computing more efficient, at first for graphics and then generalizing to scientific computing. That project nicely morphed into work at NVIDIA developing the NV50, which was launched as the G80 with the CUDA language, and laid the groundwork for GPU computing, which is now central to how most of the world's fastest computers operate and how most of the world's AI gets done. The whole AI revolution has been enabled by GPU computing.

Those are the two big contributions: interconnection networks and GPU computing.

ENGenuity: What inspired you to study computing?

Dally: My father was a professor of mechanical engineering and was a role model to me. Additionally, once I discovered computers, I was fascinated and wanted to understand how they worked and how to make them better. In junior high school, I had the opportunity to play with a computer and it was just the most amazing thing to me. Everybody gets to do that today because now we have personal computers, but this was in the 1970s and computers were big, expensive things. But somebody made one available to me, although it wasn't that easy to use. The interface was via a teletype; you would have to type on this typewriter, and it would clack out your answer and you would have to key in a program and do all your edits very laboriously with a line editor. It was amazing that you could program this thing and it would do exactly what you told it to do. I was hooked the very first time. Shortly after that, microcomputers came out and I wound up convincing somebody to hire me to program microcomputers even though I had no formal education and no training. I wound up having lots of fun playing with computers in the 1970s. When I went and took the courses about them, it all started making sense.

ENGenuity: How has your Caltech education influenced your career?

Dally: What Caltech really did for me is it turned me from being an engineer into being a scientist. Before I got to Caltech, I was a pretty good engineer. I knew how to take some problem and design some machine, usually a computing machine, to solve it. But I didn't really know how to approach the unknown, how to ask questions about what people don't know yet, or how to conduct a study to learn things that aren't yet known. It was really through apprenticeship and mentorship by a lot of smart people at Caltech—the whole ethos of the place is about science and the scientific method. By just being at Caltech and being around a lot of people, I morphed from being an engineer into being a scientist.

ENGenuity: Is there a class or professor at Caltech you remember the most?

Dally: Dick Feynman was at Caltech when I was, and he was a very interesting character. I signed up for this class he had called "the physics of computation." I thought it was an interesting class because it bridged all these things from quantum physics to how computers work; it took a bunch of topics I was very interested in and connected them together in an interesting way. And he had such great insight. He would cut through layers of complexity and have simple explanations for the way things worked.

In terms of professors, my thesis advisor was Chuck Seitz. That's the person who probably had the most influence on me. He was a really smart guy; he really cared a lot about his students. I grew a lot by having a lot of interactions with him.

ENGenuity: What advice would you give to the next generation of Caltech students and alumni?

Dally: Study something you are passionate about and use the knowledge you gain to change the world.

ENGenuity: What is your favorite story?

Dally: Soul of a New Machine. It's written by Tracy Kidder who has written a bunch of books in a similar vein where he delves into how people do their profession. It's a great book; it basically chronicles a bunch of engineers developing a Data General minicomputer in the 1980s. At the time I read the book in 1981, I was working at Bell Labs as part of a design team designing a microprocessor, and it was just so close to what we were doing. I thought this was such a great chronicle of how engineers actually spend their time and the challenges they go through on a difficult problem.

ENGenuity: What is your favorite destination?

Dally: Botswana. I really love nature. I'm a wildlife photographer and I love to go and see unspoiled parts of the earth [view Bill's wildlife photography]. It's really a shame that most places in our country have been spoiled by man; we've taken beautiful nature around us and have paved it. A lot of Africa, and Botswana in particular, is absolutely beautiful because it's unspoiled by man. It's the way it's been for thousands of years before people started really altering the earth. And I hope it stays that way. The animals there are amazing. It turns out the elephants are smart enough to realize they're safe in Botswana because the army of Botswana protects the animals against poachers. Whereas in a lot of other parts of Africa, because there's such a price on ivory from elephant tusks and especially from the rhino horns (even though it's just keratin), animals are being hunted to extinction.

ENGenuity: Is there a project you're most proud of in your career?

Dally: The J-Machine was an experimental parallel computer that I built when I was an assistant professor at MIT. It encompassed an enormous number of new ideas, a lot of which have found their way into other machines over the years. It had a neat interconnection network, it was a three-dimensional torus, and it had interesting packaging to make that network work out well in the physical realm. We built a custom microprocessor that was geared towards rapidly sending and receiving messages and acting on them. So, we could, for example, take a string search problem and decompose it down to the point where two characters were handled on every processor, and we were still getting speed-up going from four characters to two characters. It pioneered a lot of ideas which have then taken 20 years to find their way into other machines. It was also fun because I put together a group of about five or six students and we designed our own microprocessor, we designed our own interconnection network, we built circuit boards, we built our own compilers, and we wrote our own applications. We built a computer from the ground up.

ENGenuity: What gives you the most satisfaction in your work?

Dally: Solving a difficult technical problem is satisfying. When you're pursuing lots and lots of dead ends and suddenly you have this one idea and you pull on the thread and it doesn't just fray, you find something at the end of it. When you solve a problem you've been working on for a couple of weeks or a couple of months, it's a really great feeling.

ENGenuity: What is a typical day like for you?

Dally: No day is typical, and it really depends upon what I'm doing. I'm on President Biden's Council of Advisors for Science and Technology (PCAST), so if it's a day with a PCAST meeting, it's completely different. I'll probably be in Washington; I'll get up and take a walk around the mall and then head into the White House. But if it's a day at home, I'll get up early in the morning and I'll row out on the lake. I usually go out on the lake year-round, even if it's the middle of the winter. I'll have breakfast, and then I'll divide my time between working on some hard technical problem and then meeting with people. I run a research and development organization of about 300 people, so I probably have a dozen half-hour Zoom meetings every day. They often start early because I have people who work for me throughout the world, including Israel and Taiwan.

ENGenuity: What keeps you up at night?

Dally: If there's one problem that I think everybody in the world ought to be staying up at night about, it's climate change. The survivability of our planet needs to be a first-order concern. This is one reason why I donate a lot of my time to sitting on various government advisory boards; I try to push the issue in the right direction. I used to teach a class at Stanford called "green electronics," which covered the nuts and bolts of sustainable energy systems, and several students who took that class wound up starting companies to build things like efficient energy storage systems and better ways of doing photovoltaics to move us to a carbon-free environment. As much as I'm a technical person, I hate to admit that I don't think it's a technical problem right now. I think that if we really put our minds to it, we could become carbon-free in 10 years.

We need to put some urgency behind this. Because of my concern about nature and wildlife, I'm alarmed by the number of species that are going extinct every year. It's a combination of climate change and habitat destruction. I think humans will survive, at least for a little while, but they're going to put a lot of other species out of business in the meantime.

ENGenuity: What gets you out of bed in the morning?

Dally: I'm usually very excited about some technical project that I'm working on – how to make GPUs faster or make our neural network accelerators more efficient. Once I finish my row out on the lake and eat breakfast, I'm eager to just dive into where I left it off the day before.