AI as ‘Green Field’ of Opportunity

This term, Richard Ho, head of Hardware for OpenAI was interviewed for the Ronald and Valerie Sugar Distinguished Speaker Series in Advanced Computing, a lecture series at the USC School of Advanced Computing (a new “school within a school” at the USC Viterbi School of Engineering). Prior to being at OpenAI, Ho worked at Google, launched startups and is widely known as a “pioneer in the design of verification tools for chip design.” The conversation, which took place at the Dr. Allen and Charlotte Ginsburg Human-Centered Computation Hall and was moderated by School of Advanced Computing Director Gaurav Sukhatme, has been edited and condensed below:

Gaurav Sukhatme (GS): Richard, welcome, and thank you so much for joining us today.

Richard Ho (RH): Thank you, Gaurav. Thank you, Ron. And thank you all for coming.

GS: You had an incredible journey from Google to OpenAI. How have your experiences shaped your perspective on the future of AI? What lessons might the students here take from your path?

RH: My undergraduate was in microelectronics. I was always interested in computers from a young age. When I went to Stanford for my degree, I was actually interested in artificial intelligence… However, back in the 90s was the start of one of the AI winters where funding for AI research basically went to nothing because the results were not very effective.  I switched back to hardware.

I came back to AI through OpenAI and through hardware, because I think that this is an interesting era where compute infrastructure is powering a lot of capability. For those of you who have been following OpenAI or following AI kind of work, the scaling laws were talked about a few years ago, where the amount of compute you give to train a model, can very much be used to predict how intelligent that model is, how capable that model is, and what it can do. This is really a good tie between hardware and artificial intelligence.

[To students:] Your path may be very circuitous. I think the one thing I discovered is that you always have to be alert to where innovation occurs: just constantly looking for where the synergies between different domain spaces come up. The one piece of advice I would immediately offer is to constantly look for cross-functional domain spaces. AI is opening a lot of capabilities in terms of writing, in terms of coding, in terms of scientific research, and being able to combine that with your own interests and being able to pull that together with your CS or advanced [computing] background, I think is where a lot of innovation is going to occur. I think it’s a super exciting time to be coming into the industry. I’m actually excited for you all, to be honest. I think it’s a great time to be emerging in this space because it’s, it’s a ‘green field’ for possible innovations, a ‘green field’ for new ideas…and being able to come up with new ways of solving problems that just were not possible in the past.

GS: I love the fact that you think of AI as a ‘green field’ opportunity. You led the development of the TPU ecosystem at Google. You now lead hardware at OpenAI. How do you see hardware innovation influencing AI research at a place like USC? What do you think is the right interplay between the kinds of innovation that are happening in industry, particularly on the hardware side? And with research institutions like USC?

RH: I think on the hardware side, we are at a point where we need to have some breakthroughs. If you think about AI hardware, it’s really a combination of three things: compute, memory, and interconnect bandwidth. In terms of compute, areas such as new memory technologies, new material technologies, new communication paths, again with materials and protocols. We’re looking for new ideas and new research for how to get more memory into a small amount of space… In the US, I think, we are very privileged right now. We have access to a lot of this, and it’s pretty low cost, but not everyone in the world has it at the same latency we have it at, at the same rate limits that we have it at, and so I think we need to be able to solve these problems through research in order to expand the infrastructure. I think there’s a lot of room for innovation to go on there.

GS: As you know, USC launched recently our new School of Advanced computing, and one of our founding tenets is that we are interested in innovating in the full stack in computing, all the way from materials and devices onwards up through chip design and layouts, all the way up through the design of software and new algorithms and techniques. It seems like this interplay is going to be increasingly important in the future of AI. Do you agree?

RH: Totally agree. I think that the important thing right now: this co-design. [It] doesn’t just mean hardware and software, but it does mean all the way down the stack, possibly down into the materials, into the way you deliver power, the way you handle the thermal. That goes down to fluid dynamics, goes down to the materials, it goes down to chip packaging technology, it goes down to the design of the computer chips, but it also goes upwards in terms of the software models. Going back to the earlier question about my career path, I was focused on particularly narrow areas of this stack. In my current role, it’s really the full stack. I think a lot of people need to think about it in that way, where these capabilities require you to think beyond just the different disciplines and to start looking at the different disciplines together.

GS: When we were talking at lunch, you mentioned that one of the challenges in a fast-moving field like AI is in every field, you want to vector to what the field is likely to be. You don’t want to anticipate where the ball’s going to be next, not where it is right now. This is particularly difficult in AI. Students ask all the time: “How do I develop the right skills, the right mindsets. What should I prioritize? How do I lead in an AI-driven world? How do you, keep it fresh? How do you invent tomorrow, today?  How do you prepare for AI-driven world if you’re a student?

RH: I think that’s a challenge not only for students, but for us as well. I think that the way I see it is the fundamentals, if you understand the fundamentals of how these AI models actually operate, I think that’s a good starting point. It’s a good foundation. I think beyond that, it’s a lot of just trying it out. A lot of the things that I have seen people being successful at is running with the models, like doing coding with the models, asking the model questions, and then seeing what comes back, and then thinking about how that might solve a problem for you that might be in a different domain space, and being very flexible about it. Don’t think about what you’ve seen in the past but think way beyond. One of the things that we get at OpenAI when we first join on our very first day, we get a book by Isaac Asimov called Profiles of the Future. We’re asked to read the first three chapters of that. The first three chapters really talk about is: let your imagination run wild. Don’t let your current thinking about what is possible be a barrier to what you actually work on and develop. That’s a kind of key philosophy that we have. Aim really high and then see what you can do.

GS: Now is the time to aim even higher.

RH: Yeah, aim even higher. Recognize that the AI agents give you an ability to do the lower -level stuff relatively easily, automate it, and you don’t need to hire other people to do it. You can just go ahead and get an AI agent to do it. Think what you could do with that: be very creative. Think about it as ‘entrepreneurship.’ I’m hoping that there’s going to be a Cambrian explosion of ideas with AI as kind of the fundamental tool that kind of enabled that, and you get a lot more things that are possible. You let the marketplace, and you let the users sort out what’s good, what’s not good, and see that happens. But I think, in my hope, it democratizes a lot of the innovation that’s going to occur. A lot of the startups can kind of be started with a few people and some good ideas.

GS: I love that analogy with the Cambrian exploration.  I think it’s a fantastic way to think about the thought that there’s a real ‘green field’ opportunity here. There’s a new field being made. It’s exploding before our eyes. In a way, the best place to be is to be a student.

One of the questions that comes up a lot is the role of responsibility in technology development inevitably comes up when new technologies arrive. USC just launched the Institute on Ethics and Trust and Computing. It’s a university-wide initiative to think about issues of ethics and trust. It’s broadly centered around computing, but you can imagine that the impetus is largely from the recent developments in AI. So how do companies like OpenAI, how do they approach these challenges? You’re operating at the very forefront of these technologies. You’re releasing them into the world. There are hundreds of millions of people who interact with them the second you release them. So that’s sort of one question is, how do you think about these systemically at a company? And do you see any opportunities for collaboration with academia on this axis?

RH: I think this is one of the most important questions today. How do you make sure that what we build today is ethically sound, is safe, and has all the right guardrails in it? I think that is the most important question. It is one of the most important questions at OpenAI…The way we do it is basically we do have teams who are separated from the frontier model researchers, whose job it is to evaluate models as they’re being created and then evaluate them for safety, both in terms of policy, in terms of copyright, in terms of not guiding people who are in mental health crisis in the wrong way. All of those things are really super important. I think this is becoming super important where you have other agents that are kind of monitoring the output of the frontier agents and being able to guide it and being able to judge, I think that’s a really strong area of possible collaboration [with academia].

GS: Do you have any thoughts on large foundation models? Some of them are evolving into domain-specific systems or people are sort of treating them as true foundation models and then building on top of them. There’s a whole ecosystem that’s sort of being powered there. There’s a lot of emphasis on training these to be multimodal so they can look at text and video and pictures and a bunch of other stuff. Do you have any opinions on which of these directions are more promising than others? Or are we still at the let 1000 flowers bloom and let’s see what happens?

RH: There is a lot of that going on, but I think multimodal is the way to go. I think that we’re heading towards an era of pervasive AI agents, which means you have to be multimodal. I think that the image generation is one aspect of it, but the image understanding is the other part of it…I think that the expectation is that these things are just everywhere. They’re pervasive. They’re in the devices. They’re in the walls and ceilings. They understand you. They know your history… One of the things is that OpenAI, the goal is AGI, artificial general intelligence. For that, the chatbot may be sufficient to demonstrate that, but it’s not the way it would embody in things that would actually interact with people. I think that’s the research path, because it’s the simplest one, but I think that as it kind of rolls out, it will have to embody all the rest of it in terms of manipulation, sensing, producing images, all of those things will be prime research areas.

GS: Specifically, do you see how the trade-off between performance and energy efficiency is shaping? Particularly, is it already having an influence on the next generation of AI hardware design?

RH: That is one of the key focuses of my team, trying to get better performance per joule of energy that’s used in compute. Today’s GPUs, they’re actually quite inefficient at doing their job. The utilization is pretty low. They use up a lot of energy. They’re like a kilowatt each heading towards two kilowatts, three kilowatts. The utilization is super low. I think one of the key focuses is if we want these services to be used in a lot more places, you’ve got to make it more efficient to serve. You’ve got to make the data centers do more, you’ve got to make each individual chip process things better. There’s a lot of work there in order to make the utilization something called MFU, which is the model utilization, the model factor utilization. You’ve got to make the memory much more efficient so you can store stuff and be able to get it out very, very rapidly at low energy. These are key research areas, and they’re research areas that my team was working on a lot. In the next couple of years, there’s a lot of low-hanging fruit. One of the questions I have been asked before is, is this a good use of energy in the world? We’re using so much energy to power these AI systems. Is it a worthwhile use of that? I think that there will be good economic impacts, right? One of the evaluations that OpenAI released was something called GDP Eval, which is really measuring how good are these agents at things that contribute directly to economic value, the GDP of a nation. I think that we will see increasing impact on GDP, and so you use your energy to improve your GDP, I think that’s a worthwhile usage of that energy.

GS: How do you see the role of, particularly at inference time, a lot of compute on the edge? And is there room for architectures that are really hyper-focused on that?

RH: Yeah, it’s a good question. I think that, let me start with the basics, which I think that the inference time, what they call ‘test time compute,’ which is like on the analysis, right? After you get the foundational model, you do a bunch of inferences to figure out possible answers. That’s where the thinking occurs. That’s super important already, and I think that will continue to scale. But I think your question is, the question is very relevant, which is how much of that is in the data centers, and how much of it is on edge devices such as phones and other devices in the home or on your body?  I think that it’s a question of privacy. It’s a question of confidentiality. It’s a question of latency. All of these things will improve if you have it on the edge. But the amount of compute you need for some of these models and the inferences is so high that it’s very hard to imagine an edge device having the battery life and being able to have enough compute power to do this. I think that’s an area where there will be more innovation. So right now, we’re thinking about it in terms of a hybrid: compute on the cloud with something on the edge, do as much on the edge as you can, but everything else that’s really heavy compute has to go back to the cloud because you don’t have enough battery strength to do it on the edge.

 

Published on November 24th, 2025

Last updated on December 11th, 2025