Rohit Prasad, Senior Vice President and Head Scientist, Artificial General Intelligence, Amazon Interviewer: Jeremy Kahn, Fortune
Category
🤖
TechTranscript
00:00Rohit, thanks for being here with us.
00:03It's great to be here.
00:04I'm missing all the bad weather on the East Coast, so I'm excited to be here.
00:08Fantastic.
00:09So, big news out of reInvent last week was Nova, as Andrew said.
00:14You finally have a family of large language models out there.
00:17I guess a lot of people are wondering, though, what took you so long?
00:21I thought it was very fast.
00:23So, I think, first, very excited for what we did with Amazon Nova.
00:28Our next generation of foundation models that we introduced to the world last week.
00:33Let me take you a bit behind the scenes of where I sit.
00:37I have the world's best seat to thousands of generative AI applications being built by our internal teams.
00:43And we are noticing three very important things.
00:47One was choice and importance of selection.
00:50So, if you look at Amazon Nova foundation model family, it's not just one model.
00:55We actually introduced six models.
00:57Five of them are generally available as of last week.
01:00And they start with one of the models, which is a text-only model, which is called micro,
01:06by which you mean that you can take any text input and output text.
01:11And then there are three models of varying increases in size and intelligence that are taking multimodal input,
01:18as in image, text, and video, and generating text.
01:22And then you have two additional models that are taking text as input or images and generating images and videos.
01:32Those are called Nova Canvas and Nova Real.
01:35And the reason we did that, because we found internally people were using very different models,
01:40and they had very different needs for their applications.
01:43So, that goes down to the selection of what you need to do to build.
01:48And then, secondly, as people move from experimentation to large-scale loads,
01:53we were finding that cost is becoming increasingly important.
01:57So, these models are 75% less expensive than their counterpart best-performing models on Bedrock,
02:04which is our service where you have many different model providers available with their foundation models.
02:09So, clearly, cost was a big deal and so good to see that the team did massive invention to bring cost down.
02:17And lastly, as you know, with these foundation models, it's not just about the model's capability or cost,
02:23but if you deliver a very good user experience, and after time for the user experience, factors like latency matter.
02:30So, all these models, especially on the ones which take image, video, and text as input and output text, they are lightning fast.
02:38So, those were the things that we worked on, and I'm very excited about the future,
02:42and we are looking to the next generation of Nova as well now.
02:46Great. I think with all these models, one of the big questions people have is data.
02:51Where did the data come from that was used to train Nova?
02:55Yeah, we don't disclose the specifics of the data, but it's a factor of many factors.
02:59One is sources, of course, the publicly available data.
03:03Second, we have a lot of proprietary data, and we also license data from many providers.
03:08Yeah, that's interesting. Licensing seems like it's becoming more of a norm in the building of these models now,
03:14whereas it may not have been in the past.
03:15Yeah, I think you want to trust with the publishers, and everything is super important,
03:20and we want to make sure that you're using the data the right way, and all that is super critical,
03:25and I think we are very excited about some of the partnerships we have made on the data side as well.
03:29Fantastic. So, now Nova's out, but a lot of people are still waiting for an update to what is maybe Amazon's best-known product
03:38when it comes to AI, which is the Alexa digital assistant.
03:42I know you guys said there would be a new Alexa coming, but when is it actually going to arrive?
03:47Yeah, as you know, Jeremy, Alexa is near and dear to me.
03:50I joined Amazon more than a decade back to build Alexa, and it's been a very fulfilling experience.
03:57We are right now in the midst of re-engineering Alexa with the latest of state-of-the-art foundation models
04:04so that it can be the world's best personal assistant.
04:08We are going to be true to that North Star that we set ahead,
04:11and a lot of people scoffed at us when we were looking at that, but that's a reality.
04:15You can have your world's best personal assistant available to everyone,
04:19and what I see in our labs is super exciting, so all I can say today is stay tuned, Jeremy.
04:25So, no timeline?
04:26No timeline right now, just stay tuned.
04:28Okay, well, we'll stay tuned, and I want to come to the audience for questions in a minute,
04:33so please think of your questions.
04:35One of the things with Nova, I mean, you built that family of models in-house,
04:39but you also have this close partnership with Anthropic.
04:42You made investments in them.
04:43You have a strategic relationship with them, and I know they've been helping you on some things.
04:48In some ways, why build your own family of models if you have that relationship with Anthropic,
04:54and they have very capable models of their own?
04:56Yeah. First, we love Anthropic.
04:58The partnership has been great, and I'll go back and reinforce where I started,
05:03what we saw was happening internally.
05:06Customers want choice.
05:08We have seen that many times in our internal Amazon stores,
05:11if you look at how many different selections we have there, and same in AWS.
05:15There's not just one database solution.
05:17There are multiple ones.
05:18Same is turning out to be true in the foundation models.
05:22Our customers do want choice, and the partnership with Anthropic
05:25and what we did with Amazon Nova is just accentuating the choices available for our customers.
05:32The relationship Amazon has with Anthropic is somewhat similar to the relationship Microsoft has with OpenAI.
05:38There have been some reports in that relationship between Microsoft and OpenAI that there's some tension.
05:42What's the relationship like with Anthropic,
05:44and are there tensions in that relationship where they want more compute out of Amazon
05:48than you guys can give them, or what do you feel like you need from them
05:52in terms of innovation on the model side?
05:54I think, as I said, we love the partnership.
05:57I think nothing more to state here than that the partnerships are going great,
06:00and you saw some of that at reInvent last week.
06:03Great.
06:04There's a lot of discussion recently about progress in large language models hitting a wall.
06:09What's your view on that?
06:10I think that's a very debatable question in terms of what wall is coming.
06:15I think we have to be very careful about what we have seen in AI every time.
06:21Every time we come close to a wall, there's a new invention.
06:24I think we'll keep finding that.
06:26If you see what was happening with pre-training,
06:29which is where you build a token predictor,
06:31and post-training where you're giving the model a lot of different tasks,
06:35now it's shifting to inference.
06:37We are spending a lot more time in inference,
06:39and at the same time, as I mentioned with NOVA, we are making inference cheaper,
06:42which means, I think, the scaling curve is changing
06:47from different stages of training to inference,
06:50and we'll continue to see that.
06:51I'm very optimistic that in AI, sometimes every decade,
06:55sometimes every five years, every three years,
06:57you continue to ship the curve up on performance,
07:01and I think that's going to be true again in the coming years.
07:04Are you working on a reasoning model that would use test time inference
07:08the same way as the O1 model from OpenAI does?
07:11Again, I won't go to the specifics of it.
07:14In terms of the many different ways,
07:16what we are really building is for AI to be useful in real world practical applications,
07:22and a lot of that requires reasoning of many different kinds,
07:25not just mathematical reasoning,
07:27but more utilitarian reasoning as well,
07:29where you're trying to do tasks in the real world,
07:32and that's where, if you look at AI agents,
07:35you have to get to the point where they're not just solving a coding problem
07:39or math problem, but they are calling the Uber for you,
07:41or making a reservation for you,
07:44which are not really high on intellectual proficiency,
07:47but require a lot of world knowledge to do it right,
07:50so that's where the field is evolving right now.
07:52And is that where Amazon's heading is towards agents and making agents available?
07:55Yeah, I think if you saw what we talked about with Rufus,
07:59which is a shopping assistant, with Alexa,
08:01which I think of as a super agent because it does many different things,
08:05that's where the world's going to go.
08:07Great. I want to get a question from the audience.
08:09Please raise your hand if you have a question.
08:10If not, I've got lots of questions from Rohit, so keep going.
08:12There's a woman down here. Wait until we can get a mic to you,
08:15if we can get a mic handler.
08:17Just the woman in red, just down here.
08:21And if you could please state your name when you stand up,
08:24and where you're from, that would be great.
08:26Thank you. Hi, how are you?
08:27My name is Lorenza, and I work currently for the press at The Economist.
08:33And, yeah, I would love to, well, we know that Jeff Bezos is back
08:38working at Amazon full-time, and, yeah, I would love to know
08:45whether you've had, you know, work with him in certain projects,
08:49and what is his vision?
08:50Has he mentioned any specific vision that you can share with us on his mind?
08:56I would say you should ask that to Jeff.
08:58But, yes, he's very involved, as he said.
09:00I love the partnership with Jeff.
09:02I've worked with him early days of Alexa, too,
09:04and he's still very much connected with AI elements,
09:07where I spend a lot of time, and I get to see him.
09:10So, very fortunate with that partnership and mentorship.
09:13Great. I want to ask a question about AI's energy demands.
09:16We had an interesting session this morning at a breakfast on this topic.
09:20How is Amazon looking at the huge energy demands
09:23that particularly these foundation models seem to require?
09:26Yeah, I personally, on my front as well, I want AI to be sustainable.
09:31And I love that at Amazon, especially with AWS,
09:34what we are doing is we are looking at the end-to-end lifecycle of sustainable AI.
09:39And it starts with, I think, our chips.
09:42One of the main things where we have our ML accelerant chips,
09:45like Tranium, is essentially to drive better price performance
09:49so that you are able to build these foundation models
09:52in a much better and sustainable fashion.
09:57And specifically, if you look at our announcement last week on Tranium 3,
10:00which is a third generation of Tranium,
10:02that is 40% more price performant than Tranium 2.
10:05So, that kind of step function improvement in price performance
10:11is where I think sustainability will be key.
10:14At the same time, in parallel, in our data centers,
10:18we are trying to use as much renewable energy as possible.
10:21We have partnership with a company where we are absorbing the carbon
10:24that's getting emitted.
10:26All of that is key for making computing more sustainable.
10:29And I think, as a whole, the industry needs to look at it even more carefully.
10:33And is model optimization, because some people worry that,
10:36okay, you make the models more efficient, then people just use more of it.
10:39It's called the Jeevans paradox.
10:41And actually, the overall consumption is higher.
10:43Is that a concern? Are you seeing that?
10:45Also, as you drive down model price, are you seeing usage actually increase?
10:49Yeah, this actually validates what we did with our own models.
10:52If you look at the model family we launched with Amazon Nova,
10:56we are doing a lot of work to bring the capabilities of the large model
11:02into the smaller form factors or smaller sizes through a suite of techniques.
11:07And this is where, if you look at where you're going, Jeremy,
11:10inference is becoming a building block that every application in the world will use.
11:15This is why, if you thought about it as a per-request inference or a per-token request,
11:20each of that has to go down by a factor of factors.
11:24And of late, we have already managed to get an order of magnitude faster
11:27with our inference and more efficient.
11:29But we need to continue doing that work, and I'm very optimistic that we'll do so
11:33because our customers want it.
11:35This is where inference as a building block is such a key element for us
11:38to think about as you're building applications.
11:40Thanks, Ray.
11:42There's a sort of war for talent going on out there.
11:45How is Amazon competing in that war for talent?
11:47How are you able to retain the engineering talent you need?
11:51I think one thing I cherish about Amazon,
11:53that it has the world's most customer-obsessed scientist and ML talent in the world
11:59because here we are looking at what's the best way to solve
12:03and build real practical AI solutions.
12:06So I'm very proud of the team that we already have.
12:09We just today announced our San Francisco AI Hub,
12:13which is very crucial for this.
12:15We've had a presence in Sunnyvale for a long time,
12:18but we said we need to be in the city.
12:20So this is why we're in the city now.
12:22That's why I was also here today.
12:24I think it's great times for AI scientists and AI practitioners.
12:29But the good news for all of you in this room
12:31is that the bar to build with AI has suddenly reduced.
12:34You don't need a PhD in machine learning or mathematics to build with AI.
12:39This is the way where you now can just prompt these systems
12:43or run a lot of experiments where you don't need to be the AI expert.
12:47And, frankly, I believe that more and more work
12:50will be now at the application layer as well
12:52where you'll see that, again, a PhD in machine learning is not required.
12:56If you could tell the audience, a lot of whom are trying to figure out
12:59how to get ROI from AI in their own businesses,
13:02if there was just one piece of advice you could give them
13:04on how to do that, what would it be?
13:06Well, first, two things.
13:08Let's just stick to one. We're going to run out of time.
13:10First, I think we're seeing a phase where
13:14generative AI is moving from experimentation to real production use cases,
13:19which is very positive.
13:21And what I'm seeing is that if you look at our development environment,
13:25we now can save 4,500 hours of SD, software development engineer's time
13:30because you don't need to do a Java migration with a lot of people.
13:34You can simply do it with five people in a room.
13:36That's incredible.
13:37So my last-minute advice for you on that one
13:40would be focus on getting your data right,
13:43your decisions in your daily workflow captured,
13:46because the magic in AI is that once you have the data and the decisions,
13:49it can learn.
13:50If you don't have the data and the decisions, it's not going to work.
13:53So focus on getting the data right.
13:55So get your data right. I'm afraid we're out of time,
13:56but Rohit, thank you so much for being with us.