[ { "i": 0, "speaker": "Speaker 1", "text": "And I remember my first time, you know, using some of these products and it was just amazing. Um and we felt that very soon the models are going to get to a point where they're going to be much more capable than us. So for coding, it's not a matter of completing a single line." }, { "i": 1, "speaker": "Speaker 1", "text": "The model is going to be able to write an entire function, an entire file, an entire feature, an entire project. Now, I think by the end of this year they're going to be able to build entire businesses." }, { "i": 2, "speaker": "Speaker 1", "text": "And so, you know, we we kind of saw this uh you know, that we call this the exponential and this is just the way that, you know, the models are developing. There's these scaling laws that describe the way that model intelligence goes up over time. Somehow the laws are continuing and in fact it's accelerating." }, { "i": 3, "speaker": "Speaker 1", "text": "And, you know, it's an empirical kind of observation. It's not obvious why it keeps happening. But it seems to keep happening. And so all we did is we kind of traced this line and we're like, all right, right now the model is here, it's going to get here." }, { "i": 4, "speaker": "Speaker 1", "text": "The product that's here is not going to be enough to let people experience the power of the model. And so we wanted something that was purely agentic. And honestly, Claude code didn't work very well for you know, the first 6 months." }, { "i": 5, "speaker": "Speaker 1", "text": "It it actually it took time. It took time to catch up. And it wasn't until I would say Opus 5 in May of 2025 where it really started to work. And, you know, like since then the the growth for, you know, the product has been exponential also." }, { "i": 6, "speaker": "Speaker 1", "text": "It was not at first, but that's when it became exponential. And um we've been finding more and more ways to to use this. Like as the model gets more sophisticated, we figure out more ways to unlock it in the product." }, { "i": 7, "speaker": "Speaker 1", "text": "We launched Claude Code and this is kind of the latest iteration of this. It's long-running asynchronous agent that you interact with like a co-worker. It has great common sense." }, { "i": 8, "speaker": "Speaker 1", "text": "It has a great sense of when to jump in and when not to. And, you know, this will just keep happening. We're going to keep disrupting ourselves." }, { "i": 9, "speaker": "Speaker 2", "text": "So we're going to keep figuring out what is what's the next form factor. >> You know, you can talk about sort of predicting this this this line of progress." }, { "i": 10, "speaker": "Speaker 2", "text": "But to me, how did you how did you think about architecting you know, really multi-agents and how running these multiple agents can actually have us rethink how we implement our workflows." }, { "i": 11, "speaker": "Speaker 2", "text": "I mean, that that is what has been shocking to me is we actually have the opportunity to re-architect, re-engineer how all of us are getting our work done. And I I have to wonder, was this architected into your development or is that sort of just a natural outcome of the capabilities that you were developing?" }, { "i": 12, "speaker": "Speaker 1", "text": "I think it's a little bit of both. If I think back a couple years, how did how did we use models? We you know, like they did this like one-line autocomplete. And and we had agentic workflows, at least what we called agentic workflows." }, { "i": 13, "speaker": "Speaker 1", "text": "But it was essentially these deterministic systems where a step at a time might call out to an LLM to do some bit of computation, but fundamentally the workflow was deterministic." }, { "i": 14, "speaker": "Speaker 1", "text": "And I think what's changed as the model got more intelligent is we've realized actually it's much better to use the model as the coordinator for this workflow. And so what you do is the model drives and you give it tools to pull in context." }, { "i": 15, "speaker": "Speaker 1", "text": "You give it tools to interact with outside world. You ask it to do something and then you let it figure out how to orchestrate these." }, { "i": 16, "speaker": "Speaker 1", "text": "You no longer tell it, \"Okay, you do step one, then step two, and then if not two, then you know, this and then else this and it's just not how it works anymore. You give it a goal." }, { "i": 17, "speaker": "Speaker 1", "text": "You give it the starting point. You give it access to tools and then it will figure it out. And this has happened, I think, as models got more intelligent. This is just overwhelmingly the reason. But I I I think like within Claude Code we've also learned this." }, { "i": 18, "speaker": "Speaker 1", "text": "At the very beginning, we would kind of spoon-feed the model context. And then as the model got more sophisticated, we realized, Okay, maybe actually this isn't it. And we moved away from that a little bit. And, you know, like for example, we have this like file called claw.md." }, { "i": 19, "speaker": "Speaker 1", "text": "It's loaded for every session. And I think more and more what we're seeing with customers is they're actually moving more to skills. And they're moving more to tools and more to MCPs because this way the model has a little bit more control over how to load this in." }, { "i": 20, "speaker": "Speaker 1", "text": ">> Well, I have to ask. I mean, you guys across Anthropic are using Claude every day. I mean, so you're you're customer number one. Like before the rest of us see it." }, { "i": 21, "speaker": "Speaker 1", "text": "Uh, you know, you're you're making sure it it accomplishes what you need. Tell tell us about your your typical day." }, { "i": 22, "speaker": "Speaker 2", "text": "I mean, it it is it is it is is Claude like an extension of your of your body as you're as you're creating your work output? Yeah, this is sort of the weird thing about how Anthropic works." }, { "i": 23, "speaker": "Speaker 2", "text": "At Anthropic, Claude is at the center of everything that we do. It's every single business process, every single bit of everything everyone does throughout the day. Claude is just at the center of it." }, { "i": 24, "speaker": "Speaker 2", "text": "Um, a way I sometimes describe it is um, let's say you're a new hire and you're onboarding at Anthropic." }, { "i": 25, "speaker": "Speaker 1", "text": "At Anthropic, Claude is at the center of everything that we do. It's every single business process, every single bit of everything everyone does throughout the day. Claude is just at the center of it." }, { "i": 26, "speaker": "Speaker 1", "text": "Um, a a way I sometimes describe it is um, let's say you're a new hire and you're onboarding at Anthropic. Typically, if you have a question like, uh, you know, where is the office? Where's the address?" }, { "i": 27, "speaker": "Speaker 1", "text": "You know, like at a you know, most companies you would go and search a wiki or you would ask a co-worker. At Anthropic, you ask Claude. If you have a question like, where's the code base and how do I get access to it?" }, { "i": 28, "speaker": "Speaker 1", "text": "Uh, again, you don't go to a wiki, you go to Claude and you ask Claude. If your question's like, you know, like I took a trip, I have an expense report for it, how do I file that expense report?" }, { "i": 29, "speaker": "Speaker 1", "text": "And this is how it begins. But then, as you kind of get deeper into the work, Claude remains at the center of everything. And so, you know, Claude writes the code, uh, it does the code review, it does the security review, um, it brainstorms and comes up with new product ideas." }, { "i": 30, "speaker": "Speaker 1", "text": "It aggregates user feedback, it triages incidents. Every single part of the engineering SDLC, Claude is at the center of it, too. And you know, like that now now this is happening not just for engineering." }, { "i": 31, "speaker": "Speaker 1", "text": "It's happening for product and for design and for marketing and for GTM for every function. This is what we're seeing and the tools look a little different. You might not be, you know, Claude Code in a terminal." }, { "i": 32, "speaker": "Speaker 1", "text": "It might be, you know, a tool like Coda instead. Um but it's still Claude at the center. And I think this is actually what I see with the most successful adopters of of Claude is you put it just at the center of every process." }, { "i": 33, "speaker": "Speaker 1", "text": "We were talking a little bit before about my favorite Harvard Business Review article from the '90s and it was talking about like I think it was like '92 or '96 or something and it was it was titled something like computers are here, why are we not seeing the productivity benefits?" }, { "i": 34, "speaker": "Speaker 1", "text": "And you know, like this was a big question that people were debating. And I essentially what the case the article made was that there were two kinds of companies. There was one kind of company that they took a computer and they put it somewhere in the corner of" }, { "i": 35, "speaker": "Speaker 1", "text": "the office and then they kept all their, you know, paper and pen processes and all their filing cabinets and everything stayed as is, but now there's a computer in the corner. These companies are not seeing a productivity improvement. And then there's the companies that took the paper, you know, the filing cabinet and threw it away." }, { "i": 36, "speaker": "Speaker 1", "text": "They took the paper and pen and burned it. >> Right. >> And now there's a computer at the center of every process and those are the companies that unlock productivity improvements." }, { "i": 37, "speaker": "Speaker 1", "text": ">> And you know, since the beginning of this year at Anthropic we've seen an 8x increase in code output per engineer. And this is just unheard of in the industry." }, { "i": 38, "speaker": "Speaker 1", "text": "Like you know, typically companies see something like a few percent a year and you know, now our biggest customers that are using Claude code, now they're starting to see something like this. They're seeing like, you know, 50%, 100%, 150% improvement." }, { "i": 39, "speaker": "Speaker 1", "text": "And the thing that we did to get to the, you know, the 8x is we systematically improve a bottleneck at a time. So, you know, first coding is the bottleneck. we throw Claude at it and Claude does the coding. The next bottleneck is the code review." }, { "i": 40, "speaker": "Speaker 1", "text": "And you know, then we have Claude do that. Um the next bottleneck is, you know, like generating materials for GTM. You know, maybe we throw Claude at that. And so, just like one step at a time, we un-bottleneck every part of the process." }, { "i": 41, "speaker": "Speaker 2", "text": "Well, what I love that is because you're driving your development by your own real-world needs. It's a self-reinforcing cycle uh to make sure that of the value of where you're spending your resources. You're an AI-native company, so you're adopting that change at an astonishing rate." }, { "i": 42, "speaker": "Speaker 2", "text": "But, I'd love to hear your experience of how you think about the role of leaders, of engineers. I just think you're ahead of most of the rest of us in starting to see that at play uh and seeing how you're taking people with you." }, { "i": 43, "speaker": "Speaker 1", "text": "Stepping back a little bit, engineering is this thing that it's just always been changing. And you know, as engineers, we're no strangers to this. You know, my grandfather actually programmed punch cards in the Soviet Union." }, { "i": 44, "speaker": "Speaker 1", "text": "Um you know, like I grew up I programmed like basic and assembly growing up, and then, you know, I kind of learned higher-level languages like as I went." }, { "i": 45, "speaker": "Speaker 1", "text": "I worked in JavaScript for a while, and in JavaScript, every engineer knows the frameworks change like every month. There's a new set of frameworks, and everyone has to learn new skills." }, { "i": 46, "speaker": "Speaker 1", "text": "And so, I think like for engineers, the it's always been changing. The languages have always been changing. The frameworks have always been changing. What's happening now is it's accelerating. And so, we went up a level of abstraction." }, { "i": 47, "speaker": "Speaker 1", "text": "We went from, you know, hardware to punch cards, then we went from punch cards to source code. Now, we went from manipulating source code to agents. And now, we're going up a level again from managing agents to managing loops and routines." }, { "i": 48, "speaker": "Speaker 1", "text": "And the crazy thing is that these vast two steps happened in the span of 2 years. It's just faster than it was before, and I think it'll continue to accelerate. And so when I think about the engineers and the people that are really successful, I think it's people that are empirical and curious." }, { "i": 49, "speaker": "Speaker 1", "text": "So, you're able to look at the data and you're able to adjust the approach. You don't always assume the same thing they always do is going to keep working. You're able to kind of take feedback from your work. And so if there's new data, you can have a different approach." }, { "i": 50, "speaker": "Speaker 1", "text": "Um it's people that are autonomous. Because as engineering is no longer the bottleneck, now we're actually bottlenecked on the speed with which we can generate good ideas and bring them to market and you know, do so safely." }, { "i": 51, "speaker": "Speaker 1", "text": "And the people that are really effective at this, they're not people that you know, like they have to have this big process for how to get an idea to market and they need to collaborate with like 10 other teams to do it. They're sort of like one-person armies. It's like the people that are most effective are this kind of CEO archetype." }, { "i": 52, "speaker": "Speaker 1", "text": "you're able to come up with an idea, you're able to talk to users, you're able to look at the data to build, to iterate, and to bring it to the market. And we actually see this coming not just from engineers, but from all sorts of functions." }, { "i": 53, "speaker": "Speaker 1", "text": "Um I think having this engineering background helps today, but it's actually I don't think it's really essential anymore to doing this kind of thing." }, { "i": 54, "speaker": "Speaker 2", "text": ">> But it does change the kind of skills that we need to emphasize. You need to be adaptable. You need to be uh fearless of change. I love what you said. You have to be curious. You have to, you know, really be experimenting." }, { "i": 55, "speaker": "Speaker 2", "text": "I mean, that's part of what I think these new agentic workflows, uh new meaning, you know, like we've said literally in the last how do you be curious? You can experiment now in ways you could never experiment before." }, { "i": 56, "speaker": "Speaker 2", "text": "I mean, your learning cycles are vastly accelerated, aren't they? >> Yeah, totally. And I think it's on you know, it's on leaders to make space." }, { "i": 57, "speaker": "Speaker 1", "text": "If you don't adjust for this and you don't create space, everyone's going to keep doing it the old way and you kind of have to force everyone to do it the new way. >> But really, for a lot of people, it's actually just really exciting to get to try all these tools, to get to experiment." }, { "i": 58, "speaker": "Speaker 1", "text": "And so, I actually see a lot of my role as creating space for people to try stuff out, to feel safe doing it. You know, they're not going to get a bad performance review if they, you know, experiment with a new idea cuz it might actually work." }, { "i": 59, "speaker": "Speaker 1", "text": "Um and my other job is to just give people context. So, to give them business context, product context, so that they can make better decisions." }, { "i": 60, "speaker": "Speaker 2", "text": ">> But companies that aren't AI native, uh these are lessons. Like what you went through, that's going to take uh showing best practice. It's going to be, you know, educating the workforce workforce and really driving culture change uh to be as you described." }, { "i": 61, "speaker": "Speaker 1", "text": ">> Yeah. Yeah, that's right. And our job is to help companies along the way. You know, like uh one thing that I've seen companies go through is this kind of natural transition from not using AI to, you know, every engineer running one agent to every engineer running 10 agents." }, { "i": 62, "speaker": "Speaker 1", "text": "Then 100 agents, then 1,000 agents. And there's sort of these like traits that go with each of these. So, you know, with one agent, uh engineers are using this agent to kind of like write the code and you're kind of focused on it." }, { "i": 63, "speaker": "Speaker 1", "text": "You're still kind of single threaded, you're working on one task. Then at some point, you kind of get to the point where you kind of trust the agent. And you're like, \"Okay,\" maybe it's actually doing the right thing. Maybe while it works, I can start a second one." }, { "i": 64, "speaker": "Speaker 1", "text": "And engineers kind of naturally figure this out. Like, maybe you'll have like multiple checkouts of the same repository or, you know, multiple views of the same code. So, while the agent works, you can, you know, work on something else." }, { "i": 65, "speaker": "Speaker 1", "text": "And maybe you can scale up to like 10 agents this way. What you're doing is you're essentially round-robinning through the agents. You start the work in one, you move on to the second. You do the work there, you move on to the third." }, { "i": 66, "speaker": "Speaker 1", "text": "And this is kind of the workflow that we've now supported really well in the desktop app. Um so, for both Quad Code and for Cowork, you start a session, you just start a second session." }, { "i": 67, "speaker": "Speaker 1", "text": "You can move on. You can have multiple stuff running in parallel. But then it gets really interesting when you scale this up a little bit more. And I think like when I look at most companies, they're still somewhere between step one and two." }, { "i": 68, "speaker": "Speaker 1", "text": ">> And uh when I look at Anthropic, we're probably somewhere around step three on average. So, most engineers are running dozens or hundreds of agents. And um the way you do this is you have the agent run another agent." }, { "i": 69, "speaker": "Speaker 1", "text": "And we have a lot of tools for this. Um So, for example, if you're doing >> Sub-agency." }, { "i": 70, "speaker": "Speaker 2", "text": "You have agents running basically sub-agents. Is that how you think about it? Or We've coined that phrase. If I don't know if it's industry standard." }, { "i": 71, "speaker": "Speaker 1", "text": "No, no, that's exactly it. So, you have the sub-agents and the sub-agents can start sub-agents. So, it can it can actually go quite deep. And for Claude Code, we support up to five layers now" }, { "i": 72, "speaker": "Speaker 1", "text": "of this kind of nesting. And there's different ways to support this on the product side. So, you know, we've been experimenting with for example cloud execution. So, instead of the agent running locally on your computer, it runs in the cloud." }, { "i": 73, "speaker": "Speaker 1", "text": "And uh you can start the agent from the desktop app or from the mobile app. And I actually do a lot of my coding from my phone now. >> I love that. >> And it just runs in the cloud. And this is actually what you scale up." }, { "i": 74, "speaker": "Speaker 1", "text": "And then, you know, we we we're we're continuing to iterate on this to make it even more native and to scale up to thousands of agents. And the way we get there is with dynamic workflows. And" }, { "i": 75, "speaker": "Speaker 1", "text": "this is essentially using cloud to orchestrate just very large teams of agents to do the most complex work. Like large code base migrations." }, { "i": 76, "speaker": "Speaker 1", "text": "Like I think Stripe just used this to migrate um You you use something like like it would have taken months and then it took like four days to do like a 10,000 line uh Scala to Java migration." }, { "i": 77, "speaker": "Speaker 1", "text": "Um there's a lot of companies that are using this for large migrations. Like we just migrated Bun from Zig to Rust. And this is like a JavaScript front end. It's a lot of code." }, { "i": 78, "speaker": "Speaker 1", "text": "And um so for the for these kind of like very very big workflows that would have taken weeks or months or years, essentially a way to do it is this divide and conquer where you give the model a lot of agents and you just tell it like go go figure it out." }, { "i": 79, "speaker": "Speaker 2", "text": "Do this big task. So, it goes back to what we said earlier. Uh it uh it's going to be a journey. It's going to take uh it's going to take education. Uh and it is going to take culture change. And uh when you do it right, uh we actually have similar examples of what you just described." }, { "i": 80, "speaker": "Speaker 2", "text": "We have uh we we have uh created a Rust-based application. Our lead of uh AI software development uh did it over a weekend. Uh for an application that we needed uh we were we we thought it would be a competitive gap." }, { "i": 81, "speaker": "Speaker 2", "text": "Uh and and over a weekend now, we had to validate it and get it through all the, you know, the test regressions and everything where it could be shipped. But that just is unheard of before. I mean uh it it's just astonishing to me uh what multi-agents can do." }, { "i": 82, "speaker": "Speaker 2", "text": "Again, you go back to this projection you had. It sounds like that's exactly what you expected. Though. >> Right? >> I think that this is this is what we expect." }, { "i": 83, "speaker": "Speaker 1", "text": "Yeah. You know, you never you never know exactly how it's going to scale. But if you know, for some reason you know, this is uh this is continuing." }, { "i": 84, "speaker": "Speaker 1", "text": "And you know, like for for these traditional scaling laws, and it it it's funny that the people that wrote the scaling laws paper, like the first few authors, like went and started Anthropic. Cuz I think they knew where it was going." }, { "i": 85, "speaker": "Speaker 1", "text": "They know that safety becomes very important, and security becomes very important. Um and so, you know, traditionally, the the scaling laws say that we scale as a function of the the size of the network, the amount of data, and the amount of compute. And what we're seeing now is it's also a function of the test time compute." }, { "i": 86, "speaker": "Speaker 1", "text": "And you know, like essentially, it's a fancy way of saying how many tokens you throw at it." }, { "i": 87, "speaker": "Speaker 1", "text": "And so from our product design point of view and kind of model training point of view, we want to make it so you can more productively throw more tokens at a problem to get a better result." }, { "i": 88, "speaker": "Speaker 1", "text": "Cuz if you do it naively, you spend more tokens, but you don't get a better result. And so, like one one example of this is effort levels." }, { "i": 89, "speaker": "Speaker 1", "text": "So, like in the model, you can configure the effort. The higher the effort, the more the maximum amount of tokens the model will be willing to spend on that problem." }, { "i": 90, "speaker": "Speaker 1", "text": "And so, this idea of like multi-agent and dynamic workflows, I actually see it as another version of test time compute." }, { "i": 91, "speaker": "Speaker 1", "text": "You throw more tokens at it and by orchestrating the agents and having the model orchestrate it, you can use these tokens productively to to get a better result." }, { "i": 92, "speaker": "Speaker 2", "text": ">> And Boris, I've heard you uh state that it's really important to throw those tokens at it because otherwise you don't even know the roofline." }, { "i": 93, "speaker": "Speaker 1", "text": "Maybe you can share with our our audience what what you mean by that. Yeah, so I I think there's a lot of companies right now that are thinking about ROI. And and how you measure this." }, { "i": 94, "speaker": "Speaker 1", "text": "And I think some companies are actually thinking about it wrong because they're focused purely on the I part of it. And so, they think about, okay, here's my investment. How do I reduce it? How do I cost cut?" }, { "i": 95, "speaker": "Speaker 1", "text": "Obviously, this is important and you should just do this and you know, there's a bunch of tools for this. You can use uh Opus plan mode. You can use advisor models. You can use a lower effort setting." }, { "i": 96, "speaker": "Speaker 1", "text": "You can use a cheaper model. So, maybe everything doesn't need Opus or you know, maybe you can use Haiku. You maybe you can use Sonnet. And of course, you like you write evals to kind of figure this out." }, { "i": 97, "speaker": "Speaker 1", "text": "But I think actually the far more important part is the R. So, how do you improve the return that you're seeing from models? And I think the biggest lesson here that we've seen from successful customers is you want to give engineers the freedom to experiment." }, { "i": 98, "speaker": "Speaker 1", "text": "You want to give everyone the freedom to experiment so that they come up with new use cases. You want people to experiment and you want to create this culture where people feel safe experimenting and they will surprise you. It's not going to be your most senior engineer that comes up with like some brilliant idea." }, { "i": 99, "speaker": "Speaker 1", "text": "It's going to be maybe a new grad. You know, it's not going to be some engineer that figures out how do you automate this marketing process?" }, { "i": 100, "speaker": "Speaker 1", "text": "It might be a marketing person somewhere in the corner of the org that you've never met but that comes up with a brilliant idea and they were because they were experimenting. And so, the thing you want to encourage up front is this experimentation." }, { "i": 101, "speaker": "Speaker 1", "text": "If there's an internal use case and it takes off or there's a product and it takes off and it uses a lot of tokens. But it's successful. That's when you go in and you want to optimize it. But if you don't do the first part, you'll never find opportunity." }, { "i": 102, "speaker": "Speaker 1", "text": "And then once you find opportunity, then you can optimize it like any other engineering problem." }, { "i": 103, "speaker": "Speaker 2", "text": "Boris, I love that description because you've also highlighted is there's a democratization of opportunity. We've actually had junior engineers take a problem a chip design problem that was a bug. Uh we had an engineering team had been working for weeks couldn't solve the bug." }, { "i": 104, "speaker": "Speaker 2", "text": "And one of our junior engineers said, \"I think I can solve that.\" Uh threw a set of genetic processes at it and came and solved this incredibly, you know, rare, you know, 20 things at once had to occur for this uh chip bug to realize itself." }, { "i": 105, "speaker": "Speaker 2", "text": "This was obviously before we shipped the product. This was in our debug and test phase. It actually shocked to the core a number of our senior engineers and that actually drove up adoption of these approaches at AMD because every" }, { "i": 106, "speaker": "Speaker 2", "text": "time you actually have to experience it. Uh you can talk. You can give it lip service. But once you actually have a problem that you couldn't solve or uh actually experience a huge productivity gain, I just leader after leader I see them immediately converted on the spot to believers." }, { "i": 107, "speaker": "Speaker 1", "text": "That's right. And then you need a certain amount of humility also going in. Like I think this is a lesson I had to learn like over and over and over again. I tried to do things my way, which was, you know, before I just kind of gave up and like let Claude drive." }, { "i": 108, "speaker": "Speaker 1", "text": "I kind of had to manually do it. And you know, for example, maybe this is like a year ago. It was like debugging a bug also. And um I remember there was like one time I was debugging something and there was a new person that just joined the team and I was debugging it by hand." }, { "i": 109, "speaker": "Speaker 1", "text": "I was running a you know, a profiler to figure out what's going on. And they just, you know, asked Claude to do the same thing." }, { "i": 110, "speaker": "Speaker 1", "text": "And I was like, \"No, no, no, there's no way Claude won't be able to figure it out.\" >> Even you said that." }, { "i": 111, "speaker": "Speaker 1", "text": ">> Even I said that because, you know, in some ways like I'm biased cuz I've worked with Claude through all the model generations." }, { "i": 112, "speaker": "Speaker 1", "text": "And so, somewhere in my head I was still stuck on this like older model generation, which is just not where the capability was at that point because the model was improving. Um and within 20 minutes they came up with a solution and I didn't." }, { "i": 113, "speaker": "Speaker 1", "text": ">> Wow. >> And they had the right one and I just, you know, learned this lesson over and over. And I think it's like as the model gets more advanced, my relationship with the model changes." }, { "i": 114, "speaker": "Speaker 1", "text": "And it's less that like I have to handhold it and I have to micromanage it. And it becomes more like working with a senior engineer. I trust them, I give them context, I give them goals, I check in." }, { "i": 115, "speaker": "Speaker 1", "text": "The, you know, the less I trust them, maybe the newer they are to the problem, the more often I check in. >> Um but really it's about setting the right guardrails and then letting the model do the work." }, { "i": 116, "speaker": "Speaker 2", "text": "Boris, I have a question for you. I mean, it is a democratization of what teams can do. And so, what what we also see is it um people will uh be running in parallel and we're going to have five ways to drive a uh you know, an improvement out of a workflow." }, { "i": 117, "speaker": "Speaker 2", "text": "How do you think about that? How do you think about measuring and, you know, and how um you know, how how do you at Anthropic in those cases, you know, pick the solution to go with?" }, { "i": 118, "speaker": "Speaker 2", "text": "I mean, what's the role of like benchmarking and measuring uh to be able to in this gigantic area make these decisions and and not end up with uh this, you know, just massive bifurcation of approaches, you know, as we all are trying to solve problems." }, { "i": 119, "speaker": "Speaker 1", "text": "Yeah, so I think it's exactly this. There's actually two tools. One is uh evals, also benchmarks, and then one is vibes. And I think both have their place." }, { "i": 120, "speaker": "Speaker 1", "text": "Um for something like a workflow that you're going to repeat maybe thousands, tens of thousands, hundreds of thousands, millions of times, you definitely want to have evals because the evals let you measure when a new model comes out, you swap it in the harness, and you want to make sure that it improves." }, { "i": 121, "speaker": "Speaker 1", "text": "And this gives you a way to measure it. For other things like maybe a product experience, vibes actually go pretty far because there's a cost to writing evals. You don't want to write it for everything. So you kind of want to like pick and choose." }, { "i": 122, "speaker": "Speaker 1", "text": "When can intuition guide you? When does that tell you something and then when do you actually need to eval to tell you something?" }, { "i": 123, "speaker": "Speaker 1", "text": "And when you have this bifurcation and all these different products, I think one of the blessings of you know like having a model is it's actually very easy to migrate and merge code." }, { "i": 124, "speaker": "Speaker 1", "text": "And so like when you know you pick the approach that you want usually it's quite easy to just ask Claude to migrate all these other call sites to this you know workflow and you know that's that." }, { "i": 125, "speaker": "Speaker 2", "text": ">> I'm going to shift gears yet again a little bit. I want to talk about something that all of us have to focus on and I think it's just actually centered in Anthropic in the approach you have taken and that's the broad topic of trust and verification." }, { "i": 126, "speaker": "Speaker 2", "text": "I had on a previous episode Kathy Pham who lectures at Harvard and we talked about you know governance and how you build in the process." }, { "i": 127, "speaker": "Speaker 2", "text": "So there's one side there's a governance of how to ensure there's safe AI and then there's the verification like how do you ensure that you're driving the accuracy >> [laughter] >> that you need because both come together to provide trust in the deployment of AI." }, { "i": 128, "speaker": "Speaker 2", "text": "I know it's important to you've said it from the outside in our chat here, but maybe we could just spend a minute thinking about like how do you how do you build that in? Like is it a checklist or is it actually built in your own workflows that you run every day?" }, { "i": 129, "speaker": "Speaker 1", "text": "Yeah, I mean it's many layers and exactly what you said for Anthropic this is just at the core of what we do." }, { "i": 130, "speaker": "Speaker 1", "text": "As models get increasingly powerful and they get more and more central to business process, you need it to be safe, you need it to be reliable, you need it to be trustworthy. You need it to be aligned with whatever the intent is of the person that's using the model. So, you know, there's just like so many ways that we approach this at different layers." }, { "i": 131, "speaker": "Speaker 1", "text": "The most basic is model alignment. And you know, we have a very large team working on it. We publish a lot of research about it and we have a lot of blog posts about what we're learning. And this is just many, many years of research go into this." }, { "i": 132, "speaker": "Speaker 1", "text": "And essentially like, you know, there's a lot of things that go into alignment, but you know, for example, an element of alignment is truthfulness. So, you know, the model kind of says the right thing. Another element is not being sycophantic. Because a failure mode for models, for example, is agreeing with everything the user is saying." }, { "i": 133, "speaker": "Speaker 1", "text": "But actually a good model that I like working with will push back. >> Mhm. And I think with some of our later models, so like Opus 4.7, Opus 4.8, we made a lot of progress on this." }, { "i": 134, "speaker": "Speaker 1", "text": "And so, if we train kind of the model well, then if I suggest something and it's a bad idea, the model will push back. And it makes me trust it a little bit more. There's other layers, too." }, { "i": 135, "speaker": "Speaker 1", "text": "So, you know, when we talk about trust, another element of this, you know, there's a lot of stuff wrapped into trust, but another element is security. And so, this is another element that goes into training is, for example, anti-prompt injection training." }, { "i": 136, "speaker": "Speaker 1", "text": "And on every model we publish, there's a system card that talks about how resilient is the model to common attacks like prompt injection." }, { "i": 137, "speaker": "Speaker 1", "text": "And Opus 4.7, Opus 4.8, Fable, these are just the least prompt injectable models in the industry by margin of I think like five or 10x. It's like a really big margin." }, { "i": 138, "speaker": "Speaker 1", "text": ">> So, you had architected that in. I mean, how did you do that? There's a lot of training that goes into it. So, you know, this is just part of the model training process. And we train the models to be resistant to these kind of attacks." }, { "i": 139, "speaker": "Speaker 1", "text": "And especially when you combine it with a runtime classifier for prompt injection, which we actually also do, the success rate in practice now is near zero." }, { "i": 140, "speaker": "Speaker 1", "text": "And obviously this is important because as the models do more, as they interact with more systems, being resistant to these very common attacks becomes very important. As the model gets more capable, we need a way to run the model for a longer period of time safely." }, { "i": 141, "speaker": "Speaker 1", "text": "And you know, now there's workflows that run for days, weeks, months at a time. And obviously this means that there can be a person sitting there." }, { "i": 142, "speaker": "Speaker 1", "text": "And around the same time what we were realizing is if there's a person that's deciding yes or no every time, at some point the person actually just stops reading. And they just say yes, yes, yes, cuz they're tired of pressing yes." }, { "i": 143, "speaker": "Speaker 1", "text": "And I noticed myself doing this, too. You know, I would kind of stop reading the bash commands. I would just say yes. And our security people were also realizing this." }, { "i": 144, "speaker": "Speaker 1", "text": "Um and so the thing that they started working on is there a way to route this to a classifier so the classifier can decide. And you know, they started this work. It took many months to kind of get this to a mature state." }, { "i": 145, "speaker": "Speaker 1", "text": "And in the end through evals, through red teaming, through pen testing, we're able to show that this is actually much safer. And so, you know, at Anthropic we run on auto mode." }, { "i": 146, "speaker": "Speaker 2", "text": "And we recommend this to all our customers, too. >> But Boris, I have to ask you. I mean, you're removing bottlenecks at every stage. But how do you think about scale going forward?" }, { "i": 147, "speaker": "Speaker 2", "text": "I mean, the complexity of what you're taking on at Anthropic just is it just as you say, growing it dramatically at every release." }, { "i": 148, "speaker": "Speaker 1", "text": "Um how do you think about managing scale, actually maintaining that pace? I think a lot of it comes down to following the scaling laws and continuing to reinvent the product as it scales up." }, { "i": 149, "speaker": "Speaker 1", "text": "Um a lot of it is you know about finding awesome partners to work with to you know like support the scale and you know like support this kind of compute. Um and I I I I think a lot of it is figuring out the right guardrails so that we can make sure it's safe cuz you know like again as the model becomes more and more central to what we do." }, { "i": 150, "speaker": "Speaker 1", "text": "Um we have to make sure that it does the right thing. >> But I'm actually Mark I'm curious how you think about it. Like you're no stranger to scaling and you've seen this through multiple generations for CPUs for GPUs." }, { "i": 151, "speaker": "Speaker 2", "text": ">> How how do you think about it? >> Well scaling for us at AMD is actually fundamental because the old Moore's law so Moore's law was that you could double the transistor density double the performance but stay at the" }, { "i": 152, "speaker": "Speaker 2", "text": "same power envelope and cost envelope envelope and that that was fantastic for years. But it's about 10 years ago that started trailing off." }, { "i": 153, "speaker": "Speaker 2", "text": "Not that we don't still with each new semiconductor node get you know more productivity and more density but the costs are going up and as you certainly seen the power goes up. So we're having to reimagine how we optimize to be able to maintain." }, { "i": 154, "speaker": "Speaker 2", "text": "We're basically needing to stay on the old Moore's law pace without the physics of the semiconductors supporting that old Moore's law pace." }, { "i": 155, "speaker": "Speaker 2", "text": "So it's a huge challenge but we're up for it and frankly these AI flows are proving immensely helpful to let us scale because they're helping us manage" }, { "i": 156, "speaker": "Speaker 2", "text": "at a bigger I'll call it state space more variables more of our chip designs that can be optimized and I don't think we could we could continue to scale without these AI workflows. I literally it's coming about at the perfect time." }, { "i": 157, "speaker": "Speaker 2", "text": "It's like a perfect confluence to to enable us to continue to scale. And you talked about partnership." }, { "i": 158, "speaker": "Speaker 2", "text": "I mean, that's at the core of who we are at AMD like you know, our partnership with you, our partnership across the industry." }, { "i": 159, "speaker": "Speaker 2", "text": "That's also fundamentally fundamental for us to to be able to scale because we like to deeply partner to make sure we are using the best like just as the stories you've told us here." }, { "i": 160, "speaker": "Speaker 2", "text": "We want to be first adopters. Of those best practices that you're doing at at Anthropic. So great question for us at AMD for our whole chip industry paramount of innovating to keep at scale." }, { "i": 161, "speaker": "Speaker 2", "text": "I have to ask the question. I mean, you talked about really seen us progress on kind of the line you projected. So I have to ask what do you see coming in the next you know two to three years which in AI time is like an eternity." }, { "i": 162, "speaker": "Speaker 2", "text": "Two to three years is a really long time." }, { "i": 163, "speaker": "Speaker 1", "text": "Let's maybe I'll think about maybe six months. >> Six months. >> This is just otherwise my prediction is just going to be way off somewhere. Of course. I think in six months we are going to see agents running for longer on average. We're going to see people running more agents on average." }, { "i": 164, "speaker": "Speaker 1", "text": "We're going to see them be better aligned with your intent. So it's going to be less course correction, less hand holding. There's going to be a lot more autonomy." }, { "i": 165, "speaker": "Speaker 1", "text": "So I think it'll actually become pretty normal for you know, most people and definitely most engineers and hopefully more than that to be running agents for days or weeks at a time. This will just become a normal thing as opposed to something a few engineers are doing." }, { "i": 166, "speaker": "Speaker 1", "text": "I think by the end of this year we're going to see we're going to start to see Claude building bigger and bigger things. So it won't just be features. It's going to be products. We might start to see entire startups being built by Claude." }, { "i": 167, "speaker": "Speaker 1", "text": "Um so, yeah, progress progress marches on and um you know, we're excited to make it safe and to make it something that's delightful for people to use." }, { "i": 168, "speaker": "Speaker 2", "text": ">> Boris, um thank you very much. Thanks for sharing your insights. Uh thanks for your vision and the impact you're having on the industry. Uh and my god, it's an exciting future." }, { "i": 169, "speaker": "Speaker 1", "text": "Thanks so much." }, { "i": 170, "speaker": "Speaker 2", "text": ">> Thank you, Mark. >> I truly enjoyed chatting with Boris. I think Boris really gave us a preview of how work changes going forward in this agentic era of AI. A few takeaways that really stood out." }, { "i": 171, "speaker": "Speaker 2", "text": "The value of hiring people with non-traditional backgrounds to bring new perspectives and passions. Also, how at Anthropic, Claude is at the center of the employee experience. Agentic AI democratizes the opportunity for anyone at any level to make an impactful difference." }, { "i": 172, "speaker": "Speaker 2", "text": "And lastly, when thinking about return on investment, don't just focus on the investment, focus on the return. The freedom of experimentation that provides will be what comes up with the biggest wins. Thank you for joining us today." }, { "i": 173, "speaker": "Speaker 2", "text": "I look forward to bringing you [music] more in-depth conversations on the cutting edge of technology, industry insights, and visionary perspectives [music] from some of the brightest minds in the industry." }, { "i": 174, "speaker": "Speaker 2", "text": "You can watch the video show on AMD's YouTube channel or you can listen in audio-only platforms, [music] including Spotify, Amazon Music, and YouTube Music." }, { "i": 175, "speaker": "Speaker 2", "text": "Technology is always advancing and I can't wait to share more insights with you next time." } ]