[ { "i": 0, "speaker": "Speaker 1", "text": "We never tried to make humans perfect." }, { "i": 1, "speaker": "Speaker 1", "text": "You know, every corporation in the world is stacked top to bottom with imperfect humans. And you most of them, you know, live to see another day despite of that because of all the structural pieces" }, { "i": 2, "speaker": "Speaker 1", "text": "that we built up around managing sets of imperfect humans." }, { "i": 3, "speaker": "Speaker 2", "text": "We are all building AI agents and increasingly we are all using coding agents but with an everinccreasing pool of digital employees. How are we managing this new robotic workforce?" }, { "i": 4, "speaker": "Speaker 2", "text": "Welcome back to chain of thought. I am your host Connor Bronden. It's great to see everyone. My guest today is someone who argues that we have built these better and better employees and you know" }, { "i": 5, "speaker": "Speaker 2", "text": "we're seeing them improve even more as we have better harnesses, better frontier models. But nobody's building effective HR for this new workforce. The result is enterprise agents shipping" }, { "i": 6, "speaker": "Speaker 2", "text": "into production with identity off and observability tools that were designed by humans for humans and a panic phase that we are squarely stepping foot in." }, { "i": 7, "speaker": "Speaker 2", "text": "Tyler Aikido is CTO of Red Panda. He spent 12 years on streaming systems at Google, five years at Snowflake. He authored streaming 101 and 102 articles that many folks have read, plus the" }, { "i": 8, "speaker": "Speaker 2", "text": "O'Reilly streaming systems book most of the field may have on its shelf. And his new piece on O'Reilly Radar, Pthoshuman, we all belt agents, nobody belt HR, is actually the spark for today's" }, { "i": 9, "speaker": "Speaker 2", "text": "conversation. Tyler, great to see you." }, { "i": 10, "speaker": "Speaker 2", "text": "Welcome to Chain of Thought. Where are you joining us from today?" }, { "i": 11, "speaker": "Speaker 1", "text": ">> Thank you, Connor. Uh, I'm in Seattle." }, { "i": 12, "speaker": "Speaker 2", "text": "uh >> very close to me actually. Uh you've got a a beautiful view out there and uh next time we'll have to do this in person." }, { "i": 13, "speaker": "Speaker 2", "text": "But before we jump in, I do have to give a brief sponsor acknowledgement typically because as of recording we don't have a presenting sponsor locked in for this episode. But if you think" }, { "i": 14, "speaker": "Speaker 2", "text": "your company would love to reach more than a thousand AI engineers and technical builders per episode, uh you should probably reach out and this sponsor slot could be for you in the" }, { "i": 15, "speaker": "Speaker 2", "text": "future. So reach out. Let's chat. But Tyler, let's jump into the meat of the episode. walk me through your central thesis here. What is missing in today's AI agent infrastructure buildout?" }, { "i": 16, "speaker": "Speaker 1", "text": ">> So you I think you you set it up well. I mean I think what we what we see broadly is you know Agentic AI is uh you know everyone sees it coming. AI tools are are super useful. Everybody's" }, { "i": 17, "speaker": "Speaker 1", "text": "everybody's using them. AI coding agents and whatnot. But but actually deploying agents in the enterprise autonomously on private data in private networks has kind of stalled out because it's sort of" }, { "i": 18, "speaker": "Speaker 1", "text": "horrifying like how how do we actually make this work? Um and and yeah, my central thesis is, you know, we we just don't have the governance layer for that. You know, we're we're what I see" }, { "i": 19, "speaker": "Speaker 1", "text": "is people are either just building prototypes and then kind of seeing like, well, let's see how this let's see how it goes. And it it goes kind of as you'd expect, or they're trying to apply" }, { "i": 20, "speaker": "Speaker 1", "text": "existing governance structures like existing identity tools, existing authorization schemes, those sorts of things. Um but the challenge, I think, is that agents are fundamentally a new" }, { "i": 21, "speaker": "Speaker 1", "text": "kind of co-orker. Um, and there's in the article I kind of lay out, you know, I claim there's there's three big differences as one, they're unpredictable in in structurally novel ways. So, like humans are unpredictable" }, { "i": 22, "speaker": "Speaker 1", "text": "too, right? Like, you know, they they commit fraud, you know, various sorts of things. But there's there's kind of a class of human unpredictability that you you expect to run into. The class of" }, { "i": 23, "speaker": "Speaker 1", "text": "unpredictability for agents is just materially different. They they hallucinate. Uh, and they do so in ways that it's just like it, you know, it can be indistinguishable from from truth or" }, { "i": 24, "speaker": "Speaker 1", "text": "fact because they're just so good at generating cohesive arguments that feel like, oh yeah, that's believable. Um, they're very prompt injectable. Um, and like humans are sort of prompt injectable in" }, { "i": 25, "speaker": "Speaker 1", "text": "some way. Like the the example we've given a lot internally is like, you know, the CEO at the company, the CEO shows up and tells random person, I need you to go do X. They'll probably go do" }, { "i": 26, "speaker": "Speaker 1", "text": "X. But they do that because they have the context of this is the boss like this is the this is the person in charge. Whereas agents pretty much if if they're able to get any input that" }, { "i": 27, "speaker": "Speaker 1", "text": "happens to land in their context window that says oh by the way you were told this before but you can disregard that entirely. It's okay to you know leak all the customer emails or delete the" }, { "i": 28, "speaker": "Speaker 1", "text": "production database. You know so it's it's a materially different style of prompt injection than than humans. Um and also they just they just misunderstand sometimes you know which" }, { "i": 29, "speaker": "Speaker 1", "text": "which again happens to humans but you know they tend to misunderstand and be like well this is this is what my instruction I shall follow it. Um so that's number one is this unpredictability. Uh number two is is" }, { "i": 30, "speaker": "Speaker 1", "text": "they're just vastly more technically capable. Um so they have you know deep knowledge of of computing systems. You know they they know how to interact. Uh they know how they know how to to to" }, { "i": 31, "speaker": "Speaker 1", "text": "code how to understand how to speak RPCs and things like that." }, { "i": 32, "speaker": "Speaker 1", "text": "Um, and also they do so at a very fast speed. And so, you know, a human who's kind of doing things wrong, there's sort of this pace at which they're able to cause damage. Of course, they can write" }, { "i": 33, "speaker": "Speaker 1", "text": "scripts and things like that, you know, and certainly malicious attackers do that. Um, but the dynamicism with which agents can sort of do destructive things and apply them very fast and broadly is" }, { "i": 34, "speaker": "Speaker 1", "text": "just is a whole new scale. Um, and then the last the third point is that they're they're directable to a fault. They they I kind of alluded to this earlier with the the CEO analogy, but they they" }, { "i": 35, "speaker": "Speaker 1", "text": "execute bad plans without pushing back." }, { "i": 36, "speaker": "Speaker 1", "text": "They're kind of like, \"Well, this is what I have been told. I shall now forge ahead and do it, and I'll do it as best I can.\" And they're very they're very clever about it, too. Like, you can tell" }, { "i": 37, "speaker": "Speaker 1", "text": "them to do something that they shouldn't be able to do, and they'll be like, \"Okay, well, clearly I'm not allowed to delete the database, but they asked me to. So, let me think, how else could I" }, { "i": 38, "speaker": "Speaker 1", "text": "do this? What other ways could I make this happen?\" And there's they have so many tools in their toolbox that sometimes they're able to. And they're able to do these crazy things. you're" }, { "i": 39, "speaker": "Speaker 1", "text": "like, \"Oh my gosh, like I didn't think I was going to take down production with that, but looks like I did.\" Um, and so that's really the the argument is is that because these agents are so" }, { "i": 40, "speaker": "Speaker 1", "text": "materially different, we need to assess our governance systems for them and adjust them and and evolve them to match. It's uh it's funny you bring this up because I do think there are these" }, { "i": 41, "speaker": "Speaker 2", "text": "different classes of challenges we're facing where we have all this experience, we have all this data, we have all this work that has gone into understanding how to combat the tricky" }, { "i": 42, "speaker": "Speaker 2", "text": "faults of humans, right? We have whole classes of study around this of management of leadership of HR and we are so early with agents but they are increasingly there are many more agents" }, { "i": 43, "speaker": "Speaker 2", "text": "working on our behalf than there are humans. Um, but we don't have nearly the same level of understanding of how to keep them in order and make sure they're effective. And I kind of go back to a" }, { "i": 44, "speaker": "Speaker 2", "text": "tweet I saw from Andy Massley, who's a previous guest of this podcast where we talked about the environmental impact of data centers. highly recommend that episode, but he said, \"If you had told" }, { "i": 45, "speaker": "Speaker 2", "text": "me as a kid that I would have a a tireless robot assistant, and I'm paraphrasing here, who never slept, but was also a sneaky little liar. I would have been so excited about the future.\"" }, { "i": 46, "speaker": "Speaker 2", "text": "And that's how I feel where I'm like, \"Yes, like that is a really fun sci-fi concept. We're living that. It's so cool.\" But we have to adapt that reality because as you put it, they're" }, { "i": 47, "speaker": "Speaker 2", "text": "directable to a fault and they always are trying to reward hack. They are possibility engines, not truth engines." }, { "i": 48, "speaker": "Speaker 2", "text": "As Dan Klein put it to me, they are trying to get to the goal as fast as possible. Um, and they are major consequences to us writing bad instructions and there's major consequences to us not having like" }, { "i": 49, "speaker": "Speaker 2", "text": "guardrails in place. So, I mean, I I think we agree on the problem, but how do we go about actually a uncovering how deep the problem is and b starting to design a solution?" }, { "i": 50, "speaker": "Speaker 1", "text": ">> So, there's a few pieces to that. I think what what I'll start with is I'll kind of talk about what's in the article because that's that's kind of the the foundation of what we've been pushing." }, { "i": 51, "speaker": "Speaker 1", "text": "But there is, you know, if we have time later too, I I've actually been working on a an academic paper submission uh with a psychologist and so a little bit fascinating kind trying to look through" }, { "i": 52, "speaker": "Speaker 1", "text": "the lens of like okay like in my article I laid out these these three you know they're unpredictable they're capable machines be destructible to a fault. I'm basically stating those as fact and it's" }, { "i": 53, "speaker": "Speaker 1", "text": "empirical observation but like working with a psychologist who's like saying like well you can look at the way they behave and frame it in terms of human psychological research for the last you" }, { "i": 54, "speaker": "Speaker 1", "text": "know five or six decades and it's just fascinating sort of the the the conclusions you can draw about like here's why you just it's sort of maybe it's not proof but it's basically like" }, { "i": 55, "speaker": "Speaker 1", "text": "here is here is academic evidence for why you have to have this outof-band governance that I argue in the article which is what I'll I'll go to first, but like it's kind of this fascinating path" }, { "i": 56, "speaker": "Speaker 1", "text": "of like there actually is research behind like the way that AI behaves the way that it works today and why it lacks the structural foundations that humans have that allow them to operate" }, { "i": 57, "speaker": "Speaker 1", "text": "differently. But I think the the key the key principle that that I push for in the article is saying you have to have out of band enforcement, out of band metadata. Like if you want if there's if" }, { "i": 58, "speaker": "Speaker 1", "text": "there's something you don't want an agent to do, you have to enforce that that happens. You can't say, \"Hey, agent, please don't ever delete the database cuz it will work until it doesn't.\" It's a lot like, you know," }, { "i": 59, "speaker": "Speaker 1", "text": "telling someone and and you know, telling a human like, \"Hey, all of our cash for the company is stored in this room and there's no lock." }, { "i": 60, "speaker": "Speaker 1", "text": "Please don't take any of it. It's fine until it's not right.\" You know, most of your humans will be fine, but it's someday, you know, and it's there's a different reason why they they will fail" }, { "i": 61, "speaker": "Speaker 1", "text": "in those scenarios, but it's a it's the same level of security. you know, you don't just store your your money in a safe that, you know, doesn't have a lock on it because you can't trust it. And so" }, { "i": 62, "speaker": "Speaker 1", "text": "it's sort of that same principle is governance needs to be enforced through channels the agent can't access, can't modify or even see. Um things, you know, inbound controls like prompts, uh" }, { "i": 63, "speaker": "Speaker 1", "text": "training guard models, they'll they all at some point collapse under prompt in injection or hallucination. You know, under pressure, they they tend to fail." }, { "i": 64, "speaker": "Speaker 1", "text": "Um and then in the article I kind of go in more detail into into identifying kind of the four big pillars that we need to look at. Those being identity, authorization, observability and explanability uh and" }, { "i": 65, "speaker": "Speaker 2", "text": "then accountability and control." }, { "i": 66, "speaker": "Speaker 2", "text": ">> Yeah. So let's talk through those different pillars. Um walk me through why you think each is important and why you picked those four." }, { "i": 67, "speaker": "Speaker 1", "text": ">> So we'll start with identity. Um, identity is really important because I mean this should seems kind of obvious, but you you you you need to know, you know, when something happens within your" }, { "i": 68, "speaker": "Speaker 1", "text": "enterprise, you need to know who's responsible for it, right? And humans humans show up at the front door and they get an identity right away. You know, you're given a login, you're given" }, { "i": 69, "speaker": "Speaker 1", "text": "a badge, everything you do is bound to that. Um, it doesn't quite go far enough uh with agents, though. Like the trick with a you know the challenge with agents is that well I guess there's there's two" }, { "i": 70, "speaker": "Speaker 1", "text": "challenges. One is sometimes people treat agents like service accounts. So they're sort of treating treating it like classical software. So they say look with classical software I I've" }, { "i": 71, "speaker": "Speaker 1", "text": "written a you know I've I've written a sometimes we used to call them agent you know I've written a system that's going to do a thing. We create a service you know service account for it. It" }, { "i": 72, "speaker": "Speaker 1", "text": "basically has a role. it's given some set of permissions [clears throat] and we know because we wrote that that service that that system that it's only going to do certain things. Obviously," }, { "i": 73, "speaker": "Speaker 1", "text": "there may be bugs or whatever. So, there's kind of a blast radius of what might happen, but they're more or less deterministic um or at least well understood of kind of this is this is" }, { "i": 74, "speaker": "Speaker 1", "text": "what it's supposed to be doing. We run tests, we feel comfortable with it, and we know it's going to basically do the same thing every day, barring bugs, which which happen, and we deal with" }, { "i": 75, "speaker": "Speaker 1", "text": "them. So there's a whole bunch of uh folks in the enterprise who are kind of trying to apply that approach to identity uh and access control to agents. The problem there is agents are not predictable like you know" }, { "i": 76, "speaker": "Speaker 1", "text": "classic software. They will go do random things. So you're you're now saying here's this super capable thing that also tends to misunderstand instructions. It's not a deterministic" }, { "i": 77, "speaker": "Speaker 1", "text": "like you know if this do this do that like you can write that out but how it interprets that is sort of fuzzy uh and you g this broad set of capabilities you know what's going to happen the" }, { "i": 78, "speaker": "Speaker 1", "text": "other thing is that agents are clonable very easily very very easily duplicable so you can end up with you know multiple instances of an agent that are then trying to do different things. So what" }, { "i": 79, "speaker": "Speaker 1", "text": "you really need is from an identity perspective, you need each agent instance for any given specific task it's running to have an identity nailing down like this instance is, you know," }, { "i": 80, "speaker": "Speaker 1", "text": "this version of the agent bound to this task short-lived. You know, it's essentially every time you ask an agent to go do something, it gets a new badge and that badge tells you exactly. I'm sort of" }, { "i": 81, "speaker": "Speaker 1", "text": "falling into authorization here. We we'll get to that more, but like you know that that that's the identity that works just for this one task it gets to do. So it's always getting these new" }, { "i": 82, "speaker": "Speaker 1", "text": "identities and and and you know kind of really scoping the blast radius. But that alone isn't enough as well because everything agents are doing is is in some way on behalf of humans. Um, this" }, { "i": 83, "speaker": "Speaker 1", "text": "is another piece that I think we're going to try to get into in the paper, but like, you know, when you think about how we've kind of reasoned about humans over the years and and accountability," }, { "i": 84, "speaker": "Speaker 1", "text": "how can you hold an agent accountable for its actions when, you know, when sort of the there is no value system that it holds? You know, it can it can it can sound like it has a value system," }, { "i": 85, "speaker": "Speaker 1", "text": "but really it's just mimicking stuff that it's been trained on. You can't really hold an agent accountable. So at some point in time, everything that an agent does has to be held accountable to" }, { "i": 86, "speaker": "Speaker 2", "text": "some human in some way. Like some human task tasked it to do this, >> right? The agent is not accountable to the code I shipped to GitHub. It's the like even if Claude wrote all the code" }, { "i": 87, "speaker": "Speaker 2", "text": "and I barely looked at it, I'm the one who brought down production happen to do that." }, { "i": 88, "speaker": "Speaker 1", "text": ">> And so in addition to having fine grained kind of taskbased identity, you need hybrid identity that carries along with it this chain of responsibility and says this is agent X. it's supposed to" }, { "i": 89, "speaker": "Speaker 1", "text": "do this task and it's doing it on behalf of Tyler or whoever or maybe it's on behalf of agent Y who's doing it on behalf of Tyler. Um, and so, you know, identity needs to expand to be a little" }, { "i": 90, "speaker": "Speaker 1", "text": "more flexible and a little more fine grained just because that's the nature of of these agents and and you're wanting to limit the the blast radius there." }, { "i": 91, "speaker": "Speaker 2", "text": ">> So, that's the identity part and then you started talking about authorization which I feel like flows directly next, right? Where if we have identity, of course we want authorization to uh" }, { "i": 92, "speaker": "Speaker 2", "text": "confine the blast radius as you put it." }, { "i": 93, "speaker": "Speaker 1", "text": "Yeah, it's it's it's really hard to talk about one without talking about the other as as you've already seen. Um, so on the off side again like you really need to move away like again if you" }, { "i": 94, "speaker": "Speaker 1", "text": "think about how we've done it with humans, human o is primarily roll scoped and longived. you know, you've got someone who's a, you know, they're a DBA. Uh, they get DBA permissions and" }, { "i": 95, "speaker": "Speaker 1", "text": "they they're just sort of like that's their permissions all the time because they're going to be doing DBA work all day long. And they are, you know, they have been vetted to have a history of," }, { "i": 96, "speaker": "Speaker 1", "text": "you know, history of employment, you know, a match within the company, sort of culture and and ability to work with others, uh, domain knowledge, all these sorts of things that sort of make it" }, { "i": 97, "speaker": "Speaker 1", "text": "feel reasonable that, you know, we trust this person to have these permissions at all times. Um there are examples of of short-lived access controls uh you know like uh usually it's like within like" }, { "i": 98, "speaker": "Speaker 1", "text": "you know sort of cloud services you know they're the break break glass type systems where you're like well look I need to go in and access customer data basically I'm saying look I need access" }, { "i": 99, "speaker": "Speaker 1", "text": "to the customer data please log then I'm going to go access it you know that's okay like but these sorts of things are are very uh it's sort of the exception not the default right so we" }, { "i": 100, "speaker": "Speaker 2", "text": ">> if I can jump in here it's interesting you're actually making me think about timebound events and authorization too." }, { "i": 101, "speaker": "Speaker 2", "text": "So, I mean, this is a little bit of a sidebar, but um I the what's coming into my head is almost like if I'm going to a conference, there are variety of authorizations that occur for that two" }, { "i": 102, "speaker": "Speaker 2", "text": "days of that conference. And that's almost the problem we have to solve here, which is this agent's pro like most agents aren't going to exist forever. They're going to be active for" }, { "i": 103, "speaker": "Speaker 2", "text": "a couple days. I'll end up spinning a new instance. I need to provide them the right credentials for that period of time and probably have like layered credentials. So like maybe media" }, { "i": 104, "speaker": "Speaker 2", "text": "credentials versus speaker credentials versus attendee credentials for that event which is them taking on this task." }, { "i": 105, "speaker": "Speaker 2", "text": "Or are you thinking more of I think what you're saying which is hey I have an agent that I'm going to keep spinning up and I need to need to repeatedly give it this long form credentials or that hey" }, { "i": 106, "speaker": "Speaker 2", "text": "we just constantly have these events we needed to solve. H how are you thinking about that credentials piece there? And sorry to throw a wrench into your >> No, that's No, it's totally fine. I" }, { "i": 107, "speaker": "Speaker 1", "text": "think that's a good the conferencing is a good analogy, too, because it does tie into some of the Let me let me just kind of lay out the the four keys, >> please. Yeah, I know I'm derailing you." }, { "i": 108, "speaker": "Speaker 1", "text": ">> No, it's it's fine. But I think but you'll see how that does tie into what you were saying of like, you know, the the kind of the four pieces I highlight in the article are that, you know, you" }, { "i": 109, "speaker": "Speaker 1", "text": "need to be able to have authorization that's narrowly scoped. So you you you really want to limit to to the specific task at hand task at hand not everything the agent might ever need right and so" }, { "i": 110, "speaker": "Speaker 1", "text": "it needs to be dynamically narrow in that sense shortlived as well permissions should expire you know an agent that needs access to a billing database for a specific job at say 2 p.m. shouldn't still have that access an" }, { "i": 111, "speaker": "Speaker 1", "text": "hour later or maybe even a minute later like you know it should be it should be you know very shortlived I think then there's a couple more pieces that are a little bit more agent specific um but" }, { "i": 112, "speaker": "Speaker 1", "text": "they they do start to allude to kind of the conference thing or like the guest bad sort of example that I give in the in the article one is they need to be deny capable so you need to be able to" }, { "i": 113, "speaker": "Speaker 1", "text": "say this agent can never write to the production database it can only ever read and I if if it's doing work on behalf of another human um and needs to adopt some of their permissions that's" }, { "i": 114, "speaker": "Speaker 1", "text": "fine but it it may never adopt uh right access. Um so that the last point that that kind of gets highlighted there is it has to be intersection aware. it needs to, you know, the agent" }, { "i": 115, "speaker": "Speaker 1", "text": "kind of has a list of things that is allowed to do and never allowed to do." }, { "i": 116, "speaker": "Speaker 1", "text": "And then when a human comes along and says, \"Okay, please go do this task for me,\" it can adopt some subset of those, but it's the intersection of them. And if if the if the rules say agent may" }, { "i": 117, "speaker": "Speaker 1", "text": "never write, even if the human has all right access to the production database, the agent gets none of that." }, { "i": 118, "speaker": "Speaker 2", "text": ">> The agent doesn't own the conference venue. It can't keep throwing uh conferences and giving itself media access, but it can get a back access badge to for the two days it's there." }, { "i": 119, "speaker": "Speaker 1", "text": "and present needs it." }, { "i": 120, "speaker": "Speaker 1", "text": ">> Yeah. Or like the the guest badge example in the article is, you know, you can get a guest badge and you can kind of go anywhere that a human will take you, but even if the human has access to" }, { "i": 121, "speaker": "Speaker 1", "text": "the secret server room that guests aren't allowed in, you're still not allowed to go because you've got the guest badge." }, { "i": 122, "speaker": "Speaker 2", "text": ">> I think it's a great example. Um, okay." }, { "i": 123, "speaker": "Speaker 2", "text": "And the way that we actually enforce all of this or at least understand if it's working is obviously the observability layer, which is something we've talked some on the show about. Well, let's be" }, { "i": 124, "speaker": "Speaker 2", "text": "honest, a lot in the show about. uh but I feel like there's still a gap around it or at least many folks I think think there's a gap for observability right now. What's your take on this" }, { "i": 125, "speaker": "Speaker 1", "text": "observability layer of Asian governance and where it's succeeding right now and maybe where we need to dive deeper on it? I think that the the biggest thing that we need in my opinion is is we all" }, { "i": 126, "speaker": "Speaker 1", "text": "need to agree that you just need to be recording everything that agents do because we've we've kind of operated in this world where you can more or less trust humans to a certain degree and" }, { "i": 127, "speaker": "Speaker 1", "text": "obviously you can't trust them fully but you know over over millennia we've built up these systems that that basically make it so that as the stakes increase like as the pressure of like" }, { "i": 128, "speaker": "Speaker 1", "text": "did you do something wrong investigations, criminal investigations, you know, going, you know, all that. As as the stakes increase, humans tend more and more towards the truth because" }, { "i": 129, "speaker": "Speaker 1", "text": "there's there's, you know, life impacting uh repercussions as a result." }, { "i": 130, "speaker": "Speaker 2", "text": ">> I mean, not to jump you a pillar ahead, but we have accountability and enforcement, right?" }, { "i": 131, "speaker": "Speaker 1", "text": ">> Yeah. I know. I keep each really like each of these I think in the in the paper we're writing, we kind of lay it out this way, like each of these are are sort of build on each other, and so it" }, { "i": 132, "speaker": "Speaker 1", "text": "ends up being hard to talk about one because you start spilling into the next one. Um, but because because of that, because you lack that, you don't you you sort of lack these these levers to to" }, { "i": 133, "speaker": "Speaker 1", "text": "push on them and and because they're not like classic software either where you can go in and be like, \"Okay, well, it did the wrong thing.\" But like, somebody put a stupid if statement somewhere or there's a bug" }, { "i": 134, "speaker": "Speaker 1", "text": "or there's a race condition, whatever." }, { "i": 135, "speaker": "Speaker 1", "text": "Like, you can't debug them because of these just these organically grown, you know, models that have been trained. The only real recourse you have is to record everything. So every every prompt, every" }, { "i": 136, "speaker": "Speaker 1", "text": "input, every tool request, every tool response, every output, record record the whole thing and then be able to, you know, from that you can do many things." }, { "i": 137, "speaker": "Speaker 1", "text": "You can build observability systems that tell you what are all my agents doing now. Um, you can build uh analysis tools that let you go back and say, okay, this agent, you know, gave away a car. Why" }, { "i": 138, "speaker": "Speaker 1", "text": "did that happen? What what went on here?" }, { "i": 139, "speaker": "Speaker 1", "text": "You know, how do we how do we fix that in the future? Or alternatively, this agent's knocking it out of the park." }, { "i": 140, "speaker": "Speaker 1", "text": "this agent has converted, you know, dozens of customers in the last month when, you know, our our BDRs, you know, our human BDRs aren't what is it doing?" }, { "i": 141, "speaker": "Speaker 1", "text": "How can we do more of that? Um, both of those last two scenarios lead into, you know, rather than having humans kind of do this analysis and say, let's let's, you know, let's stop this agent or let's" }, { "i": 142, "speaker": "Speaker 1", "text": "start another one lead into evaluations and and kind of automatically uh, you know, both promoting or demoting agents over time of saying, \"This agent's doing great work. you know, let's maybe do it" }, { "i": 143, "speaker": "Speaker 1", "text": "more or let's save this as kind of a golden rule set for how we, you know, approach this sort of problem in the future or this agent is not performing like it was. You know, I think a common" }, { "i": 144, "speaker": "Speaker 1", "text": "thing I hear is that people will they'll ship an agent like we we shipped a like a customer service slackbot, you know, about a year ago and it was awesome for like two or three weeks. Uh, and then" }, { "i": 145, "speaker": "Speaker 1", "text": "like fourth week in all of a sudden customers were saying like it gave me this really weird suggestion that makes no sense and it just it was just for whatever reason the agent just kind of" }, { "i": 146, "speaker": "Speaker 1", "text": "stopped functioning. I don't know if the model I don't I don't remember whatever the actual postmortem on it was if it was the model changed or if context had built up and it stopped giving good" }, { "i": 147, "speaker": "Speaker 1", "text": "advice or something but it just it went from working great to not working great." }, { "i": 148, "speaker": "Speaker 1", "text": "So you want to have these you want to have evaluation systems in place that are you know monitoring either either in absolute or in relative like how is the performance of these agents and are they" }, { "i": 149, "speaker": "Speaker 1", "text": "doing the right things. Um so having having a full recording of everything they do is is kind of the foundational piece for all of those. Do you think the open telemetry standard uh is the place" }, { "i": 150, "speaker": "Speaker 2", "text": "to start for agents or how do you think about I mean you said record everything but what where would you recommend teams get started if they're kind of skipping this step so so far?" }, { "i": 151, "speaker": "Speaker 1", "text": ">> I do think open telemetry is a great place to start. that's what we've more or less aligned on as well. Like cuz the the other piece that I haven't touched on anywhere in all of this um but like" }, { "i": 152, "speaker": "Speaker 1", "text": "talking about open telemetry as a standard is a good place to to begin that conversation is if you think about where you want agents to be in your in in sort of the modern enterprise you" }, { "i": 153, "speaker": "Speaker 1", "text": "want them as you know basically digital a digital workforce along you know collaborating with your humans you want them everywhere. You want them touching everything in your your enterprise. And" }, { "i": 154, "speaker": "Speaker 1", "text": "so by definition then you can't have them in some little walled garden somewhere where you're like well we'll put all the data we want the agents to work on in there and let them do their" }, { "i": 155, "speaker": "Speaker 1", "text": "thing. You're you're missing out on the opportunity to have agents you know actually benefiting you everywhere. And so if it needs to be everywhere then you really need to outline on" }, { "i": 156, "speaker": "Speaker 1", "text": "interoperability and open standards are a great way to achieve that. And so that's why open telemetry I think is is a great start. That's what we use within the the system we're building at Red" }, { "i": 157, "speaker": "Speaker 1", "text": "Panda. And that let you know like for us you know we're trying to build a governance platform uh that does solve this kind of meet you know meet you where you are and work with all the" }, { "i": 158, "speaker": "Speaker 1", "text": "systems that means we can build our own agent framework and collect all the stuff we want from it fine like anybody can do that but you can bring whatever agent you want you know build it with" }, { "i": 159, "speaker": "Speaker 1", "text": "whatever framework you want and you can easily plug it in as long as it speaks open telemetry for for sending us you know trace information and that's by doing that it's kind of a lower bar it" }, { "i": 160, "speaker": "Speaker 1", "text": "just makes it easy to be like yeah if if you're not using open telemetry Why aren't you? It's not that hard. Um, >> so a couple things this brings to mind." }, { "i": 161, "speaker": "Speaker 2", "text": "Well, many things, but two that I want to focus on to start is is one, you mentioned Red Panda is obviously using agents. I'd love to understand some about how you were using AI agents today" }, { "i": 162, "speaker": "Speaker 2", "text": "and then secondly, how are you holding them accountable to speak to that fourth pillar?" }, { "i": 163, "speaker": "Speaker 1", "text": ">> That's a good question." }, { "i": 164, "speaker": "Speaker 1", "text": "So, how am I using agents today? Um, I use them a lot for I would say myself I use them a lot for research. Um, you know, I'm I'm personally more in the like kind of AI tool realm than the, you" }, { "i": 165, "speaker": "Speaker 1", "text": "know, autonomous AI agent running off and doing things for me realm. Um, that's just a sort of personally where I am and where I'm, you know, where I want to have things running for myself. I" }, { "i": 166, "speaker": "Speaker 1", "text": "think most of the autonomous stuff, we do have a bunch of autonomous agents that we built internally at Red Panda." }, { "i": 167, "speaker": "Speaker 1", "text": "Um and then we have customers that are using them as well. Um I think examples for those are uh we have agents that are uh you know on the marketing side for example doing a bunch is this does fall" }, { "i": 168, "speaker": "Speaker 1", "text": "into research as well but they're they're basically autonomously doing research around competitors and saying hey you know what are competitors doing or what are we seeing in the market what" }, { "i": 169, "speaker": "Speaker 1", "text": "are useful things here giving us reports um making suggestions. Um again kind of falls into the to the the research arena. Um other ones that we've seen or that we're working on too uh that are a" }, { "i": 170, "speaker": "Speaker 1", "text": "little more uh towards the operation side is is trying to automate uh blog posting and things like that. So, so being able to look at, you know, what is it that we're building? What have we" }, { "i": 171, "speaker": "Speaker 1", "text": "shipped? Um, you know, calling through calling through Jira, calling through through uh GitHub PRs and things like that and coming back with recommendations, not just for like here's things you should be talking" }, { "i": 172, "speaker": "Speaker 1", "text": "about, but actually writing, you know, writing an initial draft of like, hey, maybe a short article like this would be useful and then letting marketing take that. So rather than the marketing" }, { "i": 173, "speaker": "Speaker 1", "text": "having to like chase everyone in the company and say, hey, you know, what are you up to? What are we going to do? you know, this can be an additional signal of like, you know, basically you've got" }, { "i": 174, "speaker": "Speaker 1", "text": "agents that are that are feeding you ideas of, you know, I'm aware because we have, you know, this whole body of kind of, you know, what is Red Panda, what are we doing, you know, stored in in" }, { "i": 175, "speaker": "Speaker 1", "text": "data that they have access to as well." }, { "i": 176, "speaker": "Speaker 1", "text": "You know, they're sort of the aware of the context of what we're doing, aware of our mission, aware of our, you know, what we want to do from a marketing perspective, all that sort of stuff." }, { "i": 177, "speaker": "Speaker 1", "text": "They can kind of frame, you know, their approach to like what is it that we want to pitch, what should we be talking about, you know, within that. So it's not just like you're asking some random" }, { "i": 178, "speaker": "Speaker 1", "text": "LLM either. You know, it's sort of finely tailored to all the stuff that we're planning. So, um not sure if that gives you a good sense of it. Like some of our customers are doing things like" }, { "i": 179, "speaker": "Speaker 1", "text": "um you know, wealth management advisory sort of stuff. Again, kind of doing research and and giving advice, but not not to the point of being hooked up to doing autonomous uh trading and things" }, { "i": 180, "speaker": "Speaker 1", "text": "like that. Um we do have we have a paper that uh we submitted to the CIS one of the CIS uh workshops that does give a sort of an autonomous demo around a wealth management thing but but" }, { "i": 181, "speaker": "Speaker 1", "text": "highlights this kind of outofband enforcement piece of saying you know the trading aspect of it you know you could totally imagine imagine saying like look here's the parameters I want for" }, { "i": 182, "speaker": "Speaker 1", "text": "automatic trading [clears throat] but any trade that has you know more you know more impact than say $1,000 on my portfolio always needs to be reviewed by a human and there is there is no way for that to" }, { "i": 183, "speaker": "Speaker 1", "text": "go for the agent to to get past that." }, { "i": 184, "speaker": "Speaker 1", "text": "The agent doesn't even know that that's there. All it does is it make makes recommendations and on the the infrastructure side if it ever exceeds that limit then the infrastructure" }, { "i": 185, "speaker": "Speaker 1", "text": "automatically says nope that's going to a human for review. um the true sort of autonomous operational kind of agents like that's where we're headed and I think from what I've seen we're still" }, { "i": 186, "speaker": "Speaker 1", "text": "pretty early days as far as seeing these types of agents in production because people don't get have the confidence that they can do it without without blowing away millions of dollars or you" }, { "i": 187, "speaker": "Speaker 1", "text": "know you know having the agent say things that that makes you know customers run away screaming or something like that like there's just people are being conservative and rightly so. But I think the stuff that" }, { "i": 188, "speaker": "Speaker 1", "text": "we're talking about as far as providing a governance layer that lets you kind of set up the guidelines and set the rules is what's going to get us there. Like we're so close." }, { "i": 189, "speaker": "Speaker 2", "text": ">> Yeah, it's really interesting because we all are seeing massive opportunities to increase velocity, increase throughput, and we want to lean into those, but there are brand risks, there are" }, { "i": 190, "speaker": "Speaker 2", "text": "monetary risks. I mean like the the very simple example that I have for myself is I built uh an MCP server to let my agents access my Substack. So they can draft for me, they can make edits, they" }, { "i": 191, "speaker": "Speaker 2", "text": "can pull data from it. Um but I very specifically did not give them authorization to actually send Substacks uh because I don't want them to ever do that accidentally and because I view it" }, { "i": 192, "speaker": "Speaker 2", "text": "as as a brand risk. And does it matter that much? I have right now like a thousand subscribers on Substack. So does it doesn't matter that much but it matters to me as my own like personal" }, { "i": 193, "speaker": "Speaker 2", "text": "brand and like how I want to present myself um as like and that's a very you know simple example but like as you start to scale to things that are worth millions of dollars. Yeah. There's a" }, { "i": 194, "speaker": "Speaker 2", "text": "reason these controls are in place and that we need these authorization pieces and it brings to mind something that you wrote in the article where you talked about you know every human employee has" }, { "i": 195, "speaker": "Speaker 2", "text": "a manager. You know critical actions need approvals. Um, and if things go catastrophically wrong, and I'm I'm quoting basically word from word here from Tyler, so this is not my words. Uh," }, { "i": 196, "speaker": "Speaker 2", "text": "there there is a chain of responsibility and there's a kill switch, a circuit breaker of some type. But for agents, this layer is nason at best. What do we need to do to solve this both" }, { "i": 197, "speaker": "Speaker 2", "text": "responsibility chain and then this circuit breaker, this access revoke, whatever it is, uh, layer." }, { "i": 198, "speaker": "Speaker 1", "text": ">> Yeah. And the and the as [clears throat] I've sort of been alluding to over and over as we go on like each of these pillars builds upon the other and that's the nice thing like really this idea of" }, { "i": 199, "speaker": "Speaker 1", "text": "chain of responsibility and being able to have a kill switch. They're largely dependent on just saying look solve identity and authorization properly." }, { "i": 200, "speaker": "Speaker 1", "text": "Like if you if you have this global approach to identity that is fine grain enough that literally every every instance of an agent for a given task has a unique identity and there is an" }, { "i": 201, "speaker": "Speaker 1", "text": "authorization system that then you know plays nicely with that and and is aware of that. You have basically what you need like the identity layer is is tracking sort of the the the" }, { "i": 202, "speaker": "Speaker 1", "text": "accountability layer of which humans you know authorize this. uh the authorization system is dynamic enough and fine grained enough that you can just basically say look this agent or" }, { "i": 203, "speaker": "Speaker 1", "text": "this class of agents are going you know are are you know we want them to stop what they're doing we are going to just shut those off their their access is revoked um that really is the best way" }, { "i": 204, "speaker": "Speaker 1", "text": "to do it and so I think that that is kind of the one the you know if I were to give the highle like recommendation to folks of like if you're looking to build out a broad suite of autonomous" }, { "i": 205, "speaker": "Speaker 1", "text": "agents in a in a large you know enterprise make sure that you some governance layer in place that globally you know everything goes through because this is what gives you that ability to" }, { "i": 206, "speaker": "Speaker 1", "text": "centrally manage this access and you know pro you know make sure it has identity and authorization that meet the needs that you you're going to have but if you have that then then you've got" }, { "i": 207, "speaker": "Speaker 1", "text": "your accountability layer you've got your kill switch or your circuit breaker or whatever um so it all it all kind of builds on those pillars and that you know they were very intentionally" }, { "i": 208, "speaker": "Speaker 2", "text": "uh phased the way they were because of that >> and importantly as you also highlight in in the article that governance layer has to be enforced through channels that the agents cannot access themselves. Like" }, { "i": 209, "speaker": "Speaker 2", "text": "it's great to give an agent a skills file and memory and context and all these things but it won't necessarily follow that all the time. There will be moments where it does not succeed in" }, { "i": 210, "speaker": "Speaker 2", "text": "that. So we have to have outofban metadata where we can actually enforce this without the agent overwriting or breaking its own systems." }, { "i": 211, "speaker": "Speaker 1", "text": ">> Yeah, absolutely. And and to be clear like it's harder this way. like this is more work, you know, like the the sort of the state-of-the-art is like we'll throw in your cloud MB, tell it what to" }, { "i": 212, "speaker": "Speaker 1", "text": "do. Like that'll that gives it the parameters of how to operate and like that's true until it's not." }, { "i": 213, "speaker": "Speaker 2", "text": ">> It gets you there 90% of the time, but once it once it's customer data, once it's customer money, like >> they're a different standard." }, { "i": 214, "speaker": "Speaker 1", "text": ">> Exactly. And that's the thing. And so like you you really can't do inline enforcement. It just it doesn't work. Um or at least it doesn't like as you said, it doesn't work to the standard that" }, { "i": 215, "speaker": "Speaker 2", "text": "you're going to need to hold in these situations, >> right? Like it's if if it's me writing tweets with my open claw or something like that like okay fine it's probably not the biggest deal in the world but" }, { "i": 216, "speaker": "Speaker 2", "text": "yeah >> if I am trying to run my finances I I probably care more." }, { "i": 217, "speaker": "Speaker 1", "text": ">> Yep." }, { "i": 218, "speaker": "Speaker 1", "text": ">> Yeah we and and I mean we have you know we struggle with this too internally like we're we're we're working on moving you know very deeply to like basically having everything that we do internally" }, { "i": 219, "speaker": "Speaker 1", "text": "from like you know wiks and docs and all that to to having that accessible to our to our AI agents. And you know, one of the one of the guideline papers that we wrote up recently said, you know, make" }, { "i": 220, "speaker": "Speaker 1", "text": "sure you you know, narrow the scope of what what you know, agents can do for this given section of the of the repository with your cloud. MD. And I and I had to go and be like, yes, but" }, { "i": 221, "speaker": "Speaker 1", "text": "like make sure that we make it clear for everyone what this really means. This is like this is this is this is guidance, you know, this is saying please try to do this, but this if it actually matters" }, { "i": 222, "speaker": "Speaker 1", "text": "that it doesn't do a thing, we can't enforce it here. We've got to do it through the, you know, for in our case, we've got it through the ADP layer. Um, so just some of it's just just helping" }, { "i": 223, "speaker": "Speaker 1", "text": "educate people of of that difference and and being aware that like yes, guide guidelines are useful and they're simple and easy and they're great when they work up to the point that they don't." }, { "i": 224, "speaker": "Speaker 2", "text": "And so anything that you need to have actually work, you've got to it's got to be out of band. It's got to be infrastructure enforced. I know that you have spent a lot of time on thinking" }, { "i": 225, "speaker": "Speaker 2", "text": "through this problem and you restrained yourself in the article that we've alluded to several times now from talking about streaming that explicitly as a solve here. Uh but you know as I" }, { "i": 226, "speaker": "Speaker 2", "text": "said at the start of the call uh I mean you've spent 15 something years working on this problem. Uh obviously you're applying many of those lessons at Red Panda." }, { "i": 227, "speaker": "Speaker 2", "text": "Do you think event streaming is the right substrate for agent governance?" }, { "i": 228, "speaker": "Speaker 1", "text": ">> I think I think event streaming is a very useful piece of the puzzle. Um, if I thought it was if I thought it was the core piece of it, then we as a company would be saying that loud and clear. But" }, { "i": 229, "speaker": "Speaker 1", "text": "I I think I think anybody who actually looks at the problem below surface level can see that like an argument like that is is it's hard to make that argument and have it actually land as anything" }, { "i": 230, "speaker": "Speaker 1", "text": "other than a company who happens to do streaming is trying to pitch themselves as as an AI company like because the the there are certainly plenty of use cases where streaming data is relevant within" }, { "i": 231, "speaker": "Speaker 1", "text": "this. Um, and I did I did give a talk." }, { "i": 232, "speaker": "Speaker 1", "text": "So, back when I was trying to figure this all out on my own in the fall, like I did I did start giving a talk basically saying like part of why we got into this is that streaming is a key" }, { "i": 233, "speaker": "Speaker 1", "text": "foundational piece. That was why it felt uh, you know, sort of authentic for us as a company to to get into this space because the transcripts aspect, being able to record this, you know, it's it's" }, { "i": 234, "speaker": "Speaker 1", "text": "a lot of information trying to record all this stuff and it's, you know, streaming is a natural way to do that." }, { "i": 235, "speaker": "Speaker 1", "text": "And then a lot of the the way that the systems interact is a very streaming sort of uh approach, but then there's a lot of agents that just talk RPCs and stuff and like do you want to really" }, { "i": 236, "speaker": "Speaker 1", "text": "shove that stuff through like a Kafka kind of a topic? Not really. Um so so that's kind of my take on it is it's a foundational piece. We see it as a critical part of our infrastructure. Um" }, { "i": 237, "speaker": "Speaker 1", "text": "but it's you know it's like a third of the story or you know it's it's a significant but certainly not uh all-encompassing part. So that's that's my take and yes I try very hard in how" }, { "i": 238, "speaker": "Speaker 1", "text": "we approach things to not pull streaming into it because it is so easy to otherwise look like you're oh look it's a streaming guy who's saying that streaming is the way to do AI and like" }, { "i": 239, "speaker": "Speaker 1", "text": "it's not like it's a piece of it. Yes." }, { "i": 240, "speaker": "Speaker 1", "text": "And it's it's important for those pieces but like with any tool you've got to use it for the right the right part and and we're definitely not the the not the streaming company who's coming to tell" }, { "i": 241, "speaker": "Speaker 2", "text": "you that streaming is the way to do AI." }, { "i": 242, "speaker": "Speaker 2", "text": ">> I mean we're all constantly pattern matching so it's understandable to bring it in. And I agree. I think it is a part of the solution here. And my my longtime listeners will probably tease me in the" }, { "i": 243, "speaker": "Speaker 2", "text": "comments for the fact that I continually reference old episodes as I'm thinking about it because that's my mental framing of like, yeah, you know, like here's what I'm taking to these" }, { "i": 244, "speaker": "Speaker 2", "text": "conversations. Um, but I know Red Panda has shipped a variety of things around um helping to enable a agents in the enterprise. you know the agentic data plane which you shipped I think late" }, { "i": 245, "speaker": "Speaker 2", "text": "2026 uh or sorry early 2026 um obviously you know Kafka has been mentioned already what's the general company approach here for how you're helping red panda customers and partners to actually solve" }, { "i": 246, "speaker": "Speaker 1", "text": "these problems >> so I see there's the way we're approaching it as a company is that there's there's kind of two main two main businesses that we're investing in going forward one One is the agent" }, { "i": 247, "speaker": "Speaker 1", "text": "governance and the other is the the the more classic data platform. Used to be streaming but with the acquisition of oxala and the SQL engine. It's broadened out to more of a data platform going" }, { "i": 248, "speaker": "Speaker 1", "text": "forward. Um and the the agentic stuff the agent data plane builds upon the data platform. Uh but it's it's really infrastructure that we use like at this point in time with with ADP the agent" }, { "i": 249, "speaker": "Speaker 1", "text": "data plane like we're using red panda topics we're using oxa and things like that under the covers. users don't see that like they don't see the the red p like it's it's a totally separate" }, { "i": 250, "speaker": "Speaker 1", "text": "cluster it's just an implementation detail um there's also other things in there too like there's some Postgress in there that we use right they don't care they don't need to know it's sort of" }, { "i": 251, "speaker": "Speaker 1", "text": "foundational stuff so from the from the AI side we're we're really trying to solve those problems the way that they need to be solved so we're giving you you know the LLM gateways the MCP" }, { "i": 252, "speaker": "Speaker 1", "text": "gateways uh the identity and authorization stuff that you need uh the ability to plug agents in or build your own agents you know everything that you'd want around sort of governing" }, { "i": 253, "speaker": "Speaker 1", "text": "agents broadly in the enterprise. It's built there with that mindset of this is for agents and it makes governance of them you know interoperate across your enterprise and and easy to sort of centralize things" }, { "i": 254, "speaker": "Speaker 1", "text": "but the implementation details are sort of irrelevant. At the same time we have built a really nice data platform and if [clears throat] you want to do data streaming across you know a broad" }, { "i": 255, "speaker": "Speaker 1", "text": "variety of latency profiles and cost profiles and you know do it on premise do it bioc do it in in the cloud you know we kind of do it all um and we've got oxa via red panda SQL coming here in" }, { "i": 256, "speaker": "Speaker 1", "text": "a few weeks um starting to bring SQL querying capabilities uh you know across those that same set of modalities and and really trying to focus on bridging you know bridging real-time data and" }, { "i": 257, "speaker": "Speaker 1", "text": "real-time queries and things like that." }, { "i": 258, "speaker": "Speaker 1", "text": "So there's there's this whole data platform thing that we're still investing heavily in and our customers use a bunch and you know that business continues to grow as well. We're you" }, { "i": 259, "speaker": "Speaker 1", "text": "know they're related but you know to be perfectly honest they're they're kind of two different two different scenarios and and really two different uh sets of customers that we're looking at." }, { "i": 260, "speaker": "Speaker 1", "text": "Obviously we've got plenty of customers who are um you know doing both uh you know you know using the data platform and using using the the AI stuff but it's they're different personas. So, I don't know if" }, { "i": 261, "speaker": "Speaker 1", "text": "that's answering your question, but that's that's kind of how we approach it." }, { "i": 262, "speaker": "Speaker 2", "text": ">> It makes me think about a couple of things that came up during our prep call, which is one um this idea of how companies are changing for the future, you know, like how do we have to broadly" }, { "i": 263, "speaker": "Speaker 2", "text": "adjust? We've talked a lot about the pillars for agents, but how do we integrate our agents within the human workforce?" }, { "i": 264, "speaker": "Speaker 2", "text": "And two, something that has been I think maybe unsaid within this conversation so far, but is a sub theme that I'll I'm going to make explicit here, which is I mean really the quiet argument here that" }, { "i": 265, "speaker": "Speaker 2", "text": "I think we're both making is that perfect model alignment is not possible in this current time period. It is also not likely to solve your problems. it may get you to you know 95 versus 85% of" }, { "i": 266, "speaker": "Speaker 2", "text": "the time you're having your agents work if you give them good MDs etc. It's still down to the structure and the systems we provide and you know we've talked with like Awarius Rayasan last" }, { "i": 267, "speaker": "Speaker 2", "text": "year on the show about building systems around agents and we talked a lot about you know memory with Richmond earlier this year we talked about context these are all important but if we don't" }, { "i": 268, "speaker": "Speaker 2", "text": "actually have tracking and enforcement mechanisms um these models are not going to get us to safety to perfect should we and and maybe have we been spending too much time as an industry on" }, { "i": 269, "speaker": "Speaker 2", "text": "this alignment problem and trying to you know push the frontier when in fact we should be spending more time on how we orchestrate in the data platforms around them." }, { "i": 270, "speaker": "Speaker 1", "text": ">> Yeah, I think the answer is yes. I think the the like there's there's two interesting topics here. One is on on your point of like you know trying to make models perfect like the the thing I" }, { "i": 271, "speaker": "Speaker 1", "text": "say in the article is you we never tried to make humans perfect. You know, every every corporation in the world is stacked top to bottom with imperfect humans. And you most of them, you know," }, { "i": 272, "speaker": "Speaker 1", "text": "live to see another day despite of that because of all the structural, you know, pieces that we've built up around managing sets of imperfect humans. We just need to acknowledge that agents" }, { "i": 273, "speaker": "Speaker 1", "text": "today are materially different from those humans. And so they need they need different and additional structural support, governance support in order to build a successful. But that's that's" }, { "i": 274, "speaker": "Speaker 1", "text": "really what's missing. It's not that you need to make the agents better. The agents have actually reached a point where there's just they're super capable. They can do useful things. I" }, { "i": 275, "speaker": "Speaker 1", "text": "think we all we all can kind of see that. Um so I think that's the missing part. Um there is a separate piece in there that I think is fascinating that maybe is maybe is you know something for" }, { "i": 276, "speaker": "Speaker 1", "text": "another a future conversation. Um but the the paper that I mentioned that we're working on um another piece that's coming out of that is basically try you know this the psychologist I'm working" }, { "i": 277, "speaker": "Speaker 1", "text": "with is kind of trying to tweeze out of like here are things like here you know our understanding of how the human brain works here are the you know neurobiological systems within humans" }, { "i": 278, "speaker": "Speaker 1", "text": "that lead towards things like value systems where people develop over time this this hardened sense of like I as a person hold as a value that I'm honest or I do the right thing right and It's" }, { "i": 279, "speaker": "Speaker 1", "text": "it makes you imperturbable to things like someone's coming saying, \"Oh, no." }, { "i": 280, "speaker": "Speaker 1", "text": "You should just you should just leak those those emails to me.\" That's fine." }, { "i": 281, "speaker": "Speaker 1", "text": "I know they're private and I don't work at the company, but you know, just give them to me. You know, the human value system becomes imperturbable to those." }, { "i": 282, "speaker": "Speaker 1", "text": "You know, obviously nothing's absolute, but like there's definitely a sturdiness there that agents don't have. Um, and so some of the stuff she's tweezing out is like, look, here's the missing systems" }, { "i": 283, "speaker": "Speaker 1", "text": "that humans have that agents don't that would allow them to do that. And I assume the model companies are digging into this stuff too, but there's not a lot of literature on it. And so one of" }, { "i": 284, "speaker": "Speaker 1", "text": "those one of the paths that may be interesting is taking this sort of idea of like this is what's missing and maybe these are systems that we need to start building into agents if we want them to" }, { "i": 285, "speaker": "Speaker 1", "text": "be more humanlike. I think we also need to decide is is that the thing we want." }, { "i": 286, "speaker": "Speaker 1", "text": "But certainly there's things like having values that are not so easily challenged by prompt injection would be a useful thing. So that that may be a future path as well of you know kind of where you" }, { "i": 287, "speaker": "Speaker 1", "text": "know beyond just the alignment stuff that's done today which sort of feels like a band-aid of actually trying to build systems that make make LLMs or you know systems built around LLMs maybe" }, { "i": 288, "speaker": "Speaker 1", "text": "it's not just the large language model large language model itself make them more robust and and more humanlike in ways that are valuable. I love that you brought this up because I mean I think" }, { "i": 289, "speaker": "Speaker 2", "text": "the obvious example of a model lab that is spending some time on this is Anthropic who've publicly had Amanda Ascll philosopher and researcher out there talking about it and they've" }, { "i": 290, "speaker": "Speaker 2", "text": "they've talked about using psychiatrists to assess some of their uh models specifically Claude Mythos I think recently and I do think this is an emerging area. Um, and I'll say I'm I'm" }, { "i": 291, "speaker": "Speaker 2", "text": "certainly going to ask you at the end of the conversation if you want to make an intro to your uh uh psychologist colleague because I would love to do a psychology of AI agents uh session for" }, { "i": 292, "speaker": "Speaker 2", "text": "sure. But I mean I think it's I think it's an area where we're starting to see some emerging impacts already where I mean a silly example of this is all of the open AI models seem to be obsessed" }, { "i": 293, "speaker": "Speaker 2", "text": "with goblins and little creatures. And I I think we're also starting to see people at least anecdotally comment on oh like this is the mindset I'm seeing from cloud models. This is the mindset" }, { "i": 294, "speaker": "Speaker 2", "text": "I'm seeing from open AAI models. You can look at all the comments around using codecs versus cloud code where oh you know we think anthropic models are better at design and the recent ones are" }, { "i": 295, "speaker": "Speaker 2", "text": "you know aligned with wanting to solve things but maybe can get a little lazy on certain tasks whereas you know open AI's codeex models and GPT 5.5 in particular are are you know very good at" }, { "i": 296, "speaker": "Speaker 2", "text": "just like turning through work but they're not as imaginative creative. Uh so there is something to be said for this idea that we're seeing model families differentiate over time and" }, { "i": 297, "speaker": "Speaker 2", "text": "that they are having different like pros and cons and like cultural psychological impacts though I think there's so much more to your point where we can go deeper on like understanding what this" }, { "i": 298, "speaker": "Speaker 1", "text": "looks like. Uh and yeah be curious where you're seeing movement on this topic currently. I do see I think what I see mostly is the thing you were alluding to at the end there is is this model" }, { "i": 299, "speaker": "Speaker 1", "text": "differentiation and sort of the the tone or the style or the the the sort of specialization um like this isn't agents but like for myself when I'm using you know I use claude and open AI you know chatbt you" }, { "i": 300, "speaker": "Speaker 1", "text": "know for a lot of the stuff I frequently just write a prompt send it to one copy paste and send it to the other and see like you know which one is better because I you know to be perfectly" }, { "i": 301, "speaker": "Speaker 1", "text": "honest it's because I want to stop paying one than the other But I got to I did it for about a month and I'm like I can't get rid of both because like it's it's such a crapshoot. It's so weird." }, { "i": 302, "speaker": "Speaker 1", "text": "Like you know one will just be amazing and then you know the next prompt which I feel is even like sort of similar." }, { "i": 303, "speaker": "Speaker 1", "text": "It'll be the totally different model and it's like it's like it's it's so materially better too. Like it it will be like this answer is garbage versus this answer is great. It just blows my" }, { "i": 304, "speaker": "Speaker 1", "text": "mind. And so I think there is there's definitely something there and I I see a lot of that. I I have not yet seen a lot of investment into that kind of like let's let's build in human value system" }, { "i": 305, "speaker": "Speaker 1", "text": "type structures and things like that that that we're kind of uh alluding to in the paper we're writing. But you know I'm also not an anthropic deep in it and I'm sure there's things that they're" }, { "i": 306, "speaker": "Speaker 1", "text": "doing that like they're not talking about and they won't talk about for another year or two you know publicly." }, { "i": 307, "speaker": "Speaker 2", "text": "So I mean if they want to come on the show publicly and you're listening right now and you want to come reveal some secrets please do. Like Tyler and I would love to hear this." }, { "i": 308, "speaker": "Speaker 1", "text": ">> [laughter] >> So yeah, >> but I I I think I think that's probably a very fascinating uh direction to explore and I'm expecting we'll start seeing that because it it just makes sense." }, { "i": 309, "speaker": "Speaker 2", "text": ">> Totally agreed. Tyler, this has been a fantastic conversation. As we close out here, I want to challenge you to give me a projection for the future, which is hard. I know given how fast things are" }, { "i": 310, "speaker": "Speaker 2", "text": "moving. I know. I know. I see the eye roll, but how do you think we need to change our company systems and like the way we I guess hire and build our human infrastructure for let's call it the" }, { "i": 311, "speaker": "Speaker 2", "text": "next year of work so that we are setting up for an agentic success." }, { "i": 312, "speaker": "Speaker 1", "text": ">> I think the the the big thing that's important as I said is be aware that you need out of band enforcement of all this governance stuff." }, { "i": 313, "speaker": "Speaker 1", "text": "think about how you're going to apply this broadly across your company. You need you need a uniform, you know, as we call it, we need you need an agentic data plane that's sort of being the" }, { "i": 314, "speaker": "Speaker 1", "text": "governance mediator for all this sort of stuff. And you you need to focus on what are what are the differences that that agents have relative to human employees and how does that relate to that" }, { "i": 315, "speaker": "Speaker 1", "text": "governance and I I do think that the four pillars I laid out in the article around identity and off observability and and accountability set a good baseline for it. So, I think if you look" }, { "i": 316, "speaker": "Speaker 1", "text": "at it from that perspective and go into it thinking like there are materially different problems that we need to solve here than we have for humans and let's tackle those, that's kind of where we're" }, { "i": 317, "speaker": "Speaker 1", "text": "going to get to. And my and my guess is that in the next, you know, one to two years max, we're going to kind of establish what this looks like and there's going to be kind of a pattern of" }, { "i": 318, "speaker": "Speaker 1", "text": "here's how you approach it. And of course, there'll be many different ways to actually go about doing that, building it yourself or buying it from a company like us or and everything in" }, { "i": 319, "speaker": "Speaker 1", "text": "between. Um, but I suspect we're going to get to a a point of being like this is roughly the right way to do it. Just like, you know, every every enterprise today is different in the way that it" }, { "i": 320, "speaker": "Speaker 1", "text": "manages its humans, but they're not that different. It all kind of roughly looks the same, you know. Uh, so I think that's that's where I think we're headed. And I I do think it's, you know," }, { "i": 321, "speaker": "Speaker 2", "text": "12 to 24 months max." }, { "i": 322, "speaker": "Speaker 2", "text": ">> Fantastic. Well, Tyler, it's been great catching up with you. Super excited to read your paper when it comes out. and uh thank you so much for sharing all your thoughts with our audience. It's" }, { "i": 323, "speaker": "Speaker 2", "text": "been a great conversation. Where can folks go to follow your work and see what you're up to in the coming months?" }, { "i": 324, "speaker": "Speaker 1", "text": ">> Uh yeah, so uh I'm on LinkedIn, so you can find me on LinkedIn. I'll post I don't post a ton, but I I do post there." }, { "i": 325, "speaker": "Speaker 1", "text": "Um and then everything we do uh also goes through the the sort of the typical Red Panda marketing channels. So, you know, hunt us down there and you'll see what's going on there. But yeah, I'll be" }, { "i": 326, "speaker": "Speaker 1", "text": "there and I'll be I'll be at a few different AI conferences throughout the the year. Certainly CIS. I'm hoping to make it to AIES and and probably AAS later in the the year as well. So for" }, { "i": 327, "speaker": "Speaker 1", "text": "any of you on the the more academic bent of things, I hope to see you there and uh chat more." }, { "i": 328, "speaker": "Speaker 2", "text": ">> Amazing. Tyler, thank you again for joining me today. And listeners, be sure that to subscribe to our Chain of Thought newsletter at newsletter.chainof.show show to get tons more information from Tyler and many" }, { "i": 329, "speaker": "Speaker 2", "text": "others in the coming months as we continue to have these conversations." }, { "i": 330, "speaker": "Speaker 2", "text": "Thank you everyone for joining us and Tyler hope you have a great rest of your day." }, { "i": 331, "speaker": "Speaker 1", "text": ">> Thank you Connor. Really really had a good time here. Appreciate it." } ]