[ { "i": 0, "speaker": "Lauren Tan", "text": "And with GrokBot, we really wanted to take you know, like a fresh perspective on what would it look like if your agent was really more like a colleague? >> Mhm. >> Uh and so I started thinking about how would I make it so my agents could just automatically fix bugs for me while I sleep?" }, { "i": 1, "speaker": "Lauren Tan", "text": "And so that was the inspiration for some of our earlier experiments with, you know, building like uh like a more like persistent bot." }, { "i": 2, "speaker": "Lauren Tan", "text": "If you invest a lot in having like early on the way I see engineers roles evolve in the agent world is uh you know, like I I like I mentioned, you're sort of like an engineering manager for your agents. But I think a big part of our responsibility now as" }, { "i": 3, "speaker": "Lauren Tan", "text": "engineers is also to like uh you know, keep the code base in a condition where, you know, product managers, designers can contribute like high-quality code. Uh okay, so I assume you can see my screen." }, { "i": 4, "speaker": "Speaker 1", "text": ">> Yes." }, { "i": 5, "speaker": "Lauren Tan", "text": "Yeah, we're good. >> So yeah, today yeah, I think I think the big theme for me as I've been using agents to write code, and I'm sure a lot of you have had the same experience as well, is how do you trust it?" }, { "i": 6, "speaker": "Lauren Tan", "text": "You know, especially if you are an engineer that's been writing code for a very long time, you have a lot of opinions and lessons that you've learned about doing good engineering." }, { "i": 7, "speaker": "Lauren Tan", "text": "And when you see agents just, you know, winging it and, you know, guessing, hallucinating, uh you know, confidently stating that they found the smoking gun uh for the hundredth time, uh but it's actually not the real problem, you lose a lot of trust." }, { "i": 8, "speaker": "Lauren Tan", "text": "And when you lose when you don't have much trust in your agents, I feel like you you really can't get the most out of them. And for me, the parallel is like with management." }, { "i": 9, "speaker": "Lauren Tan", "text": "Uh so if I'm an engineering manager for a team, and I have a bunch of, you know, I have a team of engineers uh on my team, and I don't trust them, then the mode of operation I'm going to be in is going to be like micro-management, right?" }, { "i": 10, "speaker": "Lauren Tan", "text": "I'll have to spend a lot of time looking over my reports' shoulders and checking that they're doing their work well, you know, that they're not shipping bugs to production." }, { "i": 11, "speaker": "Lauren Tan", "text": "And so, I drew this chart because uh it's not it's not a very scientific chart, but like this is how I imagine myself and my journey through using agents." }, { "i": 12, "speaker": "Lauren Tan", "text": "So, you know, like fast forward or backward, uh or fast backward, uh fast backward like a year or so, when you know, nobody was or not many people were using agents to code, uh I think you you get into this mode where you are in very heavily in the loop with one or several, like a handful of agents, and you find yourself just constantly figuring uh you know, trying to understand what your agents are doing, uh and you're very very in the loop." }, { "i": 13, "speaker": "Lauren Tan", "text": "You're watching every single output. You are sitting there prompting, um and you really can't parallelize beyond that because you don't, you don't have that trust, right? You can't" }, { "i": 14, "speaker": "Lauren Tan", "text": "go to 100 agents, uh like spawn 100 agents when you don't even trust the output of one agent." }, { "i": 15, "speaker": "Lauren Tan", "text": "So, over the past 5 months, I feel like I've really been able to uh like ascend this trust curve, and now I'm at the point where uh I actually have this sounds kind of scary to say this, and it makes me sound like a rock star, but I promise I'm not." }, { "i": 16, "speaker": "Lauren Tan", "text": "But I actually have my agents now um auto-merging PRs for me, uh which is like a wild thing to say, but um like I woke up today, and there were like 20 PRs landed, and I just reviewed them on main, like they were already landed." }, { "i": 17, "speaker": "Lauren Tan", "text": "And they were good. Uh so, how did I get to that point? It's basically why I wanted to talk about today. And again, like yeah, feel free to jump in if you have questions, Colin. Um but uh Oh, yeah, of course I got to show this chart." }, { "i": 18, "speaker": "Lauren Tan", "text": "Uh where Don't trust to someone requested to control my computer. Probably won't do that. Uh but yeah, so this chart I think I I'm sharing this chart not to kind of like flex, but to kind of show like the journey." }, { "i": 19, "speaker": "Lauren Tan", "text": "Like so you can see like the curve like it sort of like inversely matches the contributions I've been able to land at Cursor. So, I joined 5 months ago. And 5 months ago like I you know, my first month I was like not very productive cuz I was you know, I was learning the code base." }, { "i": 20, "speaker": "Lauren Tan", "text": "Didn't know what the heck was going on. And as I got more confident in my agents, I've really been able to kind of ramp up my productivity. Uh and again, like yeah, last month I shipped 1,000 PRs, which is ridiculous." }, { "i": 21, "speaker": "Lauren Tan", "text": "Uh and then this month, we're only on the 12th, I'm already at like almost 800 PRs landed. Uh so the velocity is definitely high. And you will share a lot of you will definitely be questioning like how much of this code is actually good." }, { "i": 22, "speaker": "Lauren Tan", "text": "And I think yeah, like that's definitely fair to question. Um but yeah, I think I think if you set up your agents well, you can definitely get to a very similar level. Um and so I'm going to talk about how we do that." }, { "i": 23, "speaker": "Lauren Tan", "text": "Uh so for me, I think I'm curious like I guess Colin, your experience as well, but uh for me I think the most important skill that you should have in your toolbox when you work with agents is verification." }, { "i": 24, "speaker": "Lauren Tan", "text": "Uh and by verification I mean the ability for an agent to actually run the code uh or take CPU traces or heap snapshots or uh you know, open an iOS simulator, whatever, you know, however your application is exposed to users it can do the same thing and uh run it for real and actually test and verify it'll work." }, { "i": 25, "speaker": "Lauren Tan", "text": "Cuz that's the thing that really closes the loop. Uh it doesn't guarantee your agent writes good code uh but it allows them to at least write correct code uh which is a big really step forward for being able to trust your agent." }, { "i": 26, "speaker": "Lauren Tan", "text": "Um I will I can share one example that we have uh within Cursor uh oops where let me open this. Let me make this bigger. Uh let me go full screen." }, { "i": 27, "speaker": "Speaker 1", "text": "There you go." }, { "i": 28, "speaker": "Lauren Tan", "text": "Uh so for the Cursor's agent window, uh so this is actually an interesting story but uh when I joined Cursor five months ago uh they're actually uh well, I was supposed to join a different team." }, { "i": 29, "speaker": "Lauren Tan", "text": "I was supposed to join like the cloud agents team. Uh but then since I have a lot of experience working on React and agents window is a React application uh I was uh I was asked to basically help out with" }, { "i": 30, "speaker": "Lauren Tan", "text": "uh the agent window work. Uh but um there wasn't really a lot of like skills to help me. So, I just found myself that okay, uh Agent Swarm was going to launch in like a week." }, { "i": 31, "speaker": "Lauren Tan", "text": "All right, we have a really tight deadline. And um there was uh you know, I was just sitting there like okay, I'm going to open up the performance the Chrome DevTools and just like take a trace, look at it myself, and try to make sense of this flame graph." }, { "i": 32, "speaker": "Lauren Tan", "text": "And keep in mind, I was just like in my first week. So, I had no idea what I was looking at, no idea what you know, uh I mean, I had some idea, but you know, the code base was completely fresh to me." }, { "i": 33, "speaker": "Lauren Tan", "text": "Uh and I realized like my agent had no idea either, you know, like I would take a screenshot to try to download trace or send it to it and be like, \"Yeah, it kind of looks like this, you know.\" Uh and it would like confidently state like it's this thing." }, { "i": 34, "speaker": "Lauren Tan", "text": "And then I try to fix that. And it turns out that's not the actual thing. So, this was very very slow process." }, { "i": 35, "speaker": "Lauren Tan", "text": "And if you've ever done any like performance work yourself or you know, just even development with an agent where you don't have a verification skill, you are the verifier. All right, you you're the bottleneck. You you you tell your agent to do something, and then it goes off and writes some code." }, { "i": 36, "speaker": "Lauren Tan", "text": "Then you open up your, you know, local dev build, and then you start to say, \"Oh, you know, it doesn't work.\" Then you got to copy-paste screens you know, screenshots or console errors or whatever." }, { "i": 37, "speaker": "Lauren Tan", "text": "Uh and then your agent like slowly kind of like you know, works with that and then tries to understand it and um uh fix the thing. But then you're constantly just in the loop and and being a bottleneck. So, there's really no way to parallelize." }, { "i": 38, "speaker": "Lauren Tan", "text": "So, the glass goes like one of the first skills I built uh for Cursor. Uh and glass, by the way, is the code name for Agent Window that we use internally, but it's just Cursor, I guess." }, { "i": 39, "speaker": "Lauren Tan", "text": "Um and so, this skill uh is I guess that that the code itself is It's super interesting. Your agent can very easily make one for you. Uh where if if you're building an electron app or a web app or" }, { "i": 40, "speaker": "Lauren Tan", "text": "even iOS uh and applications, uh you can teach your agent how to use like the Chrome DevTools protocol or through uh Apple has some utilities as well for running the simulator and taking traces and controlling programmatically as well." }, { "i": 41, "speaker": "Lauren Tan", "text": "Uh so that's really useful. Uh but one thing I want actually want to talk about is uh the thing. Uh where is the readme?" }, { "i": 42, "speaker": "Lauren Tan", "text": "Uh so this skill comes with this very unique feature called or not a feature yet uh unique file called a feature map." }, { "i": 43, "speaker": "Lauren Tan", "text": "And so the story then is like I built this skill and so now the agent was able to uh actually run the agent window and take traces and whatnot. Uh but it had no idea what what the agent's window was." }, { "i": 44, "speaker": "Lauren Tan", "text": "So um you know, like someone would say like oh, the the left side bar is like laggy or something like that. Or you know, the right side the the PR tab is not working. And the agent would just be like in a flailing around. It would spend a lot of time trying to like look up the code and you know, where is this feature?" }, { "i": 45, "speaker": "Lauren Tan", "text": "How do I actually get to it on the UI? Which made it basically completely useless. Uh you know, like we would I would run this skill locally and you know, it would spawn a dev build uh but then it would just be churning like I just try to" }, { "i": 46, "speaker": "Lauren Tan", "text": "click here. It wouldn't know how to get to things. Um and it was just an awful experience. Uh so who was putting arrows on my screen?" }, { "i": 47, "speaker": "Lauren Tan", "text": "Um so uh yeah, this feature map has been really useful uh because it teaches the agent how to get to all of the features that you have." }, { "i": 48, "speaker": "Lauren Tan", "text": "Um and in P Stack, the plugin that I've made, uh if you search for P Stack cursor on Google, you'll find it." }, { "i": 49, "speaker": "Lauren Tan", "text": "Um but there is a create verification skill in that plugin where it actually helps you set up something like this for yourself." }, { "i": 50, "speaker": "Lauren Tan", "text": "Um including the feature map, so it will actually explore the code and build up this initial feature map that tells your agent how to get to all of the different features that you have." }, { "i": 51, "speaker": "Lauren Tan", "text": "Uh and this is extremely powerful because now that you have these user reports that come in, uh you can actually map even like a vague report or even a screenshot." }, { "i": 52, "speaker": "Lauren Tan", "text": "So we have this uh internally at Cursor where uh we have a Slack channel with, you know, lots of people giving us feedback on the agent's window and sidebar and whatnot." }, { "i": 53, "speaker": "Lauren Tan", "text": "Uh and often times the report is very bad, like very low quality, like someone just put Very often we get like a screenshot, like and then someone just says what is this? what is this? what, like what is this? >> [laughter] >> And you know, like that without this, your agent says, \"I have no clue.\"" }, { "i": 54, "speaker": "Lauren Tan", "text": "Right? But with a feature map like this, it has a lot more context and understanding of how to actually navigate, how to get to all of the different features. Uh so like, you know, example like I guess like the sidebar, like what is the sidebar?" }, { "i": 55, "speaker": "Lauren Tan", "text": "Uh you know, like all the different sub-features that are present in it. Um like from the user point of view, here's how to get to it, all the different keyboard shortcuts." }, { "i": 56, "speaker": "Lauren Tan", "text": "Uh even like the what do you call it? The DOM elements or yeah, like the attributes that you use for selecting things through the CDP uh are all there. So uh again, yeah, this is like really, really powerful uh for agents." }, { "i": 57, "speaker": "Lauren Tan", "text": "And Uh and a skill that create verification skill, but also a maintain verification skill. Uh so you can keep this up-to-date." }, { "i": 58, "speaker": "Speaker 1", "text": "Cool. Yeah, I was just going to ask how you created that. So, do you mind sharing a little bit more about that process in the context of P stack?" }, { "i": 59, "speaker": "Lauren Tan", "text": "And maybe just what P stack is for the folks who aren't familiar. Yeah, so P stack is pretty interesting because uh well, first of all, the name is kind of goofy." }, { "i": 60, "speaker": "Lauren Tan", "text": "Like the P the P in P stack is like potato potato sack. Um because I um so uh the there is a pretty uh famous person, Garry Tan, who is the CEO of Y Combinator." }, { "i": 61, "speaker": "Lauren Tan", "text": "And he's come up with this plugin called G stack. Uh Garry stack. And uh funnily enough, we share the last name. We have no relation." }, { "i": 62, "speaker": "Lauren Tan", "text": "Uh but I thought it'd be funny to kind of, you know, poke fun at Gary and make P stack my version of his plugin. Uh but kind of just tailor it to my own set of practical engineering practices." }, { "i": 63, "speaker": "Lauren Tan", "text": "Uh but I honestly actually never set out to build P stack. Uh it just started with a bunch of skills. All right? Like I started with that control class skill." }, { "i": 64, "speaker": "Lauren Tan", "text": "And then I started with another skill like called how, which I also noticed through like observing agents. Um so like, you know, in the early days of me, you know, trying to climb this ladder, I was like super in the loop." }, { "i": 65, "speaker": "Lauren Tan", "text": "And I was basically nitpicking my agents to an extreme degree. I was like I would tell it um you know, this feature has stopped working. Here's a bug report. Like, why isn't it working?" }, { "i": 66, "speaker": "Lauren Tan", "text": "And very often the agent would just like confidently state like, \"Oh, it has to be this, right? It has to be this thing.\" And I noticed like when I looked at the actual tool calls I noticed it wasn't actually reading the code that they should be using." }, { "i": 67, "speaker": "Lauren Tan", "text": "And that made me just extremely suspicious. And at that point I was just like, I'm not going to trust any agent anymore cuz it's just it's just completely hallucinating." }, { "i": 68, "speaker": "Lauren Tan", "text": "And I think I think it's very easy to just, you know, like build up that distrust and not kind of feel helpless like, you know, you don't know how to help your agents succeed." }, { "i": 69, "speaker": "Lauren Tan", "text": "But like again, I think the the management analogy is super helpful because like imagine if you were a manager of an engineering team and you had an engineer in your team who was a really good coder, no business contacts whatsoever, you know, they just you just hired them and they onboarded, you know, like uh 5 seconds ago." }, { "i": 70, "speaker": "Lauren Tan", "text": "Uh and so how do you actually teach that person to be effective? So how you do that is through skill. Uh skill being just, you know, it's just markdown, right? But you know, it codes a lot of information, instructions, a lot of uh you can really draw out a lot of" }, { "i": 71, "speaker": "Lauren Tan", "text": "intelligence from an agent by well, some people on Twitter call it like, you know, pull the agent to a different latent space." }, { "i": 72, "speaker": "Lauren Tan", "text": "Which is kind of like a fancy way of just saying like since uh you know, LLMs are sort of like they predict the next token." }, { "i": 73, "speaker": "Lauren Tan", "text": "Uh when you give it some high-quality tokens uh to begin with then, you know, it can kind of pattern match on like a higher space that's, you know, smarter. Um so that's like a very interesting model there." }, { "i": 74, "speaker": "Lauren Tan", "text": "But yeah, I built P stack very, very incrementally. Uh so uh started with just really observing how agents, you know, all the failed different failure modes of those agents were having. And every time I saw that, I just okay, I'm just going to make that a skill." }, { "i": 75, "speaker": "Lauren Tan", "text": "All right, like stop hallucinating, actually go and search up look up the code. Use a lot of sub agents. Uh and yeah, stop guessing. Yeah, that makes sense." }, { "i": 76, "speaker": "Speaker 1", "text": "One kind of follow question here both from myself and from a bunch of people in the chat. So I guess it's two parts. So one is like how do you maintain these skills? So like the product changes over time. Obviously there's a lot of people who are shipping against the code base." }, { "i": 77, "speaker": "Speaker 1", "text": "So how do these skills get maintained? And then second to that is like how do you know when your verification is good enough? Like in you know you can trust that the verification loops that you've built are going to I guess you trust that that would put when they're done." }, { "i": 78, "speaker": "Lauren Tan", "text": "Yeah, maybe I'll talk about I think I have something related. Maybe I'll start with this one first. So like how do I maintain these skills?" }, { "i": 79, "speaker": "Lauren Tan", "text": "So if you're not familiar with this concept and eval [clears throat] is essentially like a way to I think the mental model I have is like it's like a unit test for an agent. And you can actually make your own evals." }, { "i": 80, "speaker": "Lauren Tan", "text": "You don't need like a special framework for them. You can build you can you can build one depending on like you know how scientific and how rigorous you want to be." }, { "i": 81, "speaker": "Speaker 1", "text": "My screen is red. Yeah, there's a little button." }, { "i": 82, "speaker": "Lauren Tan", "text": "Sorry. Are you disabling the drawing or something?" }, { "i": 83, "speaker": "Speaker 1", "text": "I can't see my screen. Yeah, sorry. If you guys could not draw on the screen that would be great but there's a little button in the Yeah, the little drop down. I like clear. Yeah. Okay, yeah. You got it." }, { "i": 84, "speaker": "Speaker 1", "text": "Perfect." }, { "i": 85, "speaker": "Lauren Tan", "text": "Um Yeah, so evals are a way to unit test your skills basically." }, { "i": 86, "speaker": "Lauren Tan", "text": "And actually in Pystack we ship under pilot mode there's a playbook if you search for it called eval playbook. And it's uh It's like not it's actually pretty pretty rigorous the way it's done." }, { "i": 87, "speaker": "Lauren Tan", "text": "But um essentially what I do is I spawn a lot of different sub agents. I have like my main coordinator agent uh come up with a rubric for uh what I want the skill to do." }, { "i": 88, "speaker": "Lauren Tan", "text": "Um and then it spawns all these sub agents and it it creates individual directories for them uh which are cleverly named to not let the sub agent know that it's being evaluated because uh agents can actually tell and when they do they change their behavior." }, { "i": 89, "speaker": "Lauren Tan", "text": "Uh but it does a bunch of stuff like that to um essentially yeah, like test whether or not the skill I'm making or changing is actually doing what I think it does. Um and one of the really nice things about Cursor is that we are we have we support so many different models." }, { "i": 90, "speaker": "Lauren Tan", "text": "So you can actually eval your skill across all sorts of different models. Um and you know, get a sense of how well it performs across that different matrix. Um especially for the models that you use. Uh so I do this a lot." }, { "i": 91, "speaker": "Lauren Tan", "text": "Every time I I modify a skill, I will run one of these. Uh like the eval playbook uh and make sure that you know, it's actually leading to a result I want. Uh but I will say like maintaining skills is actually pretty hard." }, { "i": 92, "speaker": "Lauren Tan", "text": "Uh it requires I think a lot of taste and observation. So you kind of need to be very good at being a backseat driver. You know what I mean?" }, { "i": 93, "speaker": "Lauren Tan", "text": "Like if you do pair if you've ever done pair programming for example uh and you watch a coworker code and you just like you you could probably do this better. You know, you could do you know, you like why did you not do this, right? You ask a lot of questions to your coworker." }, { "i": 94, "speaker": "Lauren Tan", "text": "And it's kind of a similar thing here. You like you don't want to just be a passive observer of agent. You want to be very in the driver seat in the initial stages when you're building up your own set of skills." }, { "i": 95, "speaker": "Lauren Tan", "text": "Uh you know, obviously you can use something like Pystack, but if you're building your own set of skills, it's very I think, you know, opening up all the tool calls and like reading the code and reading all the uh the agent behavior and their thinking blocks is uh a really great way to see where they fail, right?" }, { "i": 96, "speaker": "Lauren Tan", "text": "Like what what you know, where are they failing? And then you can go and build a skill for that." }, { "i": 97, "speaker": "Lauren Tan", "text": "And then with verification, how you test it is it's I think it's also a very similar iteration loop uh where, you know, like I actually did the same process for verifying the verification skill where I actually got um" }, { "i": 98, "speaker": "Lauren Tan", "text": "So, one thing that's interesting about evals is that you can sort of hill climb them, meaning that uh your eval can produce a score, right?" }, { "i": 99, "speaker": "Lauren Tan", "text": "Uh a score that you can get your coordinator to produce, uh but also uh you can have a judge agent of a different model to uh kind of cross-reference and make sure that the first model is not being biased, right? The model that's judging all of the sub agents that are running the thing." }, { "i": 100, "speaker": "Lauren Tan", "text": "Uh but you can also like hill climb." }, { "i": 101, "speaker": "Lauren Tan", "text": "So, meaning that you can you can use like /loop in cursor and you can say, \"Okay, keep looping on this eval, right?\" Until everything is 10 out of 10.\" as an example." }, { "i": 102, "speaker": "Lauren Tan", "text": "Uh and I did basically the same approach with the control skill. And so, I kind of it was very it was very hands-off, actually. Uh so, you know, I uh I kind of built I built that skill that way, like the CLI in that skill." }, { "i": 103, "speaker": "Lauren Tan", "text": "Um and over time it's gotten really good. Uh but yeah, it was definitely not super smooth at the beginning. It required a lot of iteration. And I think there's an analogy here for me, which is um Well, I make this analogy later in a different slide on my drawing here." }, { "i": 104, "speaker": "Lauren Tan", "text": "Uh but I think of it like uh you know, as an engineer now, you're sort of more like you uh like maybe a manager or the analogy I like is like you're like a a chef in a restaurant. Uh you know, you're the head chef. Uh you're not cooking all the food yourself anymore." }, { "i": 105, "speaker": "Lauren Tan", "text": "You have a team of cooks, right? You have a line cooks, you have a sous chef, you have you know, all these different stations. Um and it's your job to set the environment." }, { "i": 106, "speaker": "Lauren Tan", "text": "You know, you you you're in charge of setting up the kitchen. You're in charge of you know, like giving tasks to different people. So um yeah, it's a very interesting way of working. Uh but yeah, that's that's how I basically built uh these verification skills." }, { "i": 107, "speaker": "Speaker 1", "text": "Yeah, just just one thought there and like I try to go try to go one layer deeper. So, are you let's say we wanted to build um an eval or a skill for something and we wanted to kind of get better on its own, which is what I think you're suggesting." }, { "i": 108, "speaker": "Speaker 1", "text": "Uh are you doing that in like a work tree kind of isolated with like the sub agents and then the reviewer agents and all that? Is it happening like in some type of cloud-hosted environment? Like what's the more practical steps? If I wanted to go do this uh and like set up a verification system for something," }, { "i": 109, "speaker": "Speaker 1", "text": "what would I what would I do or where would I start?" }, { "i": 110, "speaker": "Lauren Tan", "text": "Um I think that uh the best way to start is local because you can observe. You can definitely observe what your agents are doing." }, { "i": 111, "speaker": "Lauren Tan", "text": "So, uh if you're building a verification skill for yourself, uh I would definitely start local and just have your agent bring up the application, whether it's like a CLI or a desktop app or whatever. And so, you can actually observe, right?" }, { "i": 112, "speaker": "Lauren Tan", "text": "You can see how the agent is interacting with the application. You can see it, you know, how it calls like the different APIs that allow it to interact with the application. Um but uh for me personally, uh I have basically been kind of all in mostly all in on cloud agents because they're extremely powerful." }, { "i": 113, "speaker": "Lauren Tan", "text": "Uh and the really powerful thing about Cursor is the" }, { "i": 114, "speaker": "Lauren Tan", "text": "cloud agents actually, where if you spend a little bit of time setting up your environment, these control skills, these verification skills pay a huge amount of dividend because it's not just something that makes you as a single engineer better, it actually levels up your whole team uh and even your whole company because uh you can actually start thinking about cloud agents and start thinking about automations that automatically do things like uh I'll get into it uh where where you know, for example, like I talk a lot about this agent we have called Benny, right?" }, { "i": 115, "speaker": "Lauren Tan", "text": "Who who uh you know, takes all of the bug reports that we get and it automatically goes off in the cloud, opens up a cloud uh it's, you know, it's desktop" }, { "i": 116, "speaker": "Lauren Tan", "text": "desktop, it runs Cursor in its own computer, and it uses the same control skills to interact with the application and try to reproduce the bug uh or the user report, right? And this is so so powerful because at once I can immediately I get so much information from this automatically." }, { "i": 117, "speaker": "Lauren Tan", "text": "Like here in this example, you can see that uh the Benny actually reproduces the bug, uh but it's already fixed on main." }, { "i": 118, "speaker": "Lauren Tan", "text": "So, it actually confirms that we fixed this problem already, and all I need to do is just release another build of Cursor." }, { "i": 119, "speaker": "Lauren Tan", "text": "Uh so, that's like huge information there that I didn't have to go off and sit with an agent, you know, and spend an hour trying to figure out is this fixed or is this not fixed? So, you you you can save so much time, uh but you know, everybody on my team benefits from this." }, { "i": 120, "speaker": "Lauren Tan", "text": "Everybody at the company benefits from this. Uh so, definitely think that you know, keeping these using cloud agents is super powerful. Uh but yeah, it's like a journey. You have to trust it first, right?" }, { "i": 121, "speaker": "Lauren Tan", "text": "Before you get to this point, and that's it goes back to what I was saying here where you know, it's very hard it's almost impossible, and I would definitely encourage you not to try to jump from you know, like if you're still in this zone, you don't want to jump to like I'm going to spawn a hundred or a thousand of cloud agents right now" }, { "i": 122, "speaker": "Lauren Tan", "text": "because you're just going to waste a lot of tokens and it's going to be extremely expensive." }, { "i": 123, "speaker": "Speaker 1", "text": ">> Yeah, so just to kind of recap so far, basically the if we wanted to go on the journey that you've kind of gone on, it would be just start with verification, building some skills and some ways ways of determining that the agents are producing at least like correct code or they, like you said, whether it's good code or not is maybe a separate question, but like it's it's technically solving the problem by looking at you know, stack traces, looking at you know, the actual behavior in the app and so on." }, { "i": 124, "speaker": "Speaker 1", "text": "And once we trust it locally, then we can start to think about scaling into the cloud and running more agents that are picking up signals I guess on their own, right? So, whether that's like a" }, { "i": 125, "speaker": "Speaker 1", "text": "bug report that comes in or something and they can go and pick it up and solve the problem and give us back a PR. And then maybe the last step is like auto merging the PRs, which is where you're at. >> Maybe not where everyone is." }, { "i": 126, "speaker": "Speaker 1", "text": ">> Yeah, yeah. >> And then reviewing them on main, but is that about right?" }, { "i": 127, "speaker": "Lauren Tan", "text": ">> Yeah, exactly. I think yeah, that's why I drew this this this this curve, right? Because that this basically describes my journey of you know, when I started barely could use a couple agents and I was just observing every single thing." }, { "i": 128, "speaker": "Lauren Tan", "text": "I think there's really no shortcut for going from here to there because this is really about your personal level of trust in the agents. Right? Um obviously, you know, as an engineer, you don't want to just sloppy code into production." }, { "i": 129, "speaker": "Lauren Tan", "text": "So, how do you actually build up that trust? Takes um a lot of uh I guess taste and judgment. Um but uh you know, like I think plugins like Pstack definitely kind of help you get uh get up to speed much quicker." }, { "i": 130, "speaker": "Lauren Tan", "text": "Uh and so, I guess it's like if you trust me and you trust Pstack, then in by extension, you can maybe trust your agents. But if you don't trust me and I definitely would not encourage people to blindly trust me." }, { "i": 131, "speaker": "Lauren Tan", "text": "Uh uh you know, if you build up your own set of skills that you can obviously, you know, take a look at Pstack and kind of fork it, make it your own, improve the skills, definitely encourage that." }, { "i": 132, "speaker": "Lauren Tan", "text": "Uh but for me, it's really all about it just keeps coming back to trust. You know, every one of us here in this chat have a different standard for engineering. Uh and there are different things that are important for us in our code base." }, { "i": 133, "speaker": "Lauren Tan", "text": "And uh when you are able to encode all of that into skills and you can verify that your agents actually doing them, that allows you to really kind of ascend this curve. And um uh you know, start automating things." }, { "i": 134, "speaker": "Lauren Tan", "text": "Uh there's another piece I wanted to talk about um if there's >> Yeah, go for it. I'll I'll pick up more questions as I go, but >> Yeah, I think there's a third part to this which I haven't talked about yet, which is kind of an interesting one, which is like refactoring and rewriting." }, { "i": 135, "speaker": "Lauren Tan", "text": "Like one of the uh I guess most controversial one of the most controversial topics in the industry, I think, is like should you rewrite your app or not?" }, { "i": 136, "speaker": "Lauren Tan", "text": "Um because I think engineers are very prone to this where especially when you join a company, you come in and you see like the code base and you're like, \"Man, this is like who wrote this code?\" You know, it's it's terrible. I want to rewrite the whole thing." }, { "i": 137, "speaker": "Lauren Tan", "text": "There is a very common inclination and I think a lot of you know, before agents, um and I guess arguably even now, people will definitely discourage you from rewriting stuff. But I'm actually here to make a case for why you might want to consider it." }, { "i": 138, "speaker": "Lauren Tan", "text": "Um because I think it really depends." }, { "i": 139, "speaker": "Lauren Tan", "text": "Uh you know, brownfield applications I think are actually in a pretty good spot, especially if they're set up well already." }, { "i": 140, "speaker": "Lauren Tan", "text": "Uh and like recently I've been talking to some people about uh you know, I I I was just observing. I I just noticed this parallel, which is that a lot of big tech company problems are now everybody's problems." }, { "i": 141, "speaker": "Lauren Tan", "text": "Um and the big tech company problem, you know, like when I was working at Meta, like we had this giant monorepo. We had like, I don't know, tens of thousands of engineers just you know, like banging on" }, { "i": 142, "speaker": "Lauren Tan", "text": "their keyboards and and shipping code. And a lot of really great engineers at Meta, uh but uh I'll say like, you know, you'll be surprised that the code quality is actually not that good." }, { "i": 143, "speaker": "Lauren Tan", "text": "Um and so I often joke that like, you know, before AI sloth, we had human sloth." }, { "i": 144, "speaker": "Lauren Tan", "text": "Um and so uh you know, I think a lot of big tech infra, like uh like what Meta has or Google, you know, you know, really big tech companies, are actually designed for that where you you're sort of like you're catering to the the you know, like uh and this sounds so bad to say, but like the the least capable engineer on your team, right?" }, { "i": 145, "speaker": "Lauren Tan", "text": "You build you build frameworks, you build conventions, you build guardrails, you know, you restrict credentials so that, you know, your intern doesn't wipe your production database. Um there's uh you know, if you have that level of infra already, I think your agents can actually already do a very solid job." }, { "i": 146, "speaker": "Lauren Tan", "text": "Right? Because they have the the guardrails are already in place for agents to not cause havoc or not cause too much havoc in your code base. Um and you can always add more, you know, guardrails." }, { "i": 147, "speaker": "Lauren Tan", "text": "Uh but I think like greenfield applications, especially are, you know, like the brand new applications are like the biggest risk in my opinion. Uh and also the greatest opportunity." }, { "i": 148, "speaker": "Lauren Tan", "text": "Because, you know, if you write code a project uh a prototype um like we did for Grokbot, you know, Grokbot was spun up very very very quickly." }, { "i": 149, "speaker": "Lauren Tan", "text": "Um and if you if you haven't heard of Grokbot, it's like our our new application we just launched yesterday. Uh it's it's really cool. Uh lets you orchestrate your agents like individual agents that have their own identity and you can kind of orchestrate them." }, { "i": 150, "speaker": "Lauren Tan", "text": "It's super cool, definitely check it out. Um but yeah, that was it's like a very it was a very greenfield application like most prototypes are. So it was like vibe coded very quickly." }, { "i": 151, "speaker": "Lauren Tan", "text": "Humans were not reading the code at all. And uh I had this tweet recently uh where I said something about organic architecture. Um Maybe I'll find it." }, { "i": 152, "speaker": "Lauren Tan", "text": "Uh but the idea is that uh when you have a completely vibe coded application, you essentially have no guardrails whatsoever. So uh your agents when you give them a task, they will just solve it in whatever method is the most convenient." }, { "i": 153, "speaker": "Lauren Tan", "text": "And over time you get into this uh situation where you have a code base that is spiraling out of control because you don't understand it. Uh your agents understand it, I guess, in a way, but like they've built something that is you know, optimized for shortcuts." }, { "i": 154, "speaker": "Lauren Tan", "text": "Uh and uh you know, it will you will suffer you have a lot of issues with that application. Uh so, I think starting your code base with uh like very strong guardrails is very much needed. Uh because like when you have a code base that you can trust, right?" }, { "i": 155, "speaker": "Lauren Tan", "text": "When you have guardrails that actually help you uh uh help your agents, right? Good code. You can get into the you know like into this part of the curve where I I where I like I I said, you know, I woke up today and I had like 20 PRs merged uh by my agents." }, { "i": 156, "speaker": "Lauren Tan", "text": "And that's because I invested a lot of time uh over 600 PRs I I I calculated yesterday uh I when I refactored all of GrokBot to this new architecture that I've been building. Um and yeah, I I've gotten to a point where I I don't really look I really don't look at the code anymore." }, { "i": 157, "speaker": "Lauren Tan", "text": "And um I say that not just, you know, to sell you tokens, but because it I you know, it it took a lot of tokens to get the code base to this point where I no longer have to look at it. Uh but I'm very excited because you know, of the potential where, you know, it's not just it this doesn't just benefit me." }, { "i": 158, "speaker": "Lauren Tan", "text": "It benefits everyone contributing to GrokBot. And it also empowers, you know, designers and product managers and you know, even GTM people to add features to GrokBot. And I don't have to worry, you know, I don't have to wake up at night in the middle of the night and worry like, oh someone's just merged a perf regression." }, { "i": 159, "speaker": "Lauren Tan", "text": "Right? I have a ton of constraints and CI is like it's actually very annoying to write code in GrokBot, but like agents absorb all of that annoyance. Um but yeah, I'm happy to talk about what exactly that is. Um >> Yeah, I think one question um >> Yeah." }, { "i": 160, "speaker": "Speaker 1", "text": ">> before we get into this part here is just around that element of like what your CI looks like or maybe some of the constraints and then also like the average PR size." }, { "i": 161, "speaker": "Speaker 1", "text": "I saw a question about that earlier. Just to give people a you know kind of a glance. It doesn't have to be like mathematically average but just a you know like what generally the size of the a PR is um if it's only a couple lines of code or you know um yeah." }, { "i": 162, "speaker": "Lauren Tan", "text": ">> Um I think it depends." }, { "i": 163, "speaker": "Speaker 1", "text": "Um maybe I'm trying to do this in a way where I'm not going to like >> Yeah you don't have to share the actual number if you like the actual average just >> this is fine." }, { "i": 164, "speaker": "Lauren Tan", "text": ">> Benchmark. >> But like we have So okay. This is not that interesting but uh well fun fact is that virtualization in GrokBot and in uh Cursor is actually powered by uh Pretext uh which is a sort of new library that someone's built." }, { "i": 165, "speaker": "Lauren Tan", "text": "Um that's really interesting. You should check it out. But that's not really that important. Uh I think the average PR size I actually don't know I probably I don't know if I want to click on these." }, { "i": 166, "speaker": "Lauren Tan", "text": "Uh I probably can but I would say like they can range anywhere from a few hundred lines or 50 lines to like a thousand depending on what the thing is doing. Uh so like here I'm actually like deleting a bunch of files so I expect that it's just this like mostly deletion." }, { "i": 167, "speaker": "Lauren Tan", "text": "Uh but yeah it kind of varies. There's no like Yeah." }, { "i": 168, "speaker": "Speaker 1", "text": ">> like a hard cap for how many of them are like 50 line PRs?" }, { "i": 169, "speaker": "Lauren Tan", "text": ">> cap. There's definitely no hard cap but I do encourage my agents to split up their work into multiple PRs." }, { "i": 170, "speaker": "Lauren Tan", "text": "Uh I do that mostly because uh I like I like the idea of I guess maybe this is much harder to do now is in the world of agents and you have like so many commits." }, { "i": 171, "speaker": "Lauren Tan", "text": "But I like the idea that you know the Git history is a very rich source of context. Uh and I like each PR to sort of atomically describe what that small piece of thing is doing." }, { "i": 172, "speaker": "Lauren Tan", "text": "Which also makes it easier for me to revert changes and like figure out, you know, oh, I shipped a bug and it's right here, right? It's not in this 40,000 line PR where you know, who knows what landed in there." }, { "i": 173, "speaker": "Lauren Tan", "text": "Uh, but yeah, I don't have a hard cap on PR size. >> Cool." }, { "i": 174, "speaker": "Speaker 1", "text": "And then, um, yeah, also quick question on like CI. So, again, you don't have to go into like, uh, the screen share of like your CI does, but just generally, would you describe" }, { "i": 175, "speaker": "Speaker 1", "text": "what the CI kind of looks like uh, or how strict it is?" }, { "i": 176, "speaker": "Lauren Tan", "text": ">> Uh, yeah, so uh, well, specifically for Grokbot, so Dune is the sort of cheeky code name for the architecture that we built for Grokbot. Um, the CI looks pretty annoying because there's checks for everything." }, { "i": 177, "speaker": "Lauren Tan", "text": "So, like literally, I have um, uh, well, if you've read any React for example, you know, you know that one of the biggest footguns in React is use effect." }, { "i": 178, "speaker": "Lauren Tan", "text": "Uh, so in uh, Dune and in Grokbot, we've banned use effect." }, { "i": 179, "speaker": "Lauren Tan", "text": "So, Dune is just you can think of the mental model of what Dune is uh, you can kind of think of it as like Next.js for uh, Electron apps and it's designed for agents to write." }, { "i": 180, "speaker": "Lauren Tan", "text": "Uh, and it's like custom for, you know, our agent powered applications. Um, so the CI checks are very like specific to that. Like, you know, don't use use effect. It's it's it's banned." }, { "i": 181, "speaker": "Lauren Tan", "text": "Like, CI will fail uh, and yell at you. We have like some of the more interesting ones that people might raise eyebrows is like I actually banned code comments as well. Uh, which is very interesting." }, { "i": 182, "speaker": "Lauren Tan", "text": "Um, but I've noticed that 99% of the time agents just write code comments that kind of describe some historical thing that is actually totally irrelevant to the code. Um like it will often say like, you know, \"Oh, Lauren said you should never do this.\"" }, { "i": 183, "speaker": "Lauren Tan", "text": "And it's now in the code comment. I'm like, \"What? Like, why what that was I didn't say that as\" like a durable, you know, global rule. I just meant like your PR sucks and you should change that part." }, { "i": 184, "speaker": "Lauren Tan", "text": "Like, agents don't really understand this that well, surprisingly. Uh and or they kind of assume too much and kind of do things in like very stupid ways. So, like, yeah, we just ban everything everything you can imagine like the agents are bad at, we ban." }, { "i": 185, "speaker": "Lauren Tan", "text": "Uh so, one example that we actually suffer a lot in the agents window is we have uh you know, you know, if you've used agents window, you've definitely seen performance issues and, you know, we're constantly trying to fix them." }, { "i": 186, "speaker": "Lauren Tan", "text": "Uh but it's like a it's a never-ending struggle because there's so many pull requests that get merged. Every one of them could just regress performance or stability or reliability. Uh you know, the agent's window doesn't have this architecture yet. I plan to do bring this learning back there and kind of refactor everything there." }, { "i": 187, "speaker": "Lauren Tan", "text": "Uh but uh it just regresses super often uh because uh there's there's one example is like we have very poor um isolation between processes. So, like, on you know, on on Electron, you have a renderer thread that renders your UI." }, { "i": 188, "speaker": "Lauren Tan", "text": "But you also have like a main thread that you can run other code that, you know, doesn't need to block the renderer. Um but we do a poor job of separating those things. And so, often times you just accidentally have code that gets pulled into running on the renderer thread." }, { "i": 189, "speaker": "Lauren Tan", "text": "Uh and then all of a sudden you're competing with the renderer that you know, that has a very if you want like 60 FPS, you have to every frame that gets drawn has to be done in 16 ms. So, very very small, you know, deadline per frame." }, { "i": 190, "speaker": "Lauren Tan", "text": "Uh, if you want, you know, a very smooth product. Uh, and when you start building, bringing in, accidentally bringing in, you know, things that are like very computationally heavy or they have a lot of IO, uh, then you just get into like a lot of jank." }, { "i": 191, "speaker": "Lauren Tan", "text": "And your FPS really drops, you start, uh, you know, losing frames, you get long tasks that take more than 16 milliseconds and you just get this really choppy experience." }, { "i": 192, "speaker": "Lauren Tan", "text": "So, all of those patterns that we've learned basically building Electron apps, we've encoded into this framework. And it becomes like a hard failure." }, { "i": 193, "speaker": "Lauren Tan", "text": "So, I literally in Grokbot, we literally have a directory called electron main, electron renderer, and we have, uh, import, uh, CI, I guess, where we actually check the dependency graph to make sure you're not accidentally importing code from one directory to another." }, { "i": 194, "speaker": "Lauren Tan", "text": "Uh, so that's enforced by the CI, um, as well as Bobby, uh, which is our cursor, um, like code review tool that runs on CI. you know, in our agents.md, it's everywhere." }, { "i": 195, "speaker": "Lauren Tan", "text": "Like, so I I I I um I have this thing here where I talk about like, um, you know, like there are multiple layers, I think, for building a good code base. Uh, obviously the code base is one where, uh, if you have an architecture like this, where it's extremely strict, uh, you know, the the way to build features is very conventional." }, { "i": 196, "speaker": "Lauren Tan", "text": "That's like the strongest strongest level of enforcement because agents just love to copy existing patterns. So, uh, one example of this in Grokbot is like we have this these concepts called like a feature and we have entry points and transcript cards." }, { "i": 197, "speaker": "Lauren Tan", "text": "Like, all you know, the cards that you see in the chat. These are all like nouns, I guess, in the framework. And so there's a very conventional way of creating them. And so like a feature is all in a single directory as an example, and so all of" }, { "i": 198, "speaker": "Lauren Tan", "text": "the code that contributes to that feature lives in one directory. So, it's all co-located in one place. Makes it super easy, you know, agents don't have to like grep around and try to figure About like where all the things are." }, { "i": 199, "speaker": "Lauren Tan", "text": "It just looks at the feature and like, \"Oh, okay, I'm working on the onboarding feature in GrokBot.\" I'm just going to work in this directory.\" And for 80% of the work, it's mostly just very encapsulated there." }, { "i": 200, "speaker": "Lauren Tan", "text": ">> [clears throat] >> But like it's like very it's like designed again for, you know, like the dumbest agent. Like, you don't have to think. All right, the the the One of the key principles I have for this framework is like the shortest the shortest path is the best path." }, { "i": 201, "speaker": "Lauren Tan", "text": "So, because that plays exactly to how agents love to write code. Like, they like to take shortcuts, really. You know, they'll they'll find the quickest way to solve the problem. So, why not make that the best way to solve the problem?" }, { "i": 202, "speaker": "Lauren Tan", "text": "So, I probably won't get into all the specific details. And this framework is really more of a collection of ideas and principles rather than something that will open source." }, { "i": 203, "speaker": "Lauren Tan", "text": "You can you can, you know, screenshot this, I guess, if you want and tell your agent to do some build build something like this for you, too. Um yeah, but it's really all about the layers." }, { "i": 204, "speaker": "Lauren Tan", "text": "You know, like the code base is one part with features and directories and, you know, import blocking dependencies that shouldn't be imported. But and and it all enforces that and static analysis. So, like there's CI checks." }, { "i": 205, "speaker": "Lauren Tan", "text": "We have a lot of lints for bad patterns that we observe. Compiler diagnostics. There's also rules in bug bot which are, um, I think like three, four, five are more soft, right?" }, { "i": 206, "speaker": "Lauren Tan", "text": "These two actually make make CI red, right? So, that you know, there's a hard constraint where the agent can just write crappy code. For rules and skills and bug bot, your agents can still forget, all right?" }, { "i": 207, "speaker": "Lauren Tan", "text": "You can still or it may not always consistently apply them. So, I like to layer them, but I don't I don't like to rely on them as the only source of enforcement because it's very very soft, right?" }, { "i": 208, "speaker": "Lauren Tan", "text": "And if you if you only have rules and bug bot and skills and a style guide for your code, you will it's only a matter of time before your code base looks like complete trash. I'm sorry to say that, but I definitely recommend, yeah, like you know, investing in, you know, things that can be hard enforced, right?" }, { "i": 209, "speaker": "Lauren Tan", "text": "And that's this why, you know, maybe uh, the choice of tech stack that you use is also very important. Um, like I think for example, Rust is sort of making, you know, it's like getting super popular again, uh, because the compiler is so strict, right?" }, { "i": 210, "speaker": "Lauren Tan", "text": "The compiler enforces so many different things, you know, there's a borrow checker that you have to appease. And if as long as you make sure your agents don't write unsafe code blocks, uh, you can more or less feel somewhat confident that if the code compiles, it probably works and it's good." }, { "i": 211, "speaker": "Lauren Tan", "text": "Uh, but you see, it gives you that level of trust and confidence that you as a human engineer no longer need to go and check it yourself. You you know, you you rely on code and static analysis to actually make that, uh, a lot smoother." }, { "i": 212, "speaker": "Lauren Tan", "text": "Um, and I guess the worst part the worst place to be in is if you are stuck in code review land where you actually enforce all of the constraints, the invariants in your code base by literally the human person saying, you know, reading the code and be like, \"Okay, you should not do this.\"" }, { "i": 213, "speaker": "Lauren Tan", "text": "Right? Every time you have to do that, you should consider that as a code smell, like an anti-pattern. And you should say, \"Okay, instead of me commenting on the PR, how do I turn this into a hard rule?\"" }, { "i": 214, "speaker": "Lauren Tan", "text": "Right? How do I turn this into a lint rule? How do I turn this into a CI failure? Or how do I even categorically eliminate this problem uh entirely? Uh I can talk about another migration I've done, but I'll probably pause here." }, { "i": 215, "speaker": "Speaker 1", "text": ">> Sure." }, { "i": 216, "speaker": "Lauren Tan", "text": "Yeah, I feel like that's that's where I am, to be honest, is is what you're describing right now, which is that like I don't have all of these rules. Uh so I have some things to go do after this session in terms of being able to scale my agents." }, { "i": 217, "speaker": "Lauren Tan", "text": "I'm I'm definitely on like the uh you know, maybe a couple of parallel ones locally staged, so like two to three locally. And I'm sure most people here are on the same, so uh yeah, I know we only have a couple minutes left. Uh Lauren, was there anything else that you wanted to to highlight?" }, { "i": 218, "speaker": "Speaker 1", "text": "I obviously there's lots of questions, so I can grab more, but I want to give you a few minutes if there's anything else you want to talk about. Okay, cool. Uh one question that I had uh a couple of times came up a couple times was just around like token usage." }, { "i": 219, "speaker": "Speaker 1", "text": "So the question is like is what you're describing a realistic thing for people who are on you know, uh a normal set of token usage, they don't have, you know, basically unlimited tokens uh to work with. Well, I think that's a really good point." }, { "i": 220, "speaker": "Speaker 1", "text": "I mean, like obviously, you know, I work at an AI lab where we have unlimited tokens, so uh I definitely cannot say that, you know, this is something you know, everyone should do in the exact same way that I did it." }, { "i": 221, "speaker": "Speaker 1", "text": "I think it's possible to get to this point without, you know, breaking the bank." }, { "i": 222, "speaker": "Lauren Tan", "text": "But, you know, if you are like an engineering leader or, you know, you're you're you have a startup that you lead, um I think the main is a question of ROI." }, { "i": 223, "speaker": "Speaker 1", "text": "Um and it's like yes, you spend a lot of money on tokens in the upfront stage. You know, you like refactoring a code base is going to take a lot of tokens. Uh adding all these things uh is going to take a bunch of tokens." }, { "i": 224, "speaker": "Speaker 1", "text": ">> Okay, cool. Uh one question that I had uh a couple of came up a couple times was just around like token usage. So the the question is like is is what you're describing a realistic thing for people who are on you know, uh a normal set of token usage, they don't have, you know, basically unlimited" }, { "i": 225, "speaker": "Speaker 1", "text": "tokens uh to work with." }, { "i": 226, "speaker": "Lauren Tan", "text": "Well, I think that's a really good point. I mean, like obviously, you know, I work at a AI lab where we have unlimited tokens, so uh I definitely cannot say that, you know, this is something everyone should do in the exact same way that I did it." }, { "i": 227, "speaker": "Lauren Tan", "text": "I think it's possible to get to this point without, you know, breaking the bank. But, you know, if you are like an engineering leader or, you know, you're you're you have a startup that you lead, um I think the main is a question of ROI." }, { "i": 228, "speaker": "Lauren Tan", "text": "Um and it's like yes, you spend a lot of money on tokens in the upfront stage. You know, you like refactoring a code base is going to take a lot of tokens. Uh adding all these things uh is going to take a bunch of tokens." }, { "i": 229, "speaker": "Lauren Tan", "text": "But, if we're heading to a world where agents are writing all the code, and you know, you want to be very lean, right? You don't want to have to hire, you don't want to be, you don't want to become like meta, right?" }, { "i": 230, "speaker": "Lauren Tan", "text": "Like I mean like in terms of you don't want to become a 10,000-person engineering org because I mean, that's a cool problem to have, but also you need you have so much overhead." }, { "i": 231, "speaker": "Lauren Tan", "text": "There's like planning, you know, like it's uh personally I wouldn't uh it's not super fun. But, um I think you want to stay very nimble, right? And you want to be like agents are all about allowing you to do things that you couldn't do before." }, { "i": 232, "speaker": "Lauren Tan", "text": "That's really to me like the value of agents. You know, it's not just throwing tokens on every single little thing, but um to me like the thing I couldn't do before is to enforce this level of constraints in a code base by myself." }, { "i": 233, "speaker": "Lauren Tan", "text": "Right? Like I'm just a single person, you know, uh it would have taken me years to build this framework uh and do all the refactoring and test everything myself and verify, you know, like run, imagine if there was just me, right?" }, { "i": 234, "speaker": "Lauren Tan", "text": "No, in the pre-agent era, just like running, you know, by myself, it would take me so long, right? And my salary is pretty high, right?" }, { "i": 235, "speaker": "Lauren Tan", "text": "Like So, you know, the question I think an engineering leader might have is just then, you know, like what is there's a trade-off of do you hire someone to do this, or do you spend the tokens to set up a code base so that even the most naive, right, the dumbest" }, { "i": 236, "speaker": "Lauren Tan", "text": "agents can do a good job. And when you actually get to this point, like even agents that are not, you know, large size, do an excellent job of writing code." }, { "i": 237, "speaker": "Lauren Tan", "text": "And this pays a lot of dividends as well for me personally where I have empowered not just myself, but again, like PMs, designers, engineers who are not familiar with Grok to just contribute in a way that is sustainable." }, { "i": 238, "speaker": "Lauren Tan", "text": "So, I think, yeah, it's definitely like a trade-off for sure, you know, like nothing is free for sure. Uh and tokens are pretty expensive." }, { "i": 239, "speaker": "Lauren Tan", "text": "Uh but oh, actually, uh I I I don't know how many of you have seen this, but we actually announced Grok 4.6 today. So, very exciting, finally out. Um so, yeah, Grok 4.6 would" }, { "i": 240, "speaker": "Lauren Tan", "text": "be like a great it is very, very smart. Uh it's really good on the benchmarks. Uh and it's the same for tokens uh I hopefully I'm not saying this incorrectly, but uh I believe the cost per token is the same as 4.5." }, { "i": 241, "speaker": "Lauren Tan", "text": "So, you're actually getting more intelligence for the same cost. Uh I think this is an area that Cursor and SpaceX AI try to really optimize for, like that Pareto frontier of, you know, cost versus intelligence." }, { "i": 242, "speaker": "Lauren Tan", "text": "Uh you know, we don't necessarily want to build the biggest model ever because that is extremely expensive to run. It's really about like how do you find that sweet spot, right?" }, { "i": 243, "speaker": "Lauren Tan", "text": "You don't You don't need a giant model, but it's just super smart, right, and it's not very expensive for inference. Uh but um yeah, I think to kind of round it up, um I think it's like a it's it's there's a if you do your own analysis, I feel like it's pretty positive." }, { "i": 244, "speaker": "Lauren Tan", "text": "It'll be pretty positive that the ROI you get from investing in stuff like this just empowers not just yourself but your whole team to be so much more productive. Right?" }, { "i": 245, "speaker": "Lauren Tan", "text": "Like imagine if you have an army of engineers like me who are shipping so much improvements and and bug fixes uh you know, every day. All right, like that is pretty exciting." }, { "i": 246, "speaker": "Speaker 1", "text": ">> Cool. Uh one last question before we wrap up. Uh this one is for the people in product on the call. So, let's say we do have an army of engineers who are shipping like Lauren." }, { "i": 247, "speaker": "Speaker 1", "text": "I'm just curious like how is the product team or other functions of your company keeping up given that like if you're shipping so quickly, have have are they using AI more to do their jobs? Like as much as you can speak to that." }, { "i": 248, "speaker": "Lauren Tan", "text": "I obviously you don't have like you're not in that role but just curious about how that works. >> Um I think this is where Grok Bot has been actually exceedingly powerful." }, { "i": 249, "speaker": "Lauren Tan", "text": "Uh where so before Grok Bot like you know, obviously Cursor only had Cursor. Like we only had agent window, we had a CLI, we had an IDE." }, { "i": 250, "speaker": "Lauren Tan", "text": "And these are really like power user tools, right? Like they're designed for developers so it's very very developer-centric. You can do knowledge work in them but it like the UI is not really optimized for that." }, { "i": 251, "speaker": "Lauren Tan", "text": "So, we actually didn't really have Well, I think like a lot of people in like you know, GTM product like they might have used Cursor to do their work but it definitely wasn't like a delightful experience for them." }, { "i": 252, "speaker": "Lauren Tan", "text": "Um I think now with Grok Bot uh it's become Grok Bot is basically like the Cursor moment for people who are not in tech in my opinion." }, { "i": 253, "speaker": "Lauren Tan", "text": "Like it's like it's like a very very accessible way to use agents in a very comfortable, very familiar interface. It looks like iMessage. Um and it's very fun too. You know, you can give your agent a name." }, { "i": 254, "speaker": "Lauren Tan", "text": "Uh you can have you can kind of do orchestration with in a very like natural way where you can sort of you know, each agent is like a person and now you have a team of agents like working on you have one agent per account that you manage as an example." }, { "i": 255, "speaker": "Lauren Tan", "text": "Or if you're a PM, you have you know, you can have an agent that summarizes all the work that Lauren did last night and then now you know what I did, right? So I think our PMs are leveraging that a lot." }, { "i": 256, "speaker": "Lauren Tan", "text": "And they're shipping code too. So you know, like often times they will just say oh here's a bug, I fix it. Can you look at it? And then I'll go review it and actually it's just perfect. I'm like okay, stamp." }, { "i": 257, "speaker": "Lauren Tan", "text": "So I think that shows that you know, the the Dune architecture is holding up, right? The all the really strict constraints allow people who are not experts in engineering to contribute at a high level." }, { "i": 258, "speaker": "Lauren Tan", "text": "So I'm I feel like I'm already seeing that pay off a lot where you know, designers and PMs are just able to ship features directly. And that just makes the Grok Bot team super fast, right? Where we can ship so quickly." }, { "i": 259, "speaker": "Lauren Tan", "text": "Um and we have a lot planned so I'm very excited to you know, to ship more stuff." } ]