[ { "i": 0, "speaker": "Lauren Tan", "text": "All right. So again, my name is Lauren. I work on Grok at SpaceX. And can I ask you a quick show of hands? How many of you have ever looked at code that somebody else wrote and you thought, that's all slop?" }, { "i": 1, "speaker": "Lauren Tan", "text": "Well, I have too. But today, I'm going to convince you that you can turn that slop into gold. And so I did something crazy last month, and I shipped 3,000 lines to production." }, { "i": 2, "speaker": "Lauren Tan", "text": "Now, before Agents, we never really cared about volume, except for the outliers." }, { "i": 3, "speaker": "Lauren Tan", "text": "And at my previous company, we called these people coding machines, people like Brett Taylor, who famously rewrote all of Google Maps in a weekend, just an insanely" }, { "i": 4, "speaker": "Lauren Tan", "text": "productive engineer. And well, with Agents, anyone can now be a coding machine. And my thesis is that the job of the software engineer, the meta of being a software engineer, is figuring out how to maintain or even improve the quality of your code while massively increasing your productivity." }, { "i": 5, "speaker": "Lauren Tan", "text": "To get there, some people like to call this a software factory. But I like to keep it simple, and I think of it as all about trust. Not blind trust, but investing in your agent's environment so you can trust that your agents are going to do the right thing." }, { "i": 6, "speaker": "Lauren Tan", "text": "Now, it takes quite a lot of effort to get there, and it took me six months to get to a point where I could ship thousands of pull requests, a lot of them, in my sleep. But it's certainly possible, and it will take you much less time than I did." }, { "i": 7, "speaker": "Lauren Tan", "text": "So let's start by talking about how we move from a low-trust environment to a high-trust one. So this inverted pyramid is how I think about where to spend time on your agent's environment." }, { "i": 8, "speaker": "Lauren Tan", "text": "The first two layers are deterministic, so using your code itself, so things like good API design, architecture, refactoring, or using lint rules, type systems, compilers, or different kinds of testing." }, { "i": 9, "speaker": "Lauren Tan", "text": "And you can use both of these layers together to constrain what agents can do and force them to do the right thing by default." }, { "i": 10, "speaker": "Lauren Tan", "text": "Verification, both runtime and formal, are great tools to layer on to find bugs and check properties and the invariance in your system that you want to hold true, and also help you check whether or not the agent understood your goal." }, { "i": 11, "speaker": "Lauren Tan", "text": "So you want to make sure that your agent understands your goals and your intent. And then finally, you want to layer with a little bit of pros, so things like rules, skills, agents.md, things that require judgment." }, { "i": 12, "speaker": "Lauren Tan", "text": "But let's start with your code base. Agents generate their outputs based on their inputs. And so the quality of the input really matters. And it just so happens that your code base is one of the largest inputs." }, { "i": 13, "speaker": "Lauren Tan", "text": "So whenever you're correcting your agent, think about how you can make that a deterministic check, instead of adding another line to your rules or your agents.md. Because it's far more reproducible when it's in code, and it lets you trust it more." }, { "i": 14, "speaker": "Lauren Tan", "text": "And the analogy that I like to use is that your code base is kind of like a garden. You know, code used to arrive a few pull requests at a time." }, { "i": 15, "speaker": "Lauren Tan", "text": "And I remember going into work in 2023 and reviewing the three pull requests that my colleagues wrote. And today, that number is more like... So every engineer is now a team of hundreds or even thousands of agents." }, { "i": 16, "speaker": "Lauren Tan", "text": "And code is arriving so rapidly that it almost feels like a continuous stream, and it almost feels organic, like a living thing." }, { "i": 17, "speaker": "Lauren Tan", "text": "And in this analogy, your code base, the constraints that you impose, the guardrails that you put up, are the concrete foundations that the new code that your agents write grow around. Just like the building in this picture. And so in many ways, the job of being a software engineer is kind of becoming like gardening." }, { "i": 18, "speaker": "Lauren Tan", "text": "Because code is the biggest input for agents, it means that any workarounds, anti-patterns, deprecated code, tech debt, it's now likely to be just copied by the next agent, since it looks like the established pattern in your code base." }, { "i": 19, "speaker": "Lauren Tan", "text": "And it's like you have some weeds or pests growing in your garden, and then they spread everywhere. Before you know it. So you want to make your code base as pristine as possible, so that agents are always copying the good stuff." }, { "i": 20, "speaker": "Lauren Tan", "text": "All right. Let's look at some real examples. So Dune is a client framework that we created for Grob to implement this idea for organic It's not really that interesting technically." }, { "i": 21, "speaker": "Lauren Tan", "text": "You can kind of think of Dune as kind of like Next.js for electron apps. But the interesting part of Dune is that it has a lot of constraints and light rules." }, { "i": 22, "speaker": "Lauren Tan", "text": "That we've encoded a lot of the lessons that we've learned building desktop applications. And one such lesson is that anything that requires nuance to use well often leads to slop." }, { "i": 23, "speaker": "Lauren Tan", "text": "You know, your classic example is use effect in React. It has its use cases, but it's often misused. Some people joke that you need a Ph.D. to use use effect." }, { "i": 24, "speaker": "Lauren Tan", "text": "And in other words, it really requires a lot of nuance to use well. And agents are not good at using it. So our solution has been to just ban it in product code. And it's really only allowed in some places inside of Dune APIs." }, { "i": 25, "speaker": "Lauren Tan", "text": "And effectively, this problem has now been categorically eliminated. And we never need to catch that during code review again. Another interesting pattern that we use is context through directory structure." }, { "i": 26, "speaker": "Lauren Tan", "text": "So one anti-pattern that we saw with Cursor, which is an electron app. It's a little bit more complex. It's that agents would often import main thread code into the renderer thread, which would cause long tasks and jank." }, { "i": 27, "speaker": "Lauren Tan", "text": "So in GrotBot and in Dune apps, we've made this very, very conventional. We have these two top level directories literally called electron renderer and electron main." }, { "i": 28, "speaker": "Lauren Tan", "text": "And imports between them are forbidden or banned by our dependency graph analyzer and some Lint rules. And again, you know, these are simple Lint rules to set up." }, { "i": 29, "speaker": "Lauren Tan", "text": "It means one less thing that we need to review and think about during code review. And then one more surprising thing that I've seen is agents using markdown. So things like documentation, code comments, rules or skills." }, { "i": 30, "speaker": "Lauren Tan", "text": "And they use this as a place to write down their justifications for not doing the right thing. Or I don't know if you've ever seen this, but they'll over-correct on some small correction that you made. And they treat it like a hard rule that can never be broken." }, { "i": 31, "speaker": "Lauren Tan", "text": "Every time. Never again. And here's another example from Cursor. So agents really loved using imperative timers." }, { "i": 32, "speaker": "Lauren Tan", "text": "So things like set timeout, request animation frame, request idle callback." }, { "i": 33, "speaker": "Lauren Tan", "text": "And they were using these as quick fixes because they were really patching over an architecture clash between VS code's imperative style and React's declarative style." }, { "i": 34, "speaker": "Lauren Tan", "text": "So our choice of architecture actually made both the client and the agent's performance much worse. And if you've ever done any performance work in JavaScript, you'll know that when you have" }, { "i": 35, "speaker": "Lauren Tan", "text": "tons of timers being rapidly created and destroyed, it causes a lot of major garbage collection, which causes a lot of lag and jank." }, { "i": 36, "speaker": "Lauren Tan", "text": "But the worst part is that agents wrote a lot of code comments, like you can see in red here, explaining why this quick fix was justified. And before you know it, we had timers spreading everywhere, and the performance of the app really suffered." }, { "i": 37, "speaker": "Lauren Tan", "text": "Now, in GrokBot. In GrokBot, we learned our lesson there, and we decided to just ban 99% of comments because they go stale easily, you know, humans just really don't bother reading the comment slop anyway." }, { "i": 38, "speaker": "Lauren Tan", "text": "And again, it gives an agent an escape hatch to justify not doing the right thing because it's hard. And the worst part about comments, though, is that they're not enforceable." }, { "i": 39, "speaker": "Lauren Tan", "text": "So what seems well-intentioned, you know, like, I'm going to tell the next agent not to do something. It never ends up actually being anything that you can trust." }, { "i": 40, "speaker": "Lauren Tan", "text": "So in GrokBot and in Dune, we now have lint rules that give good and very helpful error messages telling the agent to fix the root cause, the real problem, rather than just patch around it." }, { "i": 41, "speaker": "Lauren Tan", "text": "And so our general principle is that every time we correct our agents, it turns into one of these deterministic checks, or we think about how to categorically eliminate that problem in the code. Through better API design." }, { "i": 42, "speaker": "Lauren Tan", "text": "Or we think about how to categorically eliminate that problem in the code. Through better architecture. And if you do this enough times, you'll find that you have just less things to look out For code review. And at some point, you'll realize there's nothing left to review." }, { "i": 43, "speaker": "Lauren Tan", "text": "And you can just let agents merge most of their own pull requests. All right. Moving on. Verification is also a really big part of how we make our agents more autonomous and also increase our confidence and trust in their work." }, { "i": 44, "speaker": "Lauren Tan", "text": "And our code review process. Has now really shrunk into looking at verification artifacts instead. So things like videos, screenshots, diagrams, formal verification. And the most basic example of this is runtime verification." }, { "i": 45, "speaker": "Lauren Tan", "text": "Which is just a fancy way of saying giving your agent a way to run the code for itself and use debugging tools that, you know, human developers use as well. Like the Chrome DevTools protocol." }, { "i": 46, "speaker": "Lauren Tan", "text": "In our runtime verification system, we have this thing that we call a feature map. It's basically kind of like a site map of all of the features that are available in your application from the user's point of view." }, { "i": 47, "speaker": "Lauren Tan", "text": "And so with this, agents don't need to guess when you say, you know, the thingy in the left sidebar. It gives you some shared vocabulary. And we also make use of custom CLIs inside these skills so that agents can reuse the" }, { "i": 48, "speaker": "Lauren Tan", "text": "CLIs and the driver every time, which cuts down on token usage and is also testable. And then recently, we've also started looking at formal verification." }, { "i": 49, "speaker": "Lauren Tan", "text": "So, we now use TLA+ to do some model checking of major user flows, such as notifications across multiple devices and platforms. We use a Lean as well for proving certain algorithms." }, { "i": 50, "speaker": "Lauren Tan", "text": "For example, we have written our own custom virtualization library and we use Lean there to check that we don't have, you know, flickering or jittering problems. And we're still experimenting with this approach overall. It seems very promising." }, { "i": 51, "speaker": "Lauren Tan", "text": "We've already found quite a few bugs with formal verification. So, we're still experimenting with this approach overall. It seems very promising. We've already found quite a few bugs with formal verification." }, { "i": 52, "speaker": "Lauren Tan", "text": "But it's certainly not a silver bullet. And we're treating it as another tool, another layer in our agents' environment. And then this is a very fast video. But another thing that we just started looking at is property-based testing as well for UIs." }, { "i": 53, "speaker": "Lauren Tan", "text": "So, we use this library called Bombadil. And this has also caught a few UI bugs for us already. So, zooming out. I saw this tweet from Cindy." }, { "i": 54, "speaker": "Lauren Tan", "text": "And six months ago, she said that she was talking about not being able to trust her agents because of subtle bugs that were getting merged. And really what was missing for her in being able to trust her agents was the verification process." }, { "i": 55, "speaker": "Lauren Tan", "text": "And now, six months later, she says that 70% of her daily tokens go into verification. And I'd say that that tracks with what I see as well, that most of my token usage goes to verification." }, { "i": 56, "speaker": "Lauren Tan", "text": "And if you really look closely at her process. None of these things here are new or novel techniques that we've never had before. These are just good software engineering tools. But ones that might have been prohibitively expensive to use before as they were hard" }, { "i": 57, "speaker": "Lauren Tan", "text": "to understand or write or maintain. But agents drastically cut this cost down. And really, we're entering a golden age of software testing and verification. Because agents let us test everything to a much greater degree than used to be economical." }, { "i": 58, "speaker": "Lauren Tan", "text": "So, formal verification can also help you better reason about how your product should work and whether your agents really understand your intent and goals." }, { "i": 59, "speaker": "Lauren Tan", "text": "I made this new feature called state chart in my plug-in P stack, P stack, which helps you visualize state machines from the user's point of view. And I find this much easier to review alongside the videos and screenshots that we provide." }, { "i": 60, "speaker": "Lauren Tan", "text": "And so, that's been great. Sorry for the flashback. Let's talk about what the promised land looks like. What do you get in return for all of this, you know, investment? So, on the Grootbot team, our velocity is taking off and rapidly increasing." }, { "i": 61, "speaker": "Lauren Tan", "text": "And last month, our team shipped a ridiculous 11,000 pull requests to production. Which is a wild number. And our pace of releasing the client has also increased." }, { "i": 62, "speaker": "Lauren Tan", "text": "And we cut a new stable version about once a day. And then, we've got a new stable version. And we've got a new stable version about once a day. And then, we've got a new stable version about once a day. And then, the core team is also shipping much more." }, { "i": 63, "speaker": "Lauren Tan", "text": "And we also have a lot more casual or drive by contributors who don't always have a lot of context. But they're also contributing. And in October, we had about 227 people who contributed at least one pull request." }, { "i": 64, "speaker": "Lauren Tan", "text": "So, despite this velocity accelerating, you might be thinking to yourself, like, isn't this going to be chaos? You know, isn't this going to be a bunch of slop? But actually, the number of PRs that get reverted. Has actually gotten lower on a relative basis and is steady at about half a percent." }, { "i": 65, "speaker": "Lauren Tan", "text": "But I think the really interesting thing about this shipping pace and this volume of pull requests is that the performance of GrokBot has stayed not just stable, but has improved on multiple dimensions compared to when we launched." }, { "i": 66, "speaker": "Lauren Tan", "text": "So startup, typing, opening new chats, switching chats, all of this has gotten faster. And keep in mind, GrokBot is still an Electron app. And I don't think it's going to be a problem." }, { "i": 67, "speaker": "Lauren Tan", "text": "It's going to be a problem. For now. So there is a ceiling, right? But it's already much faster than many other agent UIs out there." }, { "i": 68, "speaker": "Lauren Tan", "text": "And for an Electron app with this high velocity and this many people contributing, I'd say that's pretty good. So before you jump into building out a software factory and, you know, automating everything" }, { "i": 69, "speaker": "Lauren Tan", "text": "that you can, whenever you correct your agent, try and do it by improving your code base first. You know, start with a lint rule to prevent new occurrences of anti-patterns. And then clean up those old patterns as well." }, { "i": 70, "speaker": "Lauren Tan", "text": "And then make sure that your verification story is really strong. A lot of this work is a prerequisite to automation anyway. And if your inner coding loop produces slop, then automating that loop just produces more slop." }, { "i": 71, "speaker": "Lauren Tan", "text": "And you need to throw more and more and more tokens at the problem, which we all know is very expensive. And once you make these investments into, you know, code into your code base, static" }, { "i": 72, "speaker": "Lauren Tan", "text": "analysis, runtime information. Informal verification. You'll start to find that many things become possible with these foundations. And that you can really easily build your own factory on top of these foundations." }, { "i": 73, "speaker": "Lauren Tan", "text": "And when you're ready, GrotBot is a great way to build that outer loop. We have tons of connectors, routines that can kick off Cursor, cloud agents, and projects which let you orchestrate hundreds of thousands or even thousands of agents." }, { "i": 74, "speaker": "Lauren Tan", "text": "And as a quick example. You know. When we release now, GrotBot will automatically spin up a new Cursor project for us that will fuzz the entire application, the entire build." }, { "i": 75, "speaker": "Lauren Tan", "text": "And it will find bugs by running the app and using it like a real user. And then it's going to open fix PRs for each of those bugs that it finds. And it's going to auto merge it." }, { "i": 76, "speaker": "Lauren Tan", "text": "As crazy as that sounds. And that's because our code base is good enough that pull requests tend to just be good by default. And most product code that we land no longer really requires deep human review." }, { "i": 77, "speaker": "Lauren Tan", "text": "So when you have solid foundations and good organic architecture, you can turn slop into gold and you can ship with high quality, high velocity without compromising on quality. My name is Potato, also known as Lauren." }, { "i": 78, "speaker": "Lauren Tan", "text": "Thanks to you again for listening to my talk." } ]