[ { "i": 0, "speaker": "Akanksha", "text": "Welcome everyone to fall quarter and welcome to CS 329A. I hope you made it to the right class. This class is on self-improving AI agents. Anyone made it to the right class here?" }, { "i": 1, "speaker": "Speaker 1", "text": "Yes?" }, { "i": 2, "speaker": "Akanksha", "text": "Okay. How many of you are here and doing master's at Stanford?" }, { "i": 3, "speaker": "Speaker 1", "text": "PhD?" }, { "i": 4, "speaker": "Akanksha", "text": "Okay. A few of you and then undergrad? Okay. So master's is the big crowd here. >> There's a good balance. Yeah. >> It's a really good balance. Uh So we'll introduce ourselves. So I'm Akanksha." }, { "i": 5, "speaker": "Akanksha", "text": "I have been working in large language models for a while and I'm an adjunct professor at Stanford and I'm also in research at a startup called Reflection AI." }, { "i": 6, "speaker": "Azalia Mirhoseini", "text": ">> Hi, I'm Azalia Mirhoseini. I'm a I'm an assistant professor in the CS department and Akanksha and I met each other when we were at Google Brain." }, { "i": 7, "speaker": "Azalia Mirhoseini", "text": "Yeah, back in the day and then Google DeepMind." }, { "i": 8, "speaker": "Azalia Mirhoseini", "text": "And I also worked on Code and Anthropic and Gemini at Google DeepMind and we are so excited to teach this course again." }, { "i": 9, "speaker": "Azalia Mirhoseini", "text": "This is the second time we're teaching. Of course, we have made some updates both to the lectures and also the way we are kind of conducting everything during the course. And we are we were just so amazed by the amount of interest for people to join this course." }, { "i": 10, "speaker": "Azalia Mirhoseini", "text": "So we are excited to have you here." }, { "i": 11, "speaker": "Akanksha", "text": ">> The course website is at cs329a.stanford.edu. We have updated the website in terms of all the lecture materials and the papers that you'll be reading as well as what's the homework schedule to expect the project proposal and project" }, { "i": 12, "speaker": "Akanksha", "text": "structures as well. So, in today's lecture we'll start with an overview of large language models. Just how the scaling trends have evolved over the last 5 years and then we'll focus more on what we will cover in this class." }, { "i": 13, "speaker": "Akanksha", "text": "And then from there in the second half of the lecture we'll cover the course logistics and that part you should pay a lot of attention to because that influences your grades. So, let's take a look at an overview of the trends for large language models." }, { "i": 14, "speaker": "Akanksha", "text": "So, one of the aspects that has been mind-blowing since GPT-3 came out is that as you scale up the number of parameters in a model they get better and what this means is that language models like BERT" }, { "i": 15, "speaker": "Akanksha", "text": "and T5 were good but as we increase the number of parameters they got really better." }, { "i": 16, "speaker": "Akanksha", "text": "They perform much better and what this led to is something called a scaling loss for large language models which are used in building the pre-trained base model as you increase the amount of compute." }, { "i": 17, "speaker": "Akanksha", "text": "So, there are three graphs here. As you increase the amount of compute on the x-axis the loss function on the y-axis goes down and the more compute you have put in the more the test loss goes down which leads to a better model." }, { "i": 18, "speaker": "Akanksha", "text": "So, that was one big aspect that led to GPT-3 and then all the subsequent ChatGPT, PaLM, Gemini and so on. And then similarly as you increase the dataset size on x-axis the test loss goes down further on y-axis." }, { "i": 19, "speaker": "Akanksha", "text": "So, that's another aspect of like another axis of scaling that leads to better language models. And the third one is of course parameter count." }, { "i": 20, "speaker": "Akanksha", "text": "So, if you increase the number of layers in a transformer or the amount of parameters that are there in the language model that improves the loss and that leads to a better model. And then this has been the foundation for a long time until last year where this was starting to hit some kind of a saturation point." }, { "i": 21, "speaker": "Akanksha", "text": "So, from 2018 to almost 2024, this graph is a little bit outdated but from 2018 to 2024, as the model size has gone up almost consistently, we have been increasing the size of the model." }, { "i": 22, "speaker": "Akanksha", "text": "So, we started with something like BERT was 340 million parameters, GPT-2 was 1.5 billion parameters, and then GPT-3 was 175 billion parameters, and then PaLM was 540 billion, and then there was a bunch of subsequent models, and GPT-4 is estimated to have trillions of parameters." }, { "i": 23, "speaker": "Akanksha", "text": "So, you can imagine that there has been an exponential growth in just the model size for large language models. The large stands for that they're growing in size, and that has been one of the key areas in which the models have been getting better." }, { "i": 24, "speaker": "Akanksha", "text": "So, as I was explaining that as the models get bigger, they continuously improve in performance on natural language benchmarks, on various other benchmarks, reasoning benchmarks, and so on." }, { "i": 25, "speaker": "Akanksha", "text": "And then other interesting aspect that I will show you in a moment is that they learn how to do few-shot learning." }, { "i": 26, "speaker": "Akanksha", "text": "So, earlier you have to fine-tune the model for a specific domain, but just by giving a few examples, the model can follow that template and reason in that same vein which makes it extremely easy to prototype things." }, { "i": 27, "speaker": "Akanksha", "text": "And then the third aspect is that as the models become bigger, they have emergent behavior where the capabilities like reasoning only emerge in larger models. So, just to understand few-shot learning or zero-shot learning here." }, { "i": 28, "speaker": "Akanksha", "text": "So, if the model is given a question like or a task like translate English to French, it's able to give so this is called a prompt." }, { "i": 29, "speaker": "Akanksha", "text": "Like it's given a task translate English to French and then it's asked to translate cheese to whatever is the French word. If the model can predict the answer without actually being trained on the specific task, then that would be called zero-shot learning." }, { "i": 30, "speaker": "Akanksha", "text": "And then few-shot learning, in addition to giving the description of the task, you're also giving a few examples." }, { "i": 31, "speaker": "Akanksha", "text": "So in this particular case, it's given translate English to French and it's given a few examples of the translation from English to French and then it's asked to translate cheese and it's able to translate that and you can see that this" }, { "i": 32, "speaker": "Akanksha", "text": "capability shows up in the large language models across a large number of tasks and that has been one of the key areas that GPT-3 and Palm and other models enabled and then that subsequent innovations came from reasoning aspects of this." }, { "i": 33, "speaker": "Akanksha", "text": "So on the reasoning side, as models have become bigger, they have this emergent behavior of reasoning which Azalia will tell us a little bit more about." }, { "i": 34, "speaker": "Azalia Mirhoseini", "text": ">> So as the models become bigger, not only we have this kind of a predictive scaling laws property that we kind of know how the loss function's going to go down as we throw in more compute data and parameters in the model, there are new kind of behaviors that appear in the model that they didn't have before and we never could predict that until we had these bigger models and saw these kind" }, { "i": 35, "speaker": "Azalia Mirhoseini", "text": "of behavior in them. And one of the most important one of those behaviors is the emergent chain of thought behavior. And for those of you who are not familiar with this, here is how it works." }, { "i": 36, "speaker": "Azalia Mirhoseini", "text": "So in a usual prompt prompting of a model, we can ask a model in this case a math problem and then we can give it a one-shot example. Here is a math problem, here is the answer. Now I ask you a similar but not exact and not exactly the same math problem, and the model is expected to answer that." }, { "i": 37, "speaker": "Azalia Mirhoseini", "text": "Um And the model might be able to use the one-shot example or it might not be. But we can do better here by not only give an example but also provide the reasoning or the chain of thought on how we got to that answer. So, in this case, the question is a very simple question." }, { "i": 38, "speaker": "Azalia Mirhoseini", "text": "Roger has five tennis balls. He buys two more cans of tennis balls. Each can has three tennis balls. How many total tennis balls does he have now? And instead of just saying 11, like Um Roger started with five balls." }, { "i": 39, "speaker": "Azalia Mirhoseini", "text": "Two cans of three tennis balls is six. And uh 5 + 6 is 11. Uh And with this example, then the model has become familiar with the process to get to the answer, and it can leverage that to answer new problems. Well, obviously this is a very simple problem." }, { "i": 40, "speaker": "Azalia Mirhoseini", "text": "Any like one-billion parameter model nowadays can solve this. Doesn't need an example or a few shot or a chain of thought. But the chain of thought" }, { "i": 41, "speaker": "Azalia Mirhoseini", "text": "property itself is a property that is holding to this day and it's a very very important property for the reasoning models and all the thinking models and all the progress and many much of the progress that we have seen um in the past year or so." }, { "i": 42, "speaker": "Azalia Mirhoseini", "text": "Now, um this property again appears as we have larger and larger models. Here are three models, Lambda, GPT, and PaLM. And as you can see, the chain of thought and this is some results on the math dataset." }, { "i": 43, "speaker": "Azalia Mirhoseini", "text": "math dataset. And these models when they're smaller, like eight billion parameter for Lambda or um GPT for GPT around 7 billion. They can't really benefit from chain of thought. Like it doesn't do anything for them." }, { "i": 44, "speaker": "Azalia Mirhoseini", "text": "But as they become larger or for a PaLM model again at 8 billion, it can leverage this chain of thought property and learn from the reasoning that is and the process that is provided in the context in order to solve problems better." }, { "i": 45, "speaker": "Azalia Mirhoseini", "text": "And not only that, there are other kind of abilities to solve new task as we increase the size of the models. Like for example, small models can all of a sudden solve modular arithmetic better or solve word unscramble problem just by having this larger size and this property." }, { "i": 46, "speaker": "Azalia Mirhoseini", "text": "You see that all of a sudden it appears at a certain size. And because of all of this, we still as in frontier companies and labs are still interested in pushing the scaling laws and not only get the natural kind of progression, but also we" }, { "i": 47, "speaker": "Azalia Mirhoseini", "text": "we will will see more and more of these kind of emergent behavior that we are that is going to be very important. Now, a little bit of history here. So chat GPT was launched in November 2022. Obviously, it was one of the most successful kind of apps or software ever created." }, { "i": 48, "speaker": "Azalia Mirhoseini", "text": "The time to reach 1 million users for it was 5 days, which is significantly faster than any of the other famous software that you can see there or services that you can see there." }, { "i": 49, "speaker": "Azalia Mirhoseini", "text": "And there were pieces of kind of innovation on top of just scaling up the parameter size of the model that were very important to make chat GPT what it was and to make that kind of leapfrog and over the previous models like GPT-3 and so on." }, { "i": 50, "speaker": "Azalia Mirhoseini", "text": "And two important pieces here but were um the instruction tuning and um the" }, { "i": 51, "speaker": "Azalia Mirhoseini", "text": "reinforcement learning from human feedback which we're going to briefly mention in the next slides but throughout this course we're going to learn more and more about them." }, { "i": 52, "speaker": "Azalia Mirhoseini", "text": "Um so first and then um we're going to learn about all of these in more details but let's just take a step back and see how the process works. So the pre-training step is the step that the beginning of any language model training." }, { "i": 53, "speaker": "Azalia Mirhoseini", "text": "Uh that's the easiest step and we just train the model to predict the next token out of all sorts of text and data and so on. And then this is the fine-tuning step that was a key differentiating factor for a model like ChatGPT." }, { "i": 54, "speaker": "Azalia Mirhoseini", "text": "Um and let's start with um uh the first aspect of fine-tuning which is kind of alignment between human preferences." }, { "i": 55, "speaker": "Azalia Mirhoseini", "text": "So when a model is pre-trained it has no sense it has seen all data on the internet in the books everywhere but it has no sense of like what is right and wrong and it just has it has statistically knows about the state of the world but does it it doesn't exactly know things or know how to follow instructions." }, { "i": 56, "speaker": "Azalia Mirhoseini", "text": "So the next step in training these models uh is the the the [clears throat] step where we try to steer the AI models to follow the goals and preferences uh and values of humans." }, { "i": 57, "speaker": "Azalia Mirhoseini", "text": "Um and this is still a big problem it's a big issue we've haven't like mastered it yet but as the models become as we progress models are becoming better and much more uh kind of like uh powerful in the things that they can do. So, it turns out that we can fine-tune the models to be aligned." }, { "i": 58, "speaker": "Azalia Mirhoseini", "text": "For example, here are some graphs that show what if we take our base model that's just pre-trained and fine-tune it for things like sensibleness or safety. And these are like these are data that we have collected." }, { "i": 59, "speaker": "Azalia Mirhoseini", "text": "High-quality data for kind of showing the model what we mean by what's safe and what's unsafe or what's sensible and what doesn't make sense. And these are like more curated like more highly curated data that we pass to the model for fine-tuning and to make the model more aligned with the values or the preferences of humans." }, { "i": 60, "speaker": "Azalia Mirhoseini", "text": "Um So, going back, let's just walk through this process again." }, { "i": 61, "speaker": "Azalia Mirhoseini", "text": "So, the first stop when we take a pre-trained language model is fine-tuning with next token prediction, which follows the same process as pre-training, but this time on data that is much higher quality." }, { "i": 62, "speaker": "Azalia Mirhoseini", "text": "For example, this could be data from books, creative essays, data that companies pay millions of dollars or hundreds of millions of dollars potentially to buy this data that's really high quality, and then we train the we fine-tune the model on those. And the model becomes much better as a result of that." }, { "i": 63, "speaker": "Azalia Mirhoseini", "text": "Um The next step is instruction tuning. And by instruction tuning, what we mean is that we show the model again traditionally in the past 2 years, this instruction tuning data is like a combination of human generated data or templates and then the synthetic data." }, { "i": 64, "speaker": "Azalia Mirhoseini", "text": "But, the way this instruction fine-tuning works is that we do have instruction and then question answer pairs where the model learns how to follow the questions and answers them. For example, here is an instruction for tuning data set could be about the weather like please answer the following question." }, { "i": 65, "speaker": "Azalia Mirhoseini", "text": "What is the boiling point of nitrogen? And then that this is the answer. Or, we can have chain of thought fine-tuning where we show the model how it can walk through a process to get to the answer. And then we have the label, we have the results over here. And this is like these pairs" }, { "i": 66, "speaker": "Azalia Mirhoseini", "text": "of like questions and instructions and answers. And again, this data set is very um is is there's a lot of effort that goes into this data set and the quality of the data set and the generality of it uh has a lot of impact on the quality of the model that we get at this stage." }, { "i": 67, "speaker": "Azalia Mirhoseini", "text": "Um and after this point in the model becomes more and more like um like the models we see today. Like you can ask them questions, you can go back and forth with them. They have this sense of like how to uh perceive a question and how to walk through kind of uh some sort of like process to get to an answer." }, { "i": 68, "speaker": "Azalia Mirhoseini", "text": "And the next step after instruction fine-tuning is this process of um using human preferences and fine-tuning the model based on that. And this process is called RLHF or reinforcement learning from human feedback." }, { "i": 69, "speaker": "Azalia Mirhoseini", "text": "Um the way this is different from the previous system is the way we create the objective function and uh create the data. So, in this case, instead of like just creating some supervised prompt and labels, we are basically creating a reward model out of human preferences." }, { "i": 70, "speaker": "Azalia Mirhoseini", "text": "So, companies spend a lot of money uh asking humans, sometimes experts, sometimes just humans, normal humans, uh about questions and the answers generated by the model. And we ask the humans to rate them to see which one is correct and is not correct." }, { "i": 71, "speaker": "Azalia Mirhoseini", "text": "And that's how we can create some sort of a reward model here. Um and this reward model is instead of human, then we can use the reward model in order to kind of like guide the parameters of our LLM towards uh generating answers that are in line with this reward model." }, { "i": 72, "speaker": "Azalia Mirhoseini", "text": "So, basically, we want the generations by the model uh to be such that this reward model says, \"Yes, these are good generations.\"" }, { "i": 73, "speaker": "Azalia Mirhoseini", "text": "Um we're going to learn more about this here, but I wanted you to have um the high-level idea of what RLHF means. Uh and we can collect we can have different reward types types of reward." }, { "i": 74, "speaker": "Azalia Mirhoseini", "text": "For example, we could ask the model to reward a pair of answers based on their correctness or helpfulness or specificity or harmlessness, all sort of reward functions that we can define. And" }, { "i": 75, "speaker": "Azalia Mirhoseini", "text": "depending on how we want the models to be and what we care most about, we can weight these different reward models and kind of use that in the RLHF process." }, { "i": 76, "speaker": "Azalia Mirhoseini", "text": "So, these steps at a high level, of course, there's a lot of effort going into that, but like pre-training and then um fine-tuning on higher-quality data, instruction tuning, and RLHF are the core kind of components that made um a model like ChatGPT over something that existed over all the large models um like GPT-3 and prior models." }, { "i": 77, "speaker": "Azalia Mirhoseini", "text": "So, that made the huge difference in how good and capable the models were. Um so that was very cool up until last year or a year and a half ago, pre-training and then fine-tuning were the big pieces." }, { "i": 78, "speaker": "Azalia Mirhoseini", "text": "Uh but since a year and a half ago, it turns out inference is also a frontier for making the models much more capable. Um so and that opened like a whole set of whole lot of new research and directions and ways that we can make the model better." }, { "i": 79, "speaker": "Azalia Mirhoseini", "text": "Um so let's think let's talk about it a little bit. Like this is a work we did in my lab last year uh last summer. It it was called large language monkeys." }, { "i": 80, "speaker": "Azalia Mirhoseini", "text": "Um and it's kind of inspired by the infinite monkey theorem and it's related to inference scaling. Um how many of you have heard of the infinite monkey theorem? Okay, that's a good amount." }, { "i": 81, "speaker": "Azalia Mirhoseini", "text": "So, the idea of the infinite monkey theorem uh what it says is that it's not proven, but what it says is that if you have a monkey and a kind of a typewriter and then we keep" }, { "i": 82, "speaker": "Azalia Mirhoseini", "text": "letting the monkey type all day forever, um after some time you will have the works of William Shakespeare in in the things that the monkey types. Um so we were inspired by that to name our project." }, { "i": 83, "speaker": "Azalia Mirhoseini", "text": "Uh but the idea here was that let's let's have the LLM be the monkey and uh let's uh ask the model over and over again to solve a given input problem. Uh you can think of it as parallel sampling or parallel generation. So, instead of asking the model once to solve the input problem, you can ask it many times." }, { "i": 84, "speaker": "Azalia Mirhoseini", "text": "And also, let's assume you have some sort of a verifier or selection mechanism, so you can pick which one of these generated responses was correct. Um and then you output that in your system. That becomes the final output." }, { "i": 85, "speaker": "Azalia Mirhoseini", "text": "So, the model, instead of generating one answer, generates many, many answers. And then your verifier selects which one of them is correct and then outputs one of the correct ones. Um you can think of a verifier as, say, some unit tests." }, { "i": 86, "speaker": "Azalia Mirhoseini", "text": "If the problem is generating code, the verifier can be the unit test that we run against the code and we see which which one of the code generations passes all the unit tests and so on. Um The reason this is very important is that models are not uh there is some variance in the responses the models generate." }, { "i": 87, "speaker": "Azalia Mirhoseini", "text": "It's not deterministic. And you can even control the variation in the model responses with this thing called temperature. So, you can force the model or encourage the model to generate more different responses as you ask a question. Um And here, now let's see some of the results of this repeated sampling." }, { "i": 88, "speaker": "Azalia Mirhoseini", "text": "For a bunch of tasks like math and coding benchmarks, we increased the number of samples per problem from one, this is the normal one, uh to 10,000. And here we are showing the coverage or the fraction of problems that are solved by at least one of these samples." }, { "i": 89, "speaker": "Azalia Mirhoseini", "text": "And what we are seeing here is that um so the red line, the red dashed one, was the GPT-4 model." }, { "i": 90, "speaker": "Azalia Mirhoseini", "text": "And while these blue and green lines, which were the Llama 2 8B, Llama 3 8B, and 7B, were worse than GPT for our" }, { "i": 91, "speaker": "Azalia Mirhoseini", "text": "model with one sample, but if we increase the number of samples from these models, in all of these cases, they do better than the GPT-4 model. And what it says is that it kind of seems like the model or models already know a whole lot more than what you get out of them when you just ask them once, right?" }, { "i": 92, "speaker": "Azalia Mirhoseini", "text": "So, this is the kind of the underlying property of inference scaling because it seems like we can get a whole lot more capability and problem-solving kind of ability from these models if we just bring in this inference scaling." }, { "i": 93, "speaker": "Azalia Mirhoseini", "text": "Here, you're showing repeated sampling, but there are many other ways to do that. And we are so, the reason it's called inference scaling is that we're not touching any of the parameters of the model." }, { "i": 94, "speaker": "Azalia Mirhoseini", "text": "The model is fixed, and we are just like at inference time, we are creating different ways to generate to kind of produce generations from the model and use them to generate higher-quality answers or answers to harder problems." }, { "i": 95, "speaker": "Azalia Mirhoseini", "text": "Um the interesting part here is that for some of these problems, out of these 10,000 solutions, maybe like three or four of them were correct for a single problem. So, it shows that how how kind of um uh how important it is to to scale the inference to really get to the core capability of the models." }, { "i": 96, "speaker": "Azalia Mirhoseini", "text": "Sorry, what's the question? So, the possible answers need to be generated somehow by the model, right? So, so this is like the space of all possible answers to the model is way bigger, like like right? We can like if we do like a tree search or a random search, it's like way bigger, right?" }, { "i": 97, "speaker": "Azalia Mirhoseini", "text": "There's no way to like hit even like imagine like we are asking a human like just sit down and solve this problem, right? There is a limit to Um of course there is no limit to human creativity, human's creativity, but there is a limit to how many generations we can have, right?" }, { "i": 98, "speaker": "Azalia Mirhoseini", "text": "Or if you're randomizing, there's a whole lot of of answers that can be generated and it's and this is like highly still highly sample efficient, right?" }, { "i": 99, "speaker": "Azalia Mirhoseini", "text": "10,000 is not a whole lot, especially like for example for the math um F2F, some of these problems are really hard like IMO level problems." }, { "i": 100, "speaker": "Azalia Mirhoseini", "text": "Like these are really hard problems that a tiny model like a 7B and a Llama 8B can solve." }, { "i": 101, "speaker": "Speaker 2", "text": ">> How does it compare how how does the latency compare with like when you do actually generate all the samples like compared to the GPT-3 or all, right? Like how does the latency" }, { "i": 102, "speaker": "Azalia Mirhoseini", "text": ">> Yeah, of course there is a trade-off um and we we have we touch on that in the in the lecture that I go into more details toward this. The parallel samples, the good thing about them that they're they can be running parallel." }, { "i": 103, "speaker": "Azalia Mirhoseini", "text": "Um so, from a latency perspective, that's less of an issue, but of course there is the frontier of cost trade-offs between how much compute you're spending and and we will look into that. And it varies, it varies depending on the type and complexity of the problems." }, { "i": 104, "speaker": "Azalia Mirhoseini", "text": "It's a very hard When you have verifiers, it's much easier." }, { "i": 105, "speaker": "Azalia Mirhoseini", "text": "When you don't have verifiers, there's this whole set of research on how to train LLMs to judge or LLMs to reward functions or LLMs with tools becoming and we're going to learn about all of it. We have a lecture just dedicated to verifiers, so." }, { "i": 106, "speaker": "Azalia Mirhoseini", "text": "No, yeah, we can't like have the temperature being too high because it's just it is going to be gibberish, like the model doesn't generate. So, we There's actually like a research project last year was like on tuning that or like figuring out what temperature." }, { "i": 107, "speaker": "Azalia Mirhoseini", "text": "Usually, if you go beyond like 1.2 or so, it's not great, yeah. But there are there are other tricks that you can do to kind of kind of uh make the model be more um like have more diversity in the type of answers and that can be useful. Uh All right. Um So, let's look at this." }, { "i": 108, "speaker": "Azalia Mirhoseini", "text": "So, now, again, going back to the events of last year, year and a half, um we had the DeepSeek model. I think it came out in >> December. DeepSeek came out December 2023." }, { "i": 109, "speaker": "Azalia Mirhoseini", "text": "Uh oh, well. It's been out for a short time or long time, I don't know. The both. Uh so, so then models like DeepSeek came last year." }, { "i": 110, "speaker": "Azalia Mirhoseini", "text": "So, basically, a core kind of innovation in DeepSeek and then the 01 series Gemini thinking and such were that." }, { "i": 111, "speaker": "Azalia Mirhoseini", "text": "Now, we are kind of bringing this fine-tuning and test time scaling together because with this test time scaling and the models themselves, we have this new like kind of engine that we can generate a whole lot of synthetic data um for example, for problems like for math problems where we know the answer, we can have the model generate different like kind of answers that leads to that final final kind of golden answer." }, { "i": 112, "speaker": "Azalia Mirhoseini", "text": "Or for the coding problems, we can use the model to generate tons of data during test time scaling, uh tons of like quality data to solve coding problems, and that becomes part of our training set now. We can use that to fine-tune the model to become better." }, { "i": 113, "speaker": "Azalia Mirhoseini", "text": "So, that kind of bringing these two together became a big piece in thinking models and reasoning models. Um and this this is like a an area that is like really really important. There's just so much to be explored in this because again, there there is no It's kind of open-ended, right?" }, { "i": 114, "speaker": "Azalia Mirhoseini", "text": "There's no boundary in how good the models can become with test time scaling and then bringing that back to the process of training the model or fine-tuning the model to become better is very exciting, and that's kind of the self-improving piece that we are very excited about." }, { "i": 115, "speaker": "Azalia Mirhoseini", "text": "Um so, a little bit about reasoning models. Um in the language models, what we were showing was the coverage. Like if we had access to verifiers, how uh we we can see the kind of a log linear kind of scaling laws with the number of samples." }, { "i": 116, "speaker": "Azalia Mirhoseini", "text": "Um and all OpenAI, when they released 01 last September, um they showed a a log linear relationship this time with pass@1 for these like hard hard MMLU benchmark which is like I said a really" }, { "i": 117, "speaker": "Azalia Mirhoseini", "text": "Um and all OpenAI, when they released 01 last September, um they showed a a log linear relationship this time with pass at one for these like hard hard Amy uh benchmark which is like I said a really" }, { "i": 118, "speaker": "Azalia Mirhoseini", "text": "difficult uh math questions. So, as they're show what they're showing here is that as they're increasing test time compute and this is in log scale um uh the accuracy pass rate one accuracy of the model goes up and this is for test time alone." }, { "i": 119, "speaker": "Azalia Mirhoseini", "text": "Uh previously uh it's this has been shown for training but it's very interesting that this kind of scaling applies to test time as well um without changing the parameter count of the model." }, { "i": 120, "speaker": "Azalia Mirhoseini", "text": "Um Um I'm going to walk you through like one or a few examples of how this kind of reasoning or test time uh scaling uh kind of is works with models like O1 and or Gemini." }, { "i": 121, "speaker": "Azalia Mirhoseini", "text": "So, one motivation here is that for difficult problems just like humans think a lot more, spend a lot more time, or consider many different strategies, uh thinking models do the same and they may use a chain of thought or other techniques to just go go about like solving a problem um and there are different steps to that." }, { "i": 122, "speaker": "Azalia Mirhoseini", "text": "For example, um problem analysis is one step like the model first sees a problem and tries to analyze it. Um then it can do the task decomposition breaking a task into kind of simpler tasks that are more addressable. Um then there is the self uh evolution strategies where the model" }, { "i": 123, "speaker": "Azalia Mirhoseini", "text": "kind of try something, sees the feedback, for example runs some tests on the code or use a uses a calculator or just judges the answer and it can use that feedback to optimize itself. Um there's self-correction and there are alternative proposals if something doesn't work, the model can backtrack and kind of, um, try different approach." }, { "i": 124, "speaker": "Azalia Mirhoseini", "text": "These are some key principles that the model, uh, probably at some point, uh, pieces of it were trained as part of the training kind of data set were curated by humans, but a big part of it is also the model kind of acquiring these skills during this fine-tuning process and RL like RL from synthetic data process that the model does." }, { "i": 125, "speaker": "Azalia Mirhoseini", "text": "Um, here is an example of like how O1 does analysis. Um, uh, the model is asked to write a bash script, um, uh, that takes a matrix and then outputs, uh, the transpose of the that matrix. The model starts with just like humans, like starts like thinking, what are the pieces important pieces here?" }, { "i": 126, "speaker": "Azalia Mirhoseini", "text": "So, the user is requesting a bash script, um, let's understand the input and output, uh, formats of this matrix, um, and and so on. So, the model just thinks to, um, and amazingly this kind of like it's like a progression of chain of thought." }, { "i": 127, "speaker": "Azalia Mirhoseini", "text": "This is like helping the model finding the answer, but the difference between chain of thought and this is that the model itself is producing this chain of thought." }, { "i": 128, "speaker": "Azalia Mirhoseini", "text": "Um, there's of course task decomposition, like in this case for the approach, uh, the model thinks about parsing the input, building the matrix as an array of, uh, as as an array of arrays, and so on." }, { "i": 129, "speaker": "Azalia Mirhoseini", "text": "Um, there's also self-correction, which is a big important piece of, um, this is an important piece here, and I don't know how many of you have seen that when trying with the models that you can see the thinking traces." }, { "i": 130, "speaker": "Azalia Mirhoseini", "text": "Uh, and this is actually pretty common, like the model starts something and then in the middle says, 'Wait, the correct way like there's something wrong, maybe. Um, or and maybe I need to fix that.' And it's very, very cool that the model can do that itself." }, { "i": 131, "speaker": "Azalia Mirhoseini", "text": "Um, so reasoning models when they came out, like models like O1, compared to a model like GPT-4o, which model, um, they tend to be better in, obviously, the reasoning task." }, { "i": 132, "speaker": "Azalia Mirhoseini", "text": "For example, in math calculation, data analysis, programming and such, they all kind of outperform GPT-4o, but not necessarily in personal writing or editing text and so on. Yeah. Question?" }, { "i": 133, "speaker": "Speaker 3", "text": ">> Do you think the gains due to this asking for a reasoning process due to actually the act of generating this reasoning process or just the act of asking the model to think or to use the chain of thought at all? Like, for example, if you think step-by-step but don't generate out loud, like don't actually generate those thought processes which are there but without saying" }, { "i": 134, "speaker": "Azalia Mirhoseini", "text": "Um, I repeat the question. Like, do you you're saying that is it the Do you think the reasoning models are becoming what they are is based on the Is it based on the out way of thinking?" }, { "i": 135, "speaker": "Akanksha", "text": ">> I I think the question is that Is it because of the, um, reasoning process or is it because of the step-by-step? The chain of thought." }, { "i": 136, "speaker": "Azalia Mirhoseini", "text": "Like, the model can and all of it All of these steps, like breaking down the problem, being able to backtrack, being able to do analysis, all of these are these skills that the model has learned during this thinking optimization process that now generalize to other things and they're helping the model right now." }, { "i": 137, "speaker": "Azalia Mirhoseini", "text": "So, I think the other way to look at it is as Ali presented repeated sampling. So, if you had just a base model right after pre-training, it would be able to do some reasoning." }, { "i": 138, "speaker": "Akanksha", "text": "But, at the same time it will generate different kinds of reasoning chains and it doesn't quite know which one is correct. So, a lot of the what we will cover in train time or test time scaling really comes down to it learns which is correct." }, { "i": 139, "speaker": "Akanksha", "text": "So, pass at one accuracy goes up which is what you were seeing in the reasoning models as opposed to like pass at K or coverage which is what you were seeing in the repeated sampling results." }, { "i": 140, "speaker": "Akanksha", "text": "Yeah." }, { "i": 141, "speaker": "Azalia Mirhoseini", "text": "And one other way to think about like yes, these kind of deep thinking and things that the model generates is actually helping the model every time to solve a new problem. Um and although it's like very expensive for the model to generate those things, still like we want them to generate those because it leads to better answers." }, { "i": 142, "speaker": "Azalia Mirhoseini", "text": "Um Any other questions? Yes. >> Do you use a different model for the reasoning stuff versus generating the final answer?" }, { "i": 143, "speaker": "Azalia Mirhoseini", "text": "Like maybe a smaller or like a probabilistic one just for the reasoning and the final answer is the >> Um So, typically the reasoning capabilities have gone up with the model size." }, { "i": 144, "speaker": "Akanksha", "text": "Um So, typically the reasoning capabilities have gone up with the model size. So, if if anything uh you would use the larger model for reasoning traces and then maybe you collect a bunch of reasoning traces and then have a smaller model summarize the answer." }, { "i": 145, "speaker": "Azalia Mirhoseini", "text": "Yeah. So, the models like this is like at least currently, things may change, they like their own traces more. Even if the traces are coming from a better model, they tend to like their own their own generated traces more." }, { "i": 146, "speaker": "Azalia Mirhoseini", "text": "And we're going to have talk about Swirl, this is one of the papers that uh >> the multi-step reasoning >> the multi-step reasoning that we talked about. What if we bring a different model, not in the context of generating reasoning traces, but in the context of like evaluating and giving feedback." }, { "i": 147, "speaker": "Azalia Mirhoseini", "text": "And we we're going to see that as well. But models, surprisingly or not surprisingly, like their own traces a lot more." }, { "i": 148, "speaker": "Speaker 4", "text": ">> For reasoning models, how are they taught to reason? Is it a hardcoded set of sequences or taught to do traces or they somehow fine-tuned to say, \"You need to do this much or this kind of thing that will cause the traces to generate." }, { "i": 149, "speaker": "Akanksha", "text": ">> So, I don't think there's a published piece of work that really covers this bit. Uh >> It's both, yeah. >> But it's it's a bit of both. At the same time, the base model did have the thinking capability to begin with, right? It would go do some amount of thinking." }, { "i": 150, "speaker": "Akanksha", "text": "One thing that we will cover in the classes this notion of outcome rewards model and process reward model and how you can use that feedback to get the model to improve." }, { "i": 151, "speaker": "Azalia Mirhoseini", "text": "Yeah, there's always some bootstrapping. Like, okay, here are different ways. Like, chain of thought itself, which is heavily used in the instruction tuning of data set, the shows the model some ways of like how to think. And there could be like some templates showing for the model fine-tuning." }, { "i": 152, "speaker": "Azalia Mirhoseini", "text": "But the things that the model, if you just ask it any question and it comes up with that, there's a whole lot of generalization that. So, models have gone way above like what the data or instructions that are used for training them." }, { "i": 153, "speaker": "Speaker 4", "text": "Uh can you repeat is there a way to make the number of samples dependent on the problem" }, { "i": 154, "speaker": "Akanksha", "text": "Please speak up. The number of samples, can they be dependent on problem difficulty? I don't think there's a published piece of work that does that." }, { "i": 155, "speaker": "Speaker 1", "text": "Uh but yeah," }, { "i": 156, "speaker": "Azalia Mirhoseini", "text": "we just So, So, when So, there is follow-up work to that where we use a reward model and then you can use that." }, { "i": 157, "speaker": "Azalia Mirhoseini", "text": "So, assuming the reward model has some notion of complexity, if you haven't solved it, it can guide like more sampling or whether it's repeated or parallel. Uh but that's definitely an interesting direction to explore." }, { "i": 158, "speaker": "Azalia Mirhoseini", "text": "[snorts]" }, { "i": 159, "speaker": "Akanksha", "text": "Okay, we need to get to the next, we haven't. Right. Yeah, let me cover these. Okay, so you now know about large language models and then you heard about how they learned how to think and then they learned how to reason." }, { "i": 160, "speaker": "Akanksha", "text": "So, what's next and why is this course relevant? So, a lot of what this course is about is that large language models as chatbots or as reasoning models are basically still single turn like or or just like in the chat format." }, { "i": 161, "speaker": "Akanksha", "text": "So, they're not accomplishing a task for you. They're fun to interact with, but they're not necessarily accomplishing a task for you." }, { "i": 162, "speaker": "Akanksha", "text": "What has happened this year almost in the last few months and it's been surprising is that agents like Claude or Deep Research have really enabled people to do real-world workflows." }, { "i": 163, "speaker": "Akanksha", "text": "So, they're agentic workflows which can achieve tasks that you ask them to do end-to-end. So, for example, if you want to go research and figure out where should I rent a home perhaps for the entire year if I want to take a class at Stanford." }, { "i": 164, "speaker": "Akanksha", "text": "The model would actually be able to do a whole bunch of analysis and like go look at a lot of different websites and actually summarize the results and give you pros and cons of different places. This used to not be possible before. So, this will cover a little bit of how that that becomes possible and how this course allows you to learn about that." }, { "i": 165, "speaker": "Akanksha", "text": "And then Claude Code for example, any of you are Codex which is the coding agent from OpenAI, if you're using that, what you will see is that just by giving instructions in English, you can modify files or you can like figure out test cases and and whatnot." }, { "i": 166, "speaker": "Akanksha", "text": "So, it really uh is becoming a coding productivity tool in in the day-to-day workflows for software engineers at this point in time. So, what what is the transition from LLM to agents?" }, { "i": 167, "speaker": "Akanksha", "text": "So, as I was giving you the example of Deep Research and of uh coding agents, basically the goal basically now the uh model can be given a goal and it will plan out the steps, then it will go interact with uh the environment, and based on the feedback, it will correct its steps until it achieves the goals or it will come back and say, 'I cannot achieve the goal.'" }, { "i": 168, "speaker": "Akanksha", "text": "So, this notion of like having some notion of a goal, uh taking actions towards that goal, getting the feedback, and then deciding when to stop, that's what makes agents uh different from the chatbot time of what you were doing before." }, { "i": 169, "speaker": "Akanksha", "text": "And um this also might require interacting with tools uh which are external to what the model itself is doing, getting some inputs from there, but it still stays on track of like whatever task it's uh choosing to accomplish, which might mean that it has to have some form of memory to keep track of the task that it's trying to accomplish." }, { "i": 170, "speaker": "Akanksha", "text": "So, what has been uh accomplished? So, like in in in several cases in simpler cases, you can achieve these uh um these end-to-end goals like Deep Research can like be accomplished end-to-end." }, { "i": 171, "speaker": "Akanksha", "text": "But, in most scenarios, chatbot times of what you were doing before." }, { "i": 172, "speaker": "Akanksha", "text": "And um this also might require interacting with tools uh which are external to what the model itself is doing, getting some inputs from there, but it still stays on track of like whatever task it's uh choosing to accomplish, which might mean that it has to have some form of memory to keep track of the task that it's trying to accomplish." }, { "i": 173, "speaker": "Akanksha", "text": "So, what has been uh accomplished? So, like in in in several cases in simpler cases, you can achieve these uh um these end-to-end goals like Deep Research can like be accomplished end-to-end." }, { "i": 174, "speaker": "Akanksha", "text": "But, in most scenarios, you're still having very static workflows. So, today what exists is closer to agentic workflows where you have um some sort of an input where you're giving a model the goal and then" }, { "i": 175, "speaker": "Akanksha", "text": "uh what this slide is showing you is that one model is perhaps uh giving the output like a solution and then another model is judging it and then deciding based on that whether the solution should be accepted. So, that's like one orchestration framework that's possible." }, { "i": 176, "speaker": "Akanksha", "text": "Uh Deep Research will be more of the second one where you're basically generating you're calling the LLM on multiple possible inputs and then aggregating these things to get the output which will be a summary of the deep research results." }, { "i": 177, "speaker": "Akanksha", "text": "So, this is a very cartoonish abstraction of what agentic workflows might look like and you can generalize from there, but instead of having like very open-ended loop that I was showing you here where you're" }, { "i": 178, "speaker": "Akanksha", "text": "supposed to go take an action from the environment and come back and do some feedback. Um, it's easier for open-ended problems to like construct this graph by hand of like how a human would do it and then like get this feedback perhaps from an LLM which is the LLM evaluator here." }, { "i": 179, "speaker": "Akanksha", "text": "So, a lot of the real world workflows still have this paradigm, but in certain cases we are starting to see signs of life for what's possible in this slide encoding being one and research being another one. So, typically the workflows will have things like LLM calls." }, { "i": 180, "speaker": "Akanksha", "text": "So, you would basically call an LLM with an instruction or some form of like input and then you're asking the LLM to give you an output. It will have some form of verifiers. We'll cover verifiers in a whole lecture by itself." }, { "i": 181, "speaker": "Akanksha", "text": "It will have some form of critics or judges which is effectively LLM as a judge paradigm. There might be tool calls. For example, for deep research you have to actually go search the web to figure out what exactly is the content that you should be looking at." }, { "i": 182, "speaker": "Akanksha", "text": "So, there might be tool calls of that form or getting some value about what exactly is the weather. Um, and similarly search would be another tool call. And then you would orchestrate these in some form of a workflow. So, a simplest workflow here would be prompt chaining." }, { "i": 183, "speaker": "Akanksha", "text": "So, similar to reasoning models where the task gets decomposed into subtasks, prompt chaining would simply be like you have chained a bunch of different subtasks that you're supposed to go accomplish to achieve the end-to-end goal. There might be routing." }, { "i": 184, "speaker": "Akanksha", "text": "Routing works for complex tasks where you say, \"Okay, if this is very complex, then go do this more complicated set of LLM calls, and if it's less complicated, then do this simpler workflow." }, { "i": 185, "speaker": "Akanksha", "text": "Um there might be parallelization where you like deep research is one example where you can have multiple LLM calls work simultaneously on researching different keywords that you give it as" }, { "i": 186, "speaker": "Akanksha", "text": "input to the LLM, and then uh finally you aggregate the output, or you might break the task into independent subtasks, and then uh combine uh the solution." }, { "i": 187, "speaker": "Akanksha", "text": "Or you might have this notion An orchestrator where you basically are actually using in simple words, LLM orchestrator where central LLM is actually doing the planning." }, { "i": 188, "speaker": "Akanksha", "text": "So, in Claude code, you actually start to see that that there is You might have an evaluator or a judge." }, { "i": 189, "speaker": "Akanksha", "text": "So, instead of like getting feedback from real-world like user or from some sort of an actual running of a unit test, it actually might just use LLM as a judge, and we will have uh some homeworks that And then you might have verifiers." }, { "i": 190, "speaker": "Akanksha", "text": "Verifiers are things where you can actually verify the output. So, in code for example, if you run the code, how do you know that this code is correct? So, typically as software developers, we write unit tests." }, { "i": 191, "speaker": "Akanksha", "text": "Uh so, similarly, verifiers might be running some kind of unit test to check whether what LLM generated is correct." }, { "i": 192, "speaker": "Akanksha", "text": "And in domains which are verifiable, uh math, code, um and other domains that are more rule-based, uh this verification is a good way to give feedback back to the model so that um it can correct its steps." }, { "i": 193, "speaker": "Akanksha", "text": "So, most of these uh workflows mean that the uh the LLM needs to be better at planning. It needs to be better at multi-step reasoning, and it needs to be better at self-improvement. Like when it" }, { "i": 194, "speaker": "Akanksha", "text": "makes mistakes, it needs to be able to correct itself. So, effectively, these are sort of new paradigms that the current set of LLMs just with reasoning were not quite accomplishing, and these are some of the topics we do plan to cover in subsequent lectures." }, { "i": 195, "speaker": "Akanksha", "text": "And just to drive the point home around coding agents, so this is a very simple example." }, { "i": 196, "speaker": "Akanksha", "text": "This is actually I should have updated the slide to Claude code, but basically, what this is showing is that you have an LM agent, and it's interacting with the computer these days just with the terminal. You give it an instruction where you might tell it that you want to implement a" }, { "i": 197, "speaker": "Akanksha", "text": "task, and then has some navigation of repository, searching of files, all of these tool calls around like viewing files, editing lines, and then it will go execute a bunch of commands in the terminal, and based on the output, it might be like, \"Okay, I need to go edit this other file, or I need to go look at this other file.\"" }, { "i": 198, "speaker": "Akanksha", "text": "So, if you look at this loop, this very much mirrors uh what was so, this was not quite reliable last year, and it's just starting to get reliable in what we're seeing in the realm of coding agents now. I mean, I think the paradigm is very much the same." }, { "i": 199, "speaker": "Akanksha", "text": "It's mostly a matter of more powerful models, and then better RL. RL with verifiable rewards is working. The training time scaling is working well. I mean, it's computer data, right?" }, { "i": 200, "speaker": "Akanksha", "text": "One of those two, but like just also, once the models start to get better, there's the self-improvement loop that kicks in." }, { "i": 201, "speaker": "Speaker 1", "text": "One of those two, but like just also, once the models start to get better, there's the self-improvement loop that kicks in." }, { "i": 202, "speaker": "Akanksha", "text": "Uh because you can now generate tasks, and the tasks become more reliable." }, { "i": 203, "speaker": "Akanksha", "text": "So, in the published pieces of work, if you look at say Code Monkeys, which we coded last year, if you can generate unit tests for whatever code the model generated and then see if the generated unit tests are making things better, then that that that's a very good way to verify things, right?" }, { "i": 204, "speaker": "Akanksha", "text": "Yeah, like model-based. It seems like to some extent, if we have good verifiers, um we can make the model generate the right code. Um but the question is how do we do that?" }, { "i": 205, "speaker": "Azalia Mirhoseini", "text": "And there are ways to do it. And we would be happy to for you guys to take research projects in that domain, too. But obviously, it's an active research area, but there are ways to make improvements that we can discuss. what you're alluding to is this notion of generator-verifier gap." }, { "i": 206, "speaker": "Azalia Mirhoseini", "text": "So, it's easy for models to generate a whole bunch of nonsense or sensible set of reasoning traces or useful set of content." }, { "i": 207, "speaker": "Akanksha", "text": "content. But at the end of the day, whether that's useful or not, we need feedback loop for that. And if you're creative writing, how much feedback can you get? So, human feedback ends up becoming a bottleneck." }, { "i": 208, "speaker": "Akanksha", "text": "In domains where you can have good feedback, that's where it's possible to continue to improve the model. And that's where, but getting robust verification, a lot of robust verification is hard." }, { "i": 209, "speaker": "Akanksha", "text": "So, uh Azalia will cover a paper that she did in her lab about like how to combine verifiers. And but what verification continues to be one of the bottlenecks in this space to make them better." }, { "i": 210, "speaker": "Akanksha", "text": ">> Um just like intuitively, like in pre-training, you get abilities to find the same logical abilities." }, { "i": 211, "speaker": "Speaker 5", "text": ">> So, just like to me, intuitively, I'm trying to figure out why there's a significant drop. >> Because like we're sort of dividing up what they know. >> And we have logical abilities that I guess shouldn't exist >> Mhm." }, { "i": 212, "speaker": "Speaker 5", "text": ">> in pre-training." }, { "i": 213, "speaker": "Speaker 1", "text": ">> Um So, yeah, I guess like I'm not completely resonating with like I think this is still an active area of research. So, there is different set of opinions around what really improves the model. Is it RL or is it like the pre-training diverse data by itself?" }, { "i": 214, "speaker": "Akanksha", "text": "And I don't think there's single point of consensus at this point in time. Both processes help. Um what you just said, uh which I'll repeat for the class, is that if pre-training is the place where uh the model after repeated sampling should be able to at least have one solution correct out of if it was generating a large number of samples, then uh this notion of giving feedback should improve the pass@1 accuracy, but should not improve the model. Why is it like such a big jump? But I think um that whole loop is not completely well understood. It's like the first signs of" }, { "i": 215, "speaker": "Akanksha", "text": "the model. Is it RL or is it like the the pre-training diverse data by itself? And I I don't think there's single point of consensus at this point in time. Both processes help." }, { "i": 216, "speaker": "Akanksha", "text": "Um what you just said, uh which I'll repeat for the class, is that if pre-training is the place where uh the model after repeated sampling should be able to at least have one solution correct out of if it was generating a large number of samples, then uh this notion of giving feedback should improve the pass at one accuracy, but should not improve the model." }, { "i": 217, "speaker": "Akanksha", "text": "Why is it like such a big jump? But I think um that whole loop is not completely well understood. It's like the first signs of AI and it starts to get commercialized, but I think there's a lot more research still open in this area. So, that's only one uh way of thinking about it." }, { "i": 218, "speaker": "Akanksha", "text": "There are other um at least some signs of life that say that you can continue to do RL and then that will continue to improve the model. Yeah. >> Okay." }, { "i": 219, "speaker": "Akanksha", "text": "So, uh so this is basically covering a lot of the same set of points, but uh one abstraction that is worth taking away from this particular chart is that even when you give the model a goal, it has to clarify the user intent. So, it's like not always obvious to it what the user wants." }, { "i": 220, "speaker": "Akanksha", "text": "And then it might go search for relevant files and then whatever um set of uh actions it chooses to take, often times it needs some form of verification. In this particular case, it's based on um passing the tests and it might actually generate the tests that need to pass, which um which is also uh mentioned here." }, { "i": 221, "speaker": "Akanksha", "text": "But the the key idea is that if you give it a task, how does it come back and how do you know that it's going to complete what you asked it for? Often times the users will not specify the problem well enough." }, { "i": 222, "speaker": "Akanksha", "text": "So, asking clarifying the user intent so that it knows what to go look for and how to verify it starts to become important and in models like GPT-3, I think they have seen a lot of these traces end to end." }, { "i": 223, "speaker": "Akanksha", "text": "So, they are able to like do the planning, the reasoning, the multi-step reasoning and like come back with the conclusion, but a lot of this is needed for um completing tasks end to end or having an end to end goal. And where this has been super useful is if you have repeated repetitive tasks like code migrations or version upgrades or if you need to restructure the code base." }, { "i": 224, "speaker": "Akanksha", "text": "Or if you have tasks that involve data engineering where you have to basically do some sort of extract the data and then do some sort of clean up on the data or if you have to do some sort [snorts] of data warehouse migration. So, a lot of this work tends to be extremely repetitive and it's much easier to just delegate it to coding agents." }, { "i": 225, "speaker": "Akanksha", "text": "Same for unit tests often times like much easier to like start to give it to coding agents. Another area where agents have become extremely prevalent is in customer support." }, { "i": 226, "speaker": "Akanksha", "text": "It's one of the most thankless jobs that when you are on the customer support side of things you have to do and using LLMs to do that definitely streamlines the experience in interesting ways. So, one simple set of examples ends up being like you can use them to do live transcriptions." }, { "i": 227, "speaker": "Akanksha", "text": "So, that gives you a very nice record and you're seeing that even for meetings these days. Another simple example is that you can have knowledge assist. So, if you have a database of information, the customer support agent does not need to know everything." }, { "i": 228, "speaker": "Akanksha", "text": "They can consult with the LLM and get an answer and surface the relevant article which is better than having just search like an index and search by itself. Smart replies." }, { "i": 229, "speaker": "Akanksha", "text": "So, in chat often times this has existed for a while, but you can use an agent to like give chat responses and then having a call summary can help you um you can use the call summary to really improve the customer experience by itself as well." }, { "i": 230, "speaker": "Akanksha", "text": "So, overall, I think in the customer support area, there are [snorts] multiple companies that are going after this area, but uh there are different segments of the problem that can be addressed using LLMs and they have been super useful and then there's also the end-to-end stuff that's starting to happen here." }, { "i": 231, "speaker": "Akanksha", "text": "And then the third example that we will use and actually some of the homework is that if you have very complex topics and you want to provide a comprehensive report, um earlier it used to be that you had to do a literature review and then you had to summarize each paper and then you had to synthesize these things and these days you can just give it to an LLM and actually do it for you and um study maybe more articles than you would." }, { "i": 232, "speaker": "Akanksha", "text": "So, typically, if you give it an example like uh say 2022 Winter Olympics opening ceremony, it will identify what references to go look at, then it will" }, { "i": 233, "speaker": "Akanksha", "text": "construct an outline of like, okay, these references are relevant or not, and then summarize each reference as to what is the relevant content and then combine them to create a full-length article, um which is uh very impressive in certain cases and and you'll get to try it in one of the homeworks." }, { "i": 234, "speaker": "Akanksha", "text": "And then even uh more uh forward-looking um these research agents are starting to be used as uh as AI scientists. So, basically, they're starting to assist scientists in say uh solving math problems or solving science problems." }, { "i": 235, "speaker": "Akanksha", "text": "So, here the LLM is used as a brainstorming thing to come up with ideas for the idea generation phase, then the experiment iteration phase, it might actually help you iterate on the experiments that you are hoping to work on. This is from the AI scientist paper." }, { "i": 236, "speaker": "Akanksha", "text": "And then in the paper write-up phase, it will help you improve uh the paper write-up by itself." }, { "i": 237, "speaker": "Akanksha", "text": "And what's interesting uh is that even though these models hallucinate, just the notion that they can come up with so many different set of ideas as an AI scientist it might actually brainstorm or like give you ideas outside what you would have thought if you had just like taken a bunch of courses or if you're a researcher and you've been in the field for a long time sometimes the reading the web allows these LLMs to come up with ideas that are way outside the box and can be a very good brainstorming um way uh so these have been super useful for in the AI [clears throat] scientist" }, { "i": 238, "speaker": "Akanksha", "text": "style of work. Okay, so let's move to course logistics but before that do we have any questions?" }, { "i": 239, "speaker": "Speaker 6", "text": ">> To clarify so is reasoning and chain of thought baked in training or is it just prompt engineering that was accidentally discovering these kind of capabilities in larger models? And if so are there other emergent behaviors besides our reasoning and chain of thought that you think might be discovered?" }, { "i": 240, "speaker": "Akanksha", "text": "So uh I mean it was not baked in by design it was basically discovered as in like it was I mean we gave it hard problems and then we saw that it was basically by having reasoning chains it was doing better." }, { "i": 241, "speaker": "Akanksha", "text": "So the GSM8K was the first paper that kind of showed signs of life of this and then with larger model like Palm we actually saw that this was a very big deal and" }, { "i": 242, "speaker": "Akanksha", "text": "one of the examples there was that it could explain jokes that was very impressive and then from there the reasoning models have emerged. So it was not but it has read all of the web so it's it has definitely seen data which is like more methodical and systematic." }, { "i": 243, "speaker": "Azalia Mirhoseini", "text": "But the reasoning models are trained to be reasoning more and more, right? So it's the entire reasoning is not an emergent behavior. The reasoning models they're trained to be thinking but like the models are convergent so the models are going to be trained such that they know when they need a lot of reasoning and when they don't and to generate the answers." }, { "i": 244, "speaker": "Azalia Mirhoseini", "text": "But chain of thought originally was an emergent behavior. They noticed that oh, if we explain things to the model, it's better. >> of emergent behaviors in large models." }, { "i": 245, "speaker": "Akanksha", "text": ">> I don't think it's I would see it as emergent behaviors per se, but I think as I was mentioning, we're usually going looking for certain things, right? So in the agentic workflows, what we're looking for is planning, which is a form of reasoning." }, { "i": 246, "speaker": "Akanksha", "text": "We're looking for multi-step reasoning, which we will cover in class as well. And then we're looking for self-improvement or self-correction." }, { "i": 247, "speaker": "Akanksha", "text": "So all of these capabilities would be nice to have in what gets the models there is a set of questions that are worth there are papers on this kind of thing." }, { "i": 248, "speaker": "Azalia Mirhoseini", "text": "Like self-correction, backtracking, things like that is like yes, you could call it emergent, but you could also say think about it that they have seen this kind of behavior and it's kind of like it's reinforced in the way that they're fine-tuned. So it's Yeah, it's hard to say." }, { "i": 249, "speaker": "Azalia Mirhoseini", "text": "Okay. Um so let's go through the logistics for the class. Um here is a list of all the amazing topics that you're going to learn about. Um they are um yeah. This is very cool." }, { "i": 250, "speaker": "Akanksha", "text": "[laughter] >> I think the main thing to remember here is that the overall theme stays the same and then we'll also have guest lectures around from folks in Frontier AI Labs" }, { "i": 251, "speaker": "Akanksha", "text": "and covering things all the way from say how post-training has evolved or say even multimodal agents and in robotics. So it's it's a it's going to be a mix of lectures and guest lectures and of course you have project presentations." }, { "i": 252, "speaker": "Azalia Mirhoseini", "text": ">> So, here is some logistics that you all know the pre-record there. Just make sure that you are comfortable with these pre-reqs as you uh register then in the class. So, we have the external website, but uh make sure that you're uh you check Canvas." }, { "i": 253, "speaker": "Azalia Mirhoseini", "text": "That's where we send out the latest updates at all times. Um and um we will also upload the lectures before we start every class so you have access to the lectures. Uh and we will also have the due dates for all the assignments and all the projects, so you can check it out." }, { "i": 254, "speaker": "Azalia Mirhoseini", "text": "Um So, this quarter we're going to have three homeworks. That's a difference from between this class and that's one of the differences." }, { "i": 255, "speaker": "Azalia Mirhoseini", "text": "So, we have one more amazing homework for you all that we've designed and the TAs have done a great job and that helps you um hopefully learn uh learn these topics better and more in depth. Um And then we also have a course project." }, { "i": 256, "speaker": "Azalia Mirhoseini", "text": "So, the course project is where you can kind of unleash your creativity and like your way of like building agentic systems or going deeper into a question uh and designing experiments around it and see what works and what doesn't." }, { "i": 257, "speaker": "Azalia Mirhoseini", "text": "Um we will provide some um kind of examples or suggestions, but it could be completely on you how to design this project. We'll have some examples from last year like the successful projects from last year as well." }, { "i": 258, "speaker": "Azalia Mirhoseini", "text": "Uh it's going to be uploaded to um Canvas or Actually, we'll display them on the website. >> Oh, on the website. Oh, cool. >> Yeah. The public website, so you can all see that." }, { "i": 259, "speaker": "Akanksha", "text": "Um So, yeah. >> the course project, I think the main thing to remember is that uh we will have API credits and you can work in teams of two to four people." }, { "i": 260, "speaker": "Azalia Mirhoseini", "text": "Right? >> Yeah. Or up to four, I mean to four. Okay. It can be one if you really want that. We we we suggest you team up with others. Um at the very least, you have more credits collectively, so you can run more experiments." }, { "i": 261, "speaker": "Azalia Mirhoseini", "text": "But then you might also uh find a friend collaborator along the way and you can um um do something bigger. Um and we are going to have this um What is this? The course project. Yeah, we're we're going to uh we're going to let you know about some uh some of the previous years and some ideas that you can take." }, { "i": 262, "speaker": "Azalia Mirhoseini", "text": "Um here are some course project examples that are acceptable. For example, a new evaluation data set or a new benchmark. Um you can design a project around the reliability of an agentic system that already exists." }, { "i": 263, "speaker": "Azalia Mirhoseini", "text": "You can take a benchmark that exists and try to hill climb on it with whatever idea great idea that you have. Um and then or you can just be. You're going to have a whole lot of papers covered in this course and we have we are showing all of that to you." }, { "i": 264, "speaker": "Azalia Mirhoseini", "text": "They're all on the website. So as you as we give these lectures, we kind of expect you to also have read those papers or read the papers along the way and develop better and deeper intuitions on how these methods work." }, { "i": 265, "speaker": "Azalia Mirhoseini", "text": "And your project could always be improving those or like questioning one of the decisions that they have made and trying to uh change that or uh or evaluate that. Um Negative examples are like a survey paper or like we just don't want we want something researchy here." }, { "i": 266, "speaker": "Azalia Mirhoseini", "text": "paper or like we just don't want we want something researchy here. So we just don't want like just an app that you put together and you just show us something. It has to be like Here is like, here is the hypothesis." }, { "i": 267, "speaker": "Azalia Mirhoseini", "text": "Here is the question that we wanted to answer. Here is the type of like improvement we wanted to see or the type of like properties we wanted to analyze and then we have built that, right? We want something more than live coding basically in this." }, { "i": 268, "speaker": "Akanksha", "text": ">> Um I mean in terms of milestones, I think that it's worthwhile to remember that you want to start early. So it's you need to have a project proposal somewhere around I think we put it early October. And then uh It's on the website, yeah." }, { "i": 269, "speaker": "Akanksha", "text": "It's on the website and then for the midterm project presentation, we do want you to have some progress." }, { "i": 270, "speaker": "Akanksha", "text": "So 2 weeks after the project proposal, we do expect you to have like so start thinking about like what experiments you want to run by the time you get to the project proposal so that you actually go run them for the midterm projects." }, { "i": 271, "speaker": "Akanksha", "text": "And so we do expect you to have made some progress in the midterm project presentation. It should not just be here is a proposal and then the final report has a lot of weight and then the final poster presentation uh will be the end of the quarter." }, { "i": 272, "speaker": "Azalia Mirhoseini", "text": "Um yes. I think so." }, { "i": 273, "speaker": "Speaker 5", "text": "If you add to it, right? See this one name I want to name it in the past study, right?" }, { "i": 274, "speaker": "Azalia Mirhoseini", "text": "I mean as long as you're doing actual work and it's not just like reusing the work from" }, { "i": 275, "speaker": "Akanksha", "text": "Um We had students publishing papers out of their projects uh in the last quarter." }, { "i": 276, "speaker": "Azalia Mirhoseini", "text": "So that's something you could it could be one of you or many of you this time again turning your research project, working more on it and then turning it into a publication and conferences. And so here is a save the date for our poster presentation." }, { "i": 277, "speaker": "Azalia Mirhoseini", "text": "It's going to be December 12th 4:00 to 6:00 p.m. So we would want you to be there and present the posters. We'll have people from industry joining and you can that's where you can brag about what you've done and just meet new people." }, { "i": 278, "speaker": "Azalia Mirhoseini", "text": "Here is the course grading rubric. So we have three homeworks. That's 50% of your entire grade and then another 50% is on the project. And the duration of these are kind of like a line with how much the grading is allocated to each of these." }, { "i": 279, "speaker": "Azalia Mirhoseini", "text": "And we of course expect you to honor the honor code. And let's see. Office hours will be posted on Canvas. You can ask your questions." }, { "i": 280, "speaker": "Azalia Mirhoseini", "text": "We really really encourage you to ask questions on EdStem and make it public so others probably if you have a question it's very likely others have the same question. So please like do broadcast this to everybody." }, { "i": 281, "speaker": "Azalia Mirhoseini", "text": "EdStem is again you can ask questions and then GradeScope is where you submit your project milestones and everything else. Yeah, the homeworks and everything." }, { "i": 282, "speaker": "Azalia Mirhoseini", "text": "We have the following late policy that we think it's kind of like between the generous kind of like I am generous side of things and because the class is big this quarter, we really can't make any exceptions. So, try to use these late days wisely." }, { "i": 283, "speaker": "Azalia Mirhoseini", "text": "Uh Audits are not allowed, but we will have the course the videos on YouTube eventually." }, { "i": 284, "speaker": "Speaker 1", "text": "Eventually. And that's it. Any questions? No? All right. >> Okay. Thanks everyone." } ]