[ { "i": 0, "speaker": "Speaker 1", "text": "Hello everyone. Uh my name is Bonnie. Um I am a researcher at Google Deep Mind and uh today I'm excited to talk about the paper the art of scaling reinforcement learning compute for LLMs." }, { "i": 1, "speaker": "Speaker 1", "text": "Uh so we know that RL training has became the new scaling frontier. Uh we know pre-training teaches the model about how the world works but RL really unlocks the capabilities that matters" }, { "i": 2, "speaker": "Speaker 1", "text": "the most. Uh that includes test time thinking, agentic capabilities and things like scientific discovery. Uh so the scaling loss for pre-training is understood pretty well. Uh we know that" }, { "i": 3, "speaker": "Speaker 1", "text": "there's this power law where the loss basically decreases log linearly with respect to the amount of compute. Uh but what about scaling RL training? So so far that's been pretty vibe based. Um" }, { "i": 4, "speaker": "Speaker 1", "text": "there has there are so many algorithms, many design choices and not a quite scaling framework yet. Uh so can we bring can we bring the same scaling framework? Uh so in this paper they" }, { "i": 5, "speaker": "Speaker 1", "text": "found a scaling curve for RL training." }, { "i": 6, "speaker": "Speaker 1", "text": "Uh so basically RL training follows this like sigmoid scaling curve where uh so the x- axis is compute and the y- axis is the expected reward for your RL setup. And they found that you can" }, { "i": 7, "speaker": "Speaker 1", "text": "basically fit the sigmoid curve uh during uh during training uh that's defined by the asintoic ceiling uh the compute efficiency and the inflection point. Uh so we have the asintoic" }, { "i": 8, "speaker": "Speaker 1", "text": "ceiling which basically dictates how high your reward can go. So this is already different from pre-training. In pre-training your loss always decrease but in RL you're actually bounded uh by" }, { "i": 9, "speaker": "Speaker 1", "text": "how far your curve can go. And uh you also have compute efficiency which dictates how how steep your learning curve is. Uh so what this also means is uh if you look at this graph if you stop" }, { "i": 10, "speaker": "Speaker 1", "text": "the training too early your results could actually be really misleading because you could have some curves that starts really slow in the beginning but take off very drastically later. So we" }, { "i": 11, "speaker": "Speaker 1", "text": "see that RL training is very different from pre-training. It follows the sigmoid curve that's defined by this a uh that's asmtotic ceiling and b the compute efficiency. So with different" }, { "i": 12, "speaker": "Speaker 1", "text": "recipes we you hit pretty different ceilings. Uh so they looked at uh different recipes like scale RL QN uh deepseek and you see like pretty different training curves uh which is really interesting. Uh so this tells us" }, { "i": 13, "speaker": "Speaker 1", "text": "that these algorithmic choices fundamentally limits uh what you can achieve and you can't just train longer to fix a bad recipe. So it's very important to have the right recipe. So" }, { "i": 14, "speaker": "Speaker 1", "text": "how can we build the best scale RL recipe? Uh so this is the complete scale RL recipe uh that the paper came up with and we'll go into each of the components a little bit. So there are a lot of" }, { "i": 15, "speaker": "Speaker 1", "text": "design choices and each of these design choices actually affects different aspects of scaling. So we have things like the loss function and the train inference discrepancy. So these actually" }, { "i": 16, "speaker": "Speaker 1", "text": "limits how high the ceiling like how high you can go how high your curve can go. Uh so this is really important because you want you definitely want your curve to go as high as possible." }, { "i": 17, "speaker": "Speaker 1", "text": "There are also things like off policyiness and uh normalization adaptive sampling that usually affects the efficiency of your training. Uh so I think uh so yeah so I'll go into off" }, { "i": 18, "speaker": "Speaker 1", "text": "policyness first which is a pretty big problem in large scale RL. Uh so why do we do off policy RL? Uh that's because on policy RL it's very inefficient. What does on policy RL mean? Uh it's very" }, { "i": 19, "speaker": "Speaker 1", "text": "simple. So you run your generate so you generate a bunch of samples. uh you get your rollouts and then and then you run a training step and then with your new weights you run a new step of generate" }, { "i": 20, "speaker": "Speaker 1", "text": "to get the samples and then you run training again. Uh and you see that um you see that while you're running running generating uh while you're running generate some of these samples" }, { "i": 21, "speaker": "Speaker 1", "text": "go really long. So for large amounts of time your GPUs are basically just sitting idle. Uh so this is a very big problem. Your GPUs are basically idle like for large percent of the time. Uh" }, { "i": 22, "speaker": "Speaker 1", "text": "so what can we do to fix that? So one way is to do async RL. So we basically overlap generate and training. So we do a gen so we run a step of generate we get a bunch of samples. Uh we use those" }, { "i": 23, "speaker": "Speaker 1", "text": "samples to do training but at the same time uh we run generate at the same time. So we're always always running generating and training at the same time. We're overlapping uh we're" }, { "i": 24, "speaker": "Speaker 1", "text": "overlapping the two. Uh so we see but the problem with this is that when you're running training uh the data you're training on could be generated by weights from few steps ago. So it's not" }, { "i": 25, "speaker": "Speaker 1", "text": "so your data is from a different set of weights than the weights you're training on. So this means that async RL is off policy. So the data generating policy the data generating weights uh is old" }, { "i": 26, "speaker": "Speaker 1", "text": "compared to uh compared to the training policy compared to the set of weights you're training. There's also uh pipeline RL which takes it even a step further. Uh so pipeline RL we do these" }, { "i": 27, "speaker": "Speaker 1", "text": "like inflight weight updates. So as you're generating a trajectory so for agentic tasks these trajectories can get really long. uh they we basically say if we're generating a trajectory and midway" }, { "i": 28, "speaker": "Speaker 1", "text": "during the trajectory a new set of weights is ready we just like swap in a new set of weights midway and then we continue the trajectory from there so you could have samples that are" }, { "i": 29, "speaker": "Speaker 1", "text": "generated by different set of weights uh which is pretty cool and uh yeah we can look at how pipeline RL compare with like offline off policy and we see that pipeline RL is generally a bit more" }, { "i": 30, "speaker": "Speaker 1", "text": "efficient uh but all of them converges to about the same ceiling uh there's also a very important parameter in pipeline RL which is how far how old are the data that you are willing to train" }, { "i": 31, "speaker": "Speaker 1", "text": "on. Uh so so K uh so eight here would just mean like uh you're still training uh you could be training on data generated from weights eight steps ago." }, { "i": 32, "speaker": "Speaker 1", "text": "So we see that uh a value of like four and eight are usually fine but then once you get to something like 12 the difference is too big then like the algorithm just starts plateau and" }, { "i": 33, "speaker": "Speaker 1", "text": "doesn't work anymore. Uh so this is something you have to be um very careful of. Okay so that's more on the infra side how we handle uh off policy on infra side. What about on the algorithm" }, { "i": 34, "speaker": "Speaker 1", "text": "and loss side? Um so how do we handle off policy on the loss side is we do something called important sampling. So uh let's just ignore this term for now." }, { "i": 35, "speaker": "Speaker 1", "text": "So the term here this is basically policy gradient. So that's the loss function for RL. Um but the problem with this term here is you have the learner and the sampler which are different. Uh" }, { "i": 36, "speaker": "Speaker 1", "text": "your sampler is usually a bit older than the learner. So here we basically add a different term or we add this like this ratio to correct for the for this mismatch. Uh so we basically add this" }, { "i": 37, "speaker": "Speaker 1", "text": "term that's basically the probability uh from the learner divided by the probability from the sampler uh to correct for this distribution mismatch." }, { "i": 38, "speaker": "Speaker 1", "text": "Uh so there's a lot you can a lot of math you can do on how you exactly handle this. Uh so there's the very popular GRPO. So they how they handle this ratio is um they clip this ratio." }, { "i": 39, "speaker": "Speaker 1", "text": "So we've always ideally we want this ratio to be about one because uh ideally like the sampling policy and the training policy are like pretty close to each other. So we want it to be around" }, { "i": 40, "speaker": "Speaker 1", "text": "one but sometimes it can it can get a bit too all over the place. So in GPO they basically clip this ratio to one minus epsilon to uh between one minus epsilon and one plus epsilon. And then" }, { "i": 41, "speaker": "Speaker 1", "text": "uh so there's also which is this uh very popular algorithm from bite dance. Uh so what they realize is with GRPO your policy entropy actually quickly collapse. Uh because uh the clipping is" }, { "i": 42, "speaker": "Speaker 1", "text": "restricting the trust region but it also restrict like the upper clip is restricting how much your policy is uh how much exploration your policy is doing. So they come up with this" }, { "i": 43, "speaker": "Speaker 1", "text": "strategy called clip higher. It literally just means clipping higher. Uh so on the when you do the clipping uh so on the upper end you just do a you just set a higher epsilon. So you have this" }, { "i": 44, "speaker": "Speaker 1", "text": "like asymmetric clipping. uh there's also GSPO so they do like sequence level important sampling so instead of instead of running this important sampling term on the token level we can do it on the" }, { "i": 45, "speaker": "Speaker 1", "text": "entire sequence as well and then there's syso yeah there are many variations um so previously we were like clipping these token updates which means we're basically dropping tokens uh where the" }, { "i": 46, "speaker": "Speaker 1", "text": "ratio is too off uh so here they say we don't really want to drop tokens can we just clip so they came up with this trick to basically clip the important sampling weight directly to uh to" }, { "i": 47, "speaker": "Speaker 1", "text": "stabilize training uh And we can see how different loss functions affects the different uh different asintotic ceiling. So in this setup syso and gbo worked pretty well. Uh you see the curve" }, { "i": 48, "speaker": "Speaker 1", "text": "like uh working out pretty well. But in this case like dappo actually plateaued very quickly. So for different domains you really want to get the algorithm right otherwise otherwise you just" }, { "i": 49, "speaker": "Speaker 1", "text": "cannot um you just cannot get a good enough reward. Okay. And now I want to get into one of the biggest challenge actually in uh large scale RL is train inference discrepancy in RL. So it's a" }, { "i": 50, "speaker": "Speaker 1", "text": "bit of a weird issue. Um basically your training stack and inference stack are optimized for completely different things. Uh so this these result in a lot of inherent mismatch just due to" }, { "i": 51, "speaker": "Speaker 1", "text": "different kernels precisions. So in training you're using things like VLM for inference you're probably using like Megatron. Um so what this means is even for the exact same weight you can get" }, { "i": 52, "speaker": "Speaker 1", "text": "very different result from the training stack and the inference stack. Uh so this plot here we see uh so we have the exact same weight exact same data point and we get the probability from the" }, { "i": 53, "speaker": "Speaker 1", "text": "training stack and the inference stack and they should be the same. Uh so if they're the same they should line around the red line but we see that in reality they actually really scatters around the" }, { "i": 54, "speaker": "Speaker 1", "text": "red line. So sometimes they're actually like really different from each other despite like this is just like running the exact same way exact same data. Uh so this is actually a very big issue in" }, { "i": 55, "speaker": "Speaker 1", "text": "RL. So there's this Chinese paper. So in their case they originally their algorithm it's not working at all. So if you look at the gray curve over there like it's just not learning at all and" }, { "i": 56, "speaker": "Speaker 1", "text": "then they realized oh we have this pretty big train inference discrepancy across different layers and different components. So they went into it and fixed it one by one. So first they fixed" }, { "i": 57, "speaker": "Speaker 1", "text": "KV cache there's some precision mismatch and just by fixing that you get the you get the brown curve. So just by fixing that the algorithm just started working and then they fix like more like" }, { "i": 58, "speaker": "Speaker 1", "text": "normalization attention and rope and then you just see like uh the more like just fixing them one one by one the algorithm just works better and better and at the end you get a you get the red" }, { "i": 59, "speaker": "Speaker 1", "text": "curve which is like pretty stable RL training. So in the scale RL paper uh they do something similar. So they use this fix that basically cast these matrix multiplication to like FP32 for" }, { "i": 60, "speaker": "Speaker 1", "text": "both train and inference. So you see that just with this trick uh we have before this trick we have very scattered like scattered values but then after it it's much more centered around uh the" }, { "i": 61, "speaker": "Speaker 1", "text": "values are much more centered and aligned and just by this precision fix uh you can look at the curve over there they got like a plus 17% relative ceiling. So just by fixing precision you" }, { "i": 62, "speaker": "Speaker 1", "text": "can you can affect your training performance quite a lot. Um this is also really dangerous because uh if you look at the orange curve like there's nothing wrong like like it's not clear like what" }, { "i": 63, "speaker": "Speaker 1", "text": "is wrong. Um so these discrepancy they're not obvious errors. Uh they're just like silently corrupting your performance. Uh so you really want to be minimizing your discrepancy to um for" }, { "i": 64, "speaker": "Speaker 1", "text": "stable RL training. What else do we do in RL training? Another thing we do is um adaptive sampling. So there's this trick called zero variance filtering." }, { "i": 65, "speaker": "Speaker 1", "text": "It's really simple. It basically says we have a bunch of problems. Some of them are too hard. Some of them are too easy." }, { "i": 66, "speaker": "Speaker 1", "text": "If they're too if they're too easy, the model always gets like a reward of one or always gets pretty high reward. If they're too hard, the model always gets like a reward of zero. So we just like" }, { "i": 67, "speaker": "Speaker 1", "text": "skip these samples. We just like throw throw them away during training and we only focus on training on data points where the model sometimes can solve it, sometimes cannot. Uh so this really" }, { "i": 68, "speaker": "Speaker 1", "text": "focus the training compute on the learning frontier. Uh there's also no positive resampling is also pretty simple. Uh we say that as training progresses the model uh starts to like" }, { "i": 69, "speaker": "Speaker 1", "text": "master some problems. Uh so some of these they still pass the zero variance test but uh they still contribute like very diminishing learning returns. Uh so in no positive resembling we say we" }, { "i": 70, "speaker": "Speaker 1", "text": "track the historical pass rate uh per prompt. If uh if the model can solve like solve this problem more than 90% of the time uh we just throw it away. And uh so this allows us to progressively" }, { "i": 71, "speaker": "Speaker 1", "text": "focus on harder problems. Uh what this looks like in the training progression is you might start with like 5,000 problems. The model uh will learn to solve like maybe 200 of them and then we" }, { "i": 72, "speaker": "Speaker 1", "text": "just drop those. And so gradually we just have less and less problems to train on and we focus on the on the problems that are genuinely really challenging. And so the computer is never wasted on problems that are uh" }, { "i": 73, "speaker": "Speaker 1", "text": "that the model already mastered." }, { "i": 74, "speaker": "Speaker 1", "text": "And uh we see that with adaptive sampling uh this really improves compute efficiency uh just with like these two very simple tricks. And sometimes they also improve the ceiling as well which" }, { "i": 75, "speaker": "Speaker 1", "text": "is uh which is pretty cool. Uh so yeah so we talked about the scale RL recipe." }, { "i": 76, "speaker": "Speaker 1", "text": "Uh so we talked about different loss functions like gpo dapple and sispo and also the train inference discrepancy. So these two like they really affects the ceiling and just like how high uh how" }, { "i": 77, "speaker": "Speaker 1", "text": "high your performance can get. And then we also talk about off policyiness and adaptive sampling which is really affects your efficiency like how quickly how quickly your model uh is learn is" }, { "i": 78, "speaker": "Speaker 1", "text": "learning on the problems. So that was a very information dense uh technical talk. Uh what do we learn? Uh so we learned that our own training follows uh these sigmoid scaling patterns. So" }, { "i": 79, "speaker": "Speaker 1", "text": "that's different from the power laws. We learned that different recipes have different performance ceilings and uh we can predict large scale performance from small scale training runs. Uh we also" }, { "i": 80, "speaker": "Speaker 1", "text": "talked talked a lot about different trade-offs like off policiness uh adaptive sampling and different RL algorithm. Uh they also affect uh scaling in different ways. Some affecting the efficiency, some affecting" }, { "i": 81, "speaker": "Speaker 1", "text": "the uh the stealing. Uh so yeah. So yeah, I hope you gain some insights for your next um RL training runs." }, { "i": 82, "speaker": "Speaker 2", "text": ">> Hi. Um how generalizable is this recipe different?" }, { "i": 83, "speaker": "Speaker 1", "text": "Uh, I think it's pretty generalizable. I think um, yeah, I think it's a very systematic way to think about to think about RL scaling. So, I think it's pretty generalizable and it's probably a" }, { "i": 84, "speaker": "Speaker 1", "text": "really good good starting point. I imagine if you have super specific problems, you might like alter things a little bit, but it's usually like I think it's pretty generalizable." }, { "i": 85, "speaker": "Speaker 3", "text": ">> Yeah. I was wondering why why do you think the shape is a sigma shape? Uh, and that's my first question. And second question is like is there any fundamental limit to how high the" }, { "i": 86, "speaker": "Speaker 1", "text": "ceiling can be?" }, { "i": 87, "speaker": "Speaker 1", "text": "Yeah, that's a good question. Um, my I think because RL is so different, okay, RL is very different from deep learning. In that deep learning, you're training on a fixed data set. Uh, in RL," }, { "i": 88, "speaker": "Speaker 1", "text": "your model is generating the data that you're training on. So, I think once your model is getting to a good enough point, then you suddenly get a lot of like very valuable learning points. So," }, { "i": 89, "speaker": "Speaker 1", "text": "it takes off very quickly. Uh, so that that would be my guess as to why it's a sigmoid shape. Uh, yeah. Yeah. And as for the ceiling, uh yeah, there are definitely like fundamental limits. Uh" }, { "i": 90, "speaker": "Speaker 1", "text": "things we talked about here include um like the algorithm you're using and also just like the train inference discrep discrepancy if like if you have too big of a discrepancy just doesn't work that" }, { "i": 91, "speaker": "Speaker 1", "text": "well anymore. Um yeah, I think those are like some limits on the algorithms." }, { "i": 92, "speaker": "Speaker 4", "text": "Yeah, on the algorithm side." }, { "i": 93, "speaker": "Speaker 1", "text": ">> How much of it is it related to like the base model being it's like oh yeah like a really shitty base model is like very high. Yeah, I think the base model capability I think it affects both. Uh" }, { "i": 94, "speaker": "Speaker 1", "text": "so like with a strong base model with a strong base model maybe you can improve really quickly but for like worse base model like it takes it takes a while. So I think it affects both. Uh it's" }, { "i": 95, "speaker": "Speaker 1", "text": "actually not super Yeah, it's actually not super clear to me like whether how much it affects the ceiling because like I think some small models are actually really capable like if you just train it" }, { "i": 96, "speaker": "Speaker 4", "text": "long enough. Um yeah." }, { "i": 97, "speaker": "Speaker 1", "text": ">> Is that is that how you do based on it?" }, { "i": 98, "speaker": "Speaker 1", "text": "uh I guess uh not not not as so this not as much in this paper but I think yeah you see things like uh like cursor with like composer 2.5 just with like if you even if you don't have the biggest model" }, { "i": 99, "speaker": "Speaker 5", "text": "like if you train it train it long enough on like high high quality enough data like you can get you can get pretty far so yeah >> I think on the previous question the one of the reason why it is that the RL uh" }, { "i": 100, "speaker": "Speaker 5", "text": "reward reward function is a finite donate right so eventually you'll hit a series but if you do a dynamic then it's a dependent which brings me to my question actually. So it's it's" }, { "i": 101, "speaker": "Speaker 5", "text": "wonderful that like we can predict right so and then we can from a small scale we can find out where it's going but I wonder where the trade-off is on the the innovation of it or if it whether we" }, { "i": 102, "speaker": "Speaker 5", "text": "know it phases out or not and the reason why I'm asking is that like if you assume original neural network paper from 1958 right with the limited compute I'm pretty sure they hit a ceiling very" }, { "i": 103, "speaker": "Speaker 5", "text": "very quickly they they figured okay we cannot improve over this yeah >> but like I mean assuming here for instance 100,000 GPU hours we hit a ceiling and we cannot improve over that." }, { "i": 104, "speaker": "Speaker 5", "text": "But we we don't actually know if 10 million hours will do it or not. If we if we actually fix ourselves onto a recipe based machine learning approach and I wonder how much of it is a trade" }, { "i": 105, "speaker": "Speaker 5", "text": "up on we're actually giving up on some of the innovations that might actually pan out like for instance the adaptive sampling right yeah so it's a fantastic method but like say say it starts" }, { "i": 106, "speaker": "Speaker 5", "text": "lowering the sample zone once it achieves a certain task at say 90% 95% accuracy that might be that one or two very very rare cases that might actually save a life that like we might not be" }, { "i": 107, "speaker": "Speaker 5", "text": "samp sampling that because we think that we are already very very good at that particular task and then downsampling." }, { "i": 108, "speaker": "Speaker 5", "text": "So I I I want to get your sorry if this ends up being a too long of a question but where do you see the tradeoff of like say how predictable is good versus like the how much of an innovation space" }, { "i": 109, "speaker": "Speaker 1", "text": "there is." }, { "i": 110, "speaker": "Speaker 1", "text": ">> Um yeah that's a really good question. I think I think things are generally pretty predict predictable like if you just look at the look at the learning curves like once you yeah once you get" }, { "i": 111, "speaker": "Speaker 1", "text": "to a certain point like you you were not going to get much further than that. Uh on the innovation side yeah I think I think like this is still very early work in like systematically thinking about" }, { "i": 112, "speaker": "Speaker 1", "text": "these kind of things. So like so yeah I think everything on RL side is extremely early. There's like so much room for scaling like we haven't even talked about scaling like the scaling the" }, { "i": 113, "speaker": "Speaker 1", "text": "number of environments scaling the number of domains like there's just so much you can do there. So I think yeah people should definitely work on different animations and like the the" }, { "i": 114, "speaker": "Speaker 1", "text": "adaptive sampling I think it's like very simple tricks people came up with um but yeah I don't think but yeah I think people should definitely continue to work on animations >> um yes" }, { "i": 115, "speaker": "Speaker 6", "text": ">> uh super dab assembly where the basic type on those questions yeah that we thought that the model is good enough to solve yeah >> but is it possible that you know after discarding those codes and codes easy" }, { "i": 116, "speaker": "Speaker 6", "text": "questions yeah >> the model will probably like uh gradually lean towards uh techniques or skills that are more tailored towards the so the so hard questions. Yeah. And uh in return like uh results in a like" }, { "i": 117, "speaker": "Speaker 6", "text": "regression of those cult questions was there like risk of you know this hard." }, { "i": 118, "speaker": "Speaker 1", "text": ">> Yeah. Yeah. That's a really good question. I think uh I think it's not super well studied. I actually think some of the adaptive sampling things it's not very principled because it does" }, { "i": 119, "speaker": "Speaker 1", "text": "change your training distribution in like unexpected ways. Um so yeah no I think people should study that more and I do think it changes behaviors in like very unexpected ways like sometimes like" }, { "i": 120, "speaker": "Speaker 1", "text": "your model could just like if you overtrain it on like very specific type of problems like you get very very strange behaviors from those type of uh type of things. So yeah so I think" }, { "i": 121, "speaker": "Speaker 7", "text": "people should be very very careful about that. Um yeah >> um I know that most are like out have outcome based awards. Um have we studied any ceiling for process based rewards or do we have any scaling as to that?" }, { "i": 122, "speaker": "Speaker 1", "text": ">> Um yeah so I think for a lot of domains um for a lot of domains like people are using like uh reward mus models or rubrics. So I think all these like scaling laws they still pretty much hold" }, { "i": 123, "speaker": "Speaker 1", "text": "uh maybe some maybe with some reward hacking but it still pretty much holds." }, { "i": 124, "speaker": "Speaker 7", "text": "um with process rewards. Um I'm I'm not quite sure actually. I don't Yeah, >> there is an R having a lot of like reward shaping." }, { "i": 125, "speaker": "Speaker 7", "text": ">> So they do a lot of reward shipping." }, { "i": 126, "speaker": "Speaker 7", "text": "Most of these cases it's just like a binary reorder." }, { "i": 127, "speaker": "Speaker 7", "text": ">> Um do you believe that the the similar ceiling codes for trinity?" }, { "i": 128, "speaker": "Speaker 1", "text": ">> Yeah, I think I've seen like very simplified uh settings of process rewards where you do get like different partial rewards like per step and you sum over that. Uh, so I think in those" }, { "i": 129, "speaker": "Speaker 1", "text": "cases it still holds, but for if you get into like more nuance cases of the like different process, like very rewarding, very specific parts of the process." }, { "i": 130, "speaker": "Speaker 1", "text": "Yeah. I'm not I'm not too sure. I'm not too sure what happens there. Yeah. I think it would be it probably looks something different. Yeah. Over there." }, { "i": 131, "speaker": "Speaker 8", "text": ">> Yeah. This was really interesting. It got be thinking more from a product perspective. I'm wondering if it's makes sense to start labeling models that we, you know, push out and deliver from like" }, { "i": 132, "speaker": "Speaker 8", "text": "this predictive metric, right? like would it make sense to have that label like as a marker for this is where we think the model is and its potential or or there's no >> Yeah, I think I think that would be" }, { "i": 133, "speaker": "Speaker 1", "text": "really I think that would be really useful. It's actually yeah, it's like really hard to assess like the potential of the models like even for the open source Chinese ones like you can get" }, { "i": 134, "speaker": "Speaker 1", "text": "like very surprising results from different just just by postraining it." }, { "i": 135, "speaker": "Speaker 1", "text": "So I think yeah I think that that certainly could be really interesting." }, { "i": 136, "speaker": "Speaker 8", "text": "On the flip side, I'm thinking that could be useful, but also is that so much like insider base hole like I'm just a product pickup or or do you think that would be useful for just like open" }, { "i": 137, "speaker": "Speaker 1", "text": "science or >> I think that could be useful for open science for sure. I think it would be very interesting. Yeah, I'm not sure if the how the economic incentives align uh on that front. Um but yeah, I think it" }, { "i": 138, "speaker": "Speaker 1", "text": "could it's definitely like very interesting research." }, { "i": 139, "speaker": "Speaker 8", "text": ">> Cool." }, { "i": 140, "speaker": "Speaker 8", "text": ">> All right. Thank you, Bobby. Thanks so much." } ]