Skip to main content
Simon Mo

Simon Mo

Aug 6, 2026

16:24
So in those cases, kind of what is the point of running them, and kind of what do we learn architecturally in the, in the course of running them?
16:33
Yeah.
16:33
So first, on cost, it's not necessarily they are as expensive as a proprietary model, but rather, first, the cost, cost discourse has been discussed over and over again with even Geo and five point two a few months back.
16:47
So open weight model are sometimes definitely a lot cheaper.
16:52
And but for this model, there's a big sort of s-step change where we're bridging almost a ten x gap, but strike somewhere in the middle, where Kimi K three is not as expensive as Claude or GPD Sol, but it is a lot more expensive than Geo and five point two.
17:10
Why is that? And I do believe this is the point of we're pricing intelligence with the market correctly and understanding where it is.
17:17
And but then the majority part of the discourse should be focusing on, wow, this model is bringing a Opus four point eight level model to our own infrastructure that I can use, I can run, I can fine-tune, I can be able to understand exactly how many tokens do I need, understand the exact performance profile.

21 MINS LATER

38:24
[laughs]
16:24
So in those cases, kind of what is the point of running them and kind of what do we learn architecturally in the, in the course of running them?
16:33
Yeah.
16:33
So first on cost, it's not necessarily they are as expensive as a proprietary model, but rather, first the cost, cost discourse has been discussed over and over again with even Geo 5.2 a few months back.
16:47
So open weight model are sometimes definitely a lot cheaper.
16:52
And but for this model, there's a big sort of s-step change where we're bridging almost a ten x gap but strike somewhere in the middle, where Kimi 3K is not as expensive as Claude or GPD Sol, but it is a lot more expensive than Geo 5.2. Why is that? And I do believe this is the point of we're pricing intelligence with the market correctly and understanding where it is and...
17:19
But then the majority part of the discourse should be focusing on, wow, this model is bringing a Opus 4.8 level model to our own infrastructure that I can use, I can run, I can fine-tune, I can be able to understand exactly how many tokens do I need, understand the exact performance profile.
17:37
The reason here, for example, is for proprietary models, there is a regular mode and fast mode.

21 MINS LATER

38:24
[laughs]
28:40
Do you think this means more, there will be more state managed in the inference layer over time?
28:45
As before, the paradigm has been texting, text out, and then just single request response.
28:53
But as we evolve into the year and the decade of agents, we're seeing multi-turn conversation turning into... hundreds and thousands of terms.
29:02
And then these terms also involves external tool use, like interacting with sandbox, performing web searches, running Python script or any programming languages, and be able to have this kind of long iterative process where LLM is involved, but also external environment interaction is involved.
29:22
And this really kicked off a huge wave of co-optimizing a Genentech architecture with inference architecture.
29:31
So just to give an example, it is very important for VLM to understand whether or not the conversation is still happening.
29:42
If the conversation is no longer happening, we can remove the KV cache.

10 MINS LATER

39:47
So yeah, that's some of our focus at the moment.

We value your privacy

We use cookies to understand how you use our platform and to improve your experience. Click “Accept All” to consent, or “Decline non-essential” to opt out of non-essential cookies. Read our Privacy Policy.