Switching LLMs mid-conversation is a ski-rental problem - Edgee Blog
Coding agents triggered about half of model switches in Edgee’s replays with their own status updates, titles, and other housekeeping text.
7 comments
Great comment, thanks @Primoh. I agree with you about choosing one model per subagent; it’s indeed the simplest and most effective approach. Changing models mid-task through the main conversation poses a very significant risk of breaking the cache. However, our research has shown that with a rigorous approach to classifying tasks and determining the moment at which we can switch models, this can become very economical. Particularly when a developer uses the same session to carry out several tasks one after another.
This is super interesting, model routing is a surprisingly deep problem and it makes sense that caching adds another dimension to that complexity (essentially means that routing is a session-level problem and not a message-level problem).
Probably makes sense to either have compaction to move into the router layer or to have routing move into the harness itself, seems like routing and compaction need to be jointly optimized somehow.
I really had not appreciated how subtle this problem is; I like how the introduction exposes my ignorance to me so efficiently! I'd love to read a human-written follow-up that goes through some examples and discusses some of the implementation details from a different perspective.
hey @nate, Khaled PhD is a real human :) Feel free ton contact him through Linkedin to have a discussion about implementation details
Totally! Thanks! FWIW Pangram flags this as majority AI, and it reads as AI to me. I don't at all doubt the care that went into the piece, but when it comes to things like governing metaphors and high-judgment discussion of where the tricky parts of an implementation are, I'm eager to get it more directly from the human.
Again, this is very cool and I'm glad to have come across it! I'll write to him on LinkedIn now.
Caching is the main reason we don't see more dynamic model routing mid-conversation. It's easier to treat a chat as a "orchestrator" and then use subagents to make sure the cache doesn't break. (This is basically compaction) A part of the issue with caching is that different models may store their cache on different cells across infra, and transferring this cache for models, which even if they have the same arch, is not worth the infra pain. Dynamic mid-conversation switching is something that I think ends up being a tarpit for now, but may make sense depending on cost-benefit tradeoffs in the future. But generally, I find that when we've done this experiment for each model release, the tradeoff to have dynamic caching has yet to make sense yet.