It's been clear that LLMs alone are not as powerful as "LLMs with tools".
How does LLM call tools?
It can generate a signature of the tool call, somewhat like a function call search_web("query") or bash("ls -l") etc. This usually calls a function written to either call the google API, exa, or execute a bash command locally.
This is kind of fine as long as tool calls are properly formatted — some slightly less powerful models often make mistakes in generating those signatures. The training was so tedious that lot of models were released without the "tool calling" ability.
We had to include training data involving special tokens, for example - |tool_call|.
How to fix it?
We learned that the same less powerful models are still very good at generating code, since they are already trained on them. If we ask the LLM to generate say a python/typescript code which eventually calls the search API, bash etc then that has higher chances of success - especially when multiple tool calls are involved.
One Issue
Now we will need to execute code all the time, with this comes the risk of executing malformed code (still happens with lower probibility) — to execute each code we spin-up an ephemeral linux machine & execute the code there.
There is not much overhead as most of the ephemeral sandbox providers spin-up under 200ms (including us) & you can just assume it will not be a noticeable delay.
How is it used so far?
Anthropic uses it to reduce token spends by a lot. See Programmatic Tool Calling. The code mode concept was probably introduced by Cloudfare first when models weren't as good in writing tool calls. Today, both arguments make sense - double benefit.
How does it relate to smaller models
Microsoft recently said to use specially trained models for specific tasks — now they haven't said anything about code mode but my hunch is that code mode will certainly help if you don't want additional training to make the model learn tool calling. Along with helping you reduce the token count like always.