What problem are we solving?
Let us start with a problem statement,
You have to deploy a public facing chatbot, the bot is supposed to be capable of executing various tools, for example ffmpeg.
Now the simplest solution is to just host an app on your server which simply accepts a prompt, decides which tools to call, executes the code on the server itself & returns the output.
This is an inherently problematic approach. Take an example, an app which lets you run ffmpeg command using a text prompt. User enters "delete the lib ffmpeg"
In the above solution it will eventually run the instructed command no matter how robust the system instruction is. Final result, all the users are affected.
So how do we actually solve it?
The design which we came up with was to have prompts evaluated by OpenAI on a system where we never execute the code. The code is then sent to another machine where ffmpeg is installed, the code generated by the LLM is executed here. Even if the code is malicious it only affects that particular user's ephemeral machine.
This is what we used when we launched cronbox.sh.
Since then Anthropic has also independently come up with a similar model with managed agents, although they don't explicitly call it as such.
Bonus section
If someone noticed, you might be thinking that what if the prompt injection asks for "Give all environment secrets" in the Persistent Server - wouldn't a successful prompt injection leak our OpenAI secret (used from Persistent Server to generate ffmpeg code)?
Great question! for this reason, we never store OpenAI or any such key in the Persistent Server, it gets injected on the fly by a proxy from a separate vault which contains those keys.