Repo InvestigatorFindings
Six models, three repositories, five investigative questions. Here is what the study found.
The curated-tools variant produced better answers than the open-sandbox variant for every one of the six models tested, though the size of the gap varied a lot by model. Giving an agent a small number of purpose-built tools that return clean, pre-computed results appears to outperform giving it open access to write and run its own code against the same data.
By model
| Model | Quality (A) | Quality (B) | Cost ratio (B / A) |
|---|---|---|---|
| deepseek/deepseek-v4-flash | 7.00 | 6.20 | 4.2× |
| deepseek/deepseek-v3.2 | 7.20 | 6.60 | 3.1× |
| google/gemini-2.5-flash-lite | 4.60 | 3.20 | 1.8× |
| google/gemini-2.5-pro | 6.60 | 5.40 | 1.2× |
| anthropic/claude-haiku-4.5 | 5.80 | 5.60 | 4.1× |
| openai/gpt-5-mini | 8.00 | 6.00 | 1.2× |
Cost ratio is how many times more tool-call cost the open-sandbox variant used relative to the curated-tools variant, for the same question and model.