according to the published report, in a blog post that changing the agent harness while keeping the underlying model constant significantly altered task completion rates.reported
Evidence: reported
Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?aifleet-huggingface
The claim
In a controlled experiment using GLM-5.3-flash on 90 AppWorld tasks, CUGA achieved higher success and consistency.reported The post argued that enterprise work requires harnesses that enforce rules within private boundaries, which CUGA provides through its configurable design and deployment options.
Evidence: reported
Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?aifleet-huggingface
The response
The authors clarified that this is not a verdict on Hermes or Deep Agents. They noted Hermes is built for personal assistants and Deep Agents for long research tasks. Neither competitor was designed for multi-app API work like AppWorld, which explains the performance gap in this specific context rather than general capability.
The practical upshot
IBM Research concluded that the right question is which harness fits the workload, not which is best overall. CUGA’s loop includes verification and reflection features that catch mistakes before execution. These steps act as enforcement points, allowing the harness to own more of the process and reduce model decision errors.
Claims versus evidence
The source presents evidence that harness choice drives outcomes when the model is fixed. However, it acknowledges three limits: the sample is only 90 tasks on one benchmark, setups were not identical, and policy enforcement was not tested. The 81% figure reflects CUGA’s performance on this specific set, not a universal baseline for the model.reported That suggests results may not generalize to other tasks.
Evidence: reported
Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?aifleet-huggingface
Why it matters
- Product teams can see that framework selection directly impacts automation success rates, independent of the AI model used.
- Enterprises prioritizing rule enforcement may prefer harnesses with built-in verification steps, as demonstrated by CUGA’s architecture.
- The study underscores that benchmarks like AppWorld serve as proxies for enterprise capabilities, not definitive measures of all agent uses.
Topics: agent harnesses, IBM Research, AppWorld, CUGA, AI benchmarking
Updated
The strongest doubt
Counterpoint from the sources
The authors state that setups were not identical and the sample size is small, meaning results may not generalize to other tasks or benchmarks.
For readers
Developers
- Review the harbor-eval fork to replicate the setup and understand the specific configurations used for each harness.
- Consider how verification and reflection steps in a harness can reduce model errors in your automation pipelines.
Business
- Evaluate whether your enterprise tasks require rule enforcement and private deployment, features highlighted by CUGA’s design.
- Recognize that agent success depends heavily on harness choice, not just the underlying AI model selected.
Researchers
- Analyze the limits of the 90-task sample and non-identical setups when interpreting the performance gaps between harnesses.
Ask the article
Local keyword search of this article; no generated answers.Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Questions readers ask
Which model was used in the comparison?
The same open-weight model, GLM-5.3-flash, was used for every harness run to isolate the impact of the framework.
Are there limitations to these results?
Yes, the study used only 90 tasks, had non-identical setups, and did not test policy enforcement, limiting generalizability.
Reading receipt
- The same GLM-5.3-flash model powered all runs, isolating the harness as the variable driving performance differences.
- CUGA achieved consistent scenario passes in 20 cases, compared to eight for Hermes and two for Deep Agents.
Still open: How do these results translate to benchmarks that include policy enforcement testing?
Still open: Will similar performance gaps appear when testing different open-weight models with the same harnesses?