Salesforce AI Research
MCP-Universe RL builds the missing environment- and rollout-orchestration layers once, then reuses them across domains through a shared Model Context Protocol interface.
One framework configuration · Only the task specification changes across domains
General RL backends optimize the policy, but users still have to solve two recurring systems problems for every new tool-use domain.
Hundreds of concurrent trajectories need isolated, reachable environments because shared state can collide and corrupt the reward. Provisioning, health checking, resetting, and recycling those environments is a reusable systems problem.
Long, multi-turn episodes alternate between GPU generation and slow tool calls. Too little overlap leaves the GPU idle; unconstrained concurrency can exhaust host memory with live environments.
Domains differ in their tools and evaluators, while the framework passes data—not domain logic—through the same orchestration and training path.
Provisions, isolates, checks, resets, and recycles MCP environments over a pluggable container backend, with each environment reached through its own MCP gateway.
Runs overlapping episodes, records trainable tokens, and invokes the task evaluator so finished interactions become scored trajectories.
Adapts backend-neutral batches to an existing RL backend. The current integrations support veRL and slime, while the backend and algorithm remain configurable.
The rollout engine separates stages by resource, then sizes their worker pools independently to keep generation busy without increasing the live-environment budget.
Provision or reuse an isolated environment and run any per-instance setup.
Drive the agent over MCP while other trajectories wait on their tool calls.
Score the finished episode against the live environment, then release it.
Run workers are set to at least twice the Acquire workers by default because acquisition is host-resource bound while generation is GPU bound.
All runs train the open-weight gpt-oss-20b policy with GRPO. The framework configuration stays fixed while each domain supplies its own task specification, tools, and evaluator.
R2E-Gym
The agent uses one MCP shell tool to inspect and edit a real repository. Hidden tests run in the same isolated container to produce a binary reward.
DeepDive
A ReAct agent searches and reads the open web through Google Search and Jina Scrape MCP servers, then an evaluator scores its final answer against the reference.
AgentWorldModel-1k
The agent acts across code- and database-backed applications over MCP, while deterministic SQL or code verifiers inspect the resulting application state.
These experiments test framework reuse across substantially different environments and reward checks; they are not claims of state-of-the-art benchmark performance.
On the software-engineering workload, decoupling Run concurrency from Acquire concurrency raises rollout throughput from 147 to 410 tokens per second while the Acquire stage and live-environment count stay fixed.
At matched concurrency, fully asynchronous GPU placement improves training throughput from 3.1 to 6.3 steps per hour and reduces time per step from 1,164 to 572 seconds—approximately 2× faster than colocated placement. It is one-step off-policy, using trajectories at most one step stale.
MCP-U RL is released as open source under the Apache-2.0 license. The release includes the three reusable layers and runnable configurations for software engineering, deep research, and general tool use.
View the repository