WeAgent-MMSearch: Native Text–Vision Interaction for Multimodal Search Agents

Zongkai Liu¹˒² Hui Zhang¹ Liqiang Niu¹^,‡ Zhen Cao¹ Han Li¹ Juntao Liu¹ Wenchao Chen¹ Chengduo Zhao¹ Chao Yu² Fandong Meng¹^,†

¹ Weixin AI, Tencent ² Sun Yat-sen University

‡ Project Lead † Corresponding author

Report

HuggingFace

Benchmark performance of WeAgent-MMSearch. The RL model reaches a 55.97% average across seven public benchmarks and VisTarget-Bench.

Abstract

Multimodal search agents extend parametric knowledge with newly emerging and long-tail evidence from the open web. Yet many existing agentic search environments often expose retrieved evidence only as text and omit tool-returned images from subsequent context, reducing visually grounded trajectories to text-only reasoning. Long-horizon interaction also compounds tool-call, response-length, timeout, and budget failures, which can discard salvageable trajectories, waste rollout computation, and disturb policy updates. To address these issues, we introduce WeAgent-Harness, a multimodal agentic harness that supports native text–vision interaction and runtime recovery. Retrieved images receive persistent disk references, allowing the model to inspect, process, and cite them throughout the trajectory. Based on this harness, we develop WeAgent-MMSearch, an integrated system spanning data construction, agentic post-training, and multimodal rollout. For data construction, a strong MLLM uses WeAgent-Harness to discover, synthesize, and verify MMSearch-style tasks and collect expert trajectories. During post-training, our Failure-Aware GSPO recovers salvageable abnormal rollouts and filters invalid ones to improve bounded multimodal planning and search. We also introduce VisTarget-Bench, a 150-task human-verified benchmark that pairs each question with a held-out target image, distinguishing image-retrieval failures from visual-perception failures. Evaluation on VisTarget-Bench and seven public benchmarks shows that agentic post-training improves the average score by 19.22 points, enabling our model to outperform similarly sized open-source models and rival models with roughly ten times its parameter count.

Comparison of direct answering, text-only search, and WeAgent multimodal search — A time-sensitive multi-hop example. Only the native text–vision trajectory preserves the retrieved visual evidence needed for the correct answer.

WeAgent-Harness

The asynchronous RL infrastructure and the shared state, action, and text–image observation cycle managed by WeAgent-Harness.

The harness unifies web search, page extraction, image search, reverse-image search, and code execution under a single text–image state contract. Retrieved images receive persistent disk references and remain available across turns, while cache-backed execution, structured recovery, and explicit budgets keep long-horizon trajectories finite, reproducible, and suitable for asynchronous RL.

Data Construction

Harness-native task construction and trajectory sampling. The final mixtures contain 59.5K SFT trajectories and 13.3K RL prompts.

A four-stage pipeline discovers visually grounded seeds, expands them backward into multi-hop question chains, verifies image dependence and source provenance, and samples expert trajectories with the same harness used for training and inference. The resulting mixtures provide 59.5K SFT trajectories and 13.3K RL prompts that emphasize executable search, visual inspection, and grounded synthesis.

Benchmark

VisTarget-Bench composition and representative tasks.

Composition and representative examples from VisTarget-Bench

VisTarget-Bench contains 150 human-verified tasks, each paired with a held-out target image. By separately checking whether the agent retrieves the target and whether it reads the relevant pixels correctly, the benchmark disentangles image-retrieval failures from visual-perception failures across visual categories, time-sensitive questions, and different reasoning depths.

Quantitative results

Main comparison across seven public benchmarks and VisTarget-Bench.

Main quantitative comparison across MMSearch-style tasks and multimodal knowledge question answering benchmarks

We compare direct answering, frontier models using WeAgent-Harness, lightweight search agents, and the Base, SFT, and RL variants of WeAgent-MMSearch across seven public benchmarks and VisTarget-Bench. Agentic post-training raises the average score by 19.22 points, with the RL model reaching 55.97% and competing with models roughly ten times larger.

Case Studies

Across scientific diagrams, historical scripts, and postage imagery, these trajectories show where text-only search loses the decisive visual clue. WeAgent-MMSearch keeps retrieved pixels in context, connects them to intermediate reasoning, and grounds each final answer in inspectable visual evidence.

Case 1 comparing a text-only search agent with WeAgent-MMSearch on a cytokine-receptor schematic — Case 1 · Membrane color in a Class II cytokine-receptor schematic.

01 / 03