AI search agents fail at asking, not searching

Key Takeaways

• DiscoBench benchmark reveals AI search agents fail because they don't ask clarifying questions when queries are ambiguous
• Best-performing model (Doubao Seed 2.0 Pro) achieved only 43.1% end-to-end accuracy on ambiguous multi-step queries
• Repeated searching without clarification performs worse than simply guessing, with a 51.9% vs 56.5% success rate

AI search agents don't break down when retrieving information. They break down when they encounter an ambiguous query and guess instead of asking for clarification. A new benchmark called DiscoBench, developed by researchers at Tencent Hunyuan and Tsinghua University, quantifies this failure mode across eleven leading language models. The best performer, Doubao Seed 2.0 Pro, managed only 43.1% end-to-end accuracy on tasks containing ambiguous checkpoints.

The finding matters because every major AI lab is racing to ship autonomous search agents. Google, OpenAI, Perplexity, and Tencent all have products that promise to handle multi-step research tasks. But when a user asks something vague, like 'find information about Apple,' the agent has to decide: is this about the company, the fruit, or Apple Records? Current models overwhelmingly choose to guess.

Why one bad guess ruins the entire chain

Previous benchmarks like GAIA and BrowseComp assumed user queries arrive complete and unambiguous. Real queries don't. They're vague, incomplete, or sometimes factually wrong. In a long reasoning chain, an unresolved ambiguity at an early node propagates through every subsequent step. The agent keeps searching with perfect syntax while chasing the wrong target entirely.

DiscoBench formalizes this problem with 211 tasks containing 463 ambiguous points across eleven knowledge domains: video games, sports, music, film, science, politics, and more. Each task splits into multiple checkpoints. At each checkpoint, the agent chooses one of three actions: keep searching, ask the user for clarification, or deliver an answer.

Four ways queries go wrong

The researchers categorize ambiguity into four types. A description might match multiple entities. It might apply to different time periods or software versions. It could allow for multiple valid ranking criteria. Or it might contain an outright factual error that the agent should catch and question.

The dataset is primarily written in Chinese to reflect typical search patterns on the Chinese-language web. When an agent asks a useful follow-up question, an LLM-based user simulator releases a predefined clue that narrows the search. All queries run through the agent search engine Tavily, with Gemini 3 Flash serving as the simulator.

No model breaks 50% accuracy

The team tested eleven models released in the past six months: Claude Opus 4.7, GPT 5.4, Gemini 3.1 Pro Preview, Doubao Seed 2.0 Pro, DeepSeek V4 Pro, Kimi K2.6, GLM 5.1, Qwen3.6 Max, MiniMax M2.7, MiMo v2.5 Pro, and Hunyuan 3.0 Preview.

Without an explicit hint about possible ambiguity, Doubao Seed 2.0 Pro hit the top end-to-end accuracy at 43.1%. Gemini 3.1 Pro followed at 40.8%, Claude Opus 4.7 at 39.8%. Weaker models like MiniMax M2.7 and Qwen3.6 Max managed only 16.1% and 12.3%, respectively.

There's a telling gap between individual step performance and overall results. Claude Opus 4.7 solves 57% of checkpoints correctly but reaches only 39.8% end-to-end accuracy. The individual research steps work fine. A single unresolved ambiguity is enough to collapse the entire chain.

A warning prompt doesn't fix the problem

The researchers also tested a 'Guided' mode where the system prompt explicitly tells the agent to watch for ambiguity and ask follow-up questions when in doubt. This was meant to show the ceiling reachable when models don't have to detect underspecification on their own.

Averaged across ten models, end-to-end accuracy rose from 28.6% to 33.7%. Detection F1 jumped more sharply, from 45.3% to 64.9%. The hint helped models spot ambiguity without actually helping them complete research successfully. For Claude Opus 4.7, end-to-end accuracy even dipped slightly under the guided prompt despite a higher checkpoint pass rate.

Searching more is worse than guessing

The behavioral profile analysis reveals what agents actually do at ambiguous checkpoints. Models that search first and then ask a follow-up ('SearchThenAsk') average a 93.4% success rate at those checkpoints. Guessing without asking ('DirectGuess') drops to 56.5%. Models that search repeatedly but still guess instead of asking ('SearchHeavyGuess') perform worst at 51.9%.

That last pattern is particularly damning. The repeated searches suggest the model already detected something was off but never converted that uncertainty into a user interaction. The agent burns API calls confirming its confusion, then guesses anyway.

What this means for agent builders

For teams building AI search products, DiscoBench exposes a design gap. Current architectures optimize for retrieval quality and reasoning chain coherence. They don't reward the agent for admitting uncertainty or asking questions. The result is overconfident behavior that compounds errors rather than resolving them.

The 93.4% success rate for SearchThenAsk behavior suggests the fix isn't a better retriever or a larger model. It's training agents to recognize when they should hand control back to the user. That's a different optimization target than most current systems employ.

Frequently Asked Questions

What is DiscoBench?

DiscoBench is a benchmark created by Tencent Hunyuan and Tsinghua University researchers to test whether AI search agents can detect ambiguous queries, ask clarifying questions, and correct their research path. It contains 211 tasks with 463 ambiguous checkpoints.

Which AI model performed best on DiscoBench?

Doubao Seed 2.0 Pro achieved the highest end-to-end accuracy at 43.1%, followed by Gemini 3.1 Pro at 40.8% and Claude Opus 4.7 at 39.8%.

Why do AI search agents fail on ambiguous queries?

AI search agents typically guess rather than asking users for clarification when queries are ambiguous. This error propagates through multi-step reasoning chains, causing the entire research task to fail even when individual search steps execute correctly.

Does prompting AI agents to ask clarifying questions help?

Explicit prompts to watch for ambiguity improved detection rates significantly (from 45.3% to 64.9% F1) but only raised end-to-end task completion from 28.6% to 33.7%. Detection alone doesn't translate to successful task completion.

Source: The Decoder / Jonathan Kemper