LLM Tool Calling Failures: Why the Model Picks the Wrong Tool

This post is for engineers whose agent has more than a handful of tools and occasionally calls the wrong one. LLM tool calling usually gets debugged at the model: a stronger model, a longer system prompt, a few-shot example. However, a large share of wrong picks are decided before the model reads anything. Either the right tool never reached the context window, or a lookalike sat next to it with a better-matching description. So this post measures that layer directly. It runs 49 tasks, each phrased three ways, through a BM25 tool search over a 57-tool catalog under four naming and description policies. Then it shows which tool came back first, what outranked the right one, and what each fix costs in tokens.

The numbers below came from this environment:

Measured 2026-11-09
Windows 11 Pro 25H2 (build 26200), AMD Ryzen 5 8600G (6C/12T), 16 GB RAM
Dart SDK 3.9.0 (stable)
Tokenizer: o200k_base vocabulary (SHA-256 446a9538...cfb1a2d), the Dart port
  from the context engineering bench
Catalog: 57 tools across 9 services; 49 tasks x 3 phrasings; 10 no-tool requests
Search: BM25 (k1 1.2, b 0.75), top 5, plural-stripping, no other stemming

No model was called. The bench measures which tools a search layer puts in front of the model, which is deterministic. Two consecutive runs produced byte-identical output, and one run takes about a second.

What Is LLM Tool Calling?

LLM tool calling is the mechanism that lets a model request an action from your code. You send tool definitions, each with a name, a description and a JSON schema for its arguments. The model replies with a tool name and arguments, your code runs the call, and the result goes back to the model as the next message.

Picking a tool therefore happens in two places. First, something decides which definitions are in the context at all. With a small catalog that is every tool, and with a large one it is a search step, such as Anthropic’s tool search or OpenAI’s deferred loading. Second, the model picks among whatever arrived. A failure at the first step looks exactly like a model mistake in the logs, because the model did call a tool, just not the one you expected.

The site’s guide to Claude tool use covers the request and response mechanics. This post is about the selection step, and about why it breaks.

Why Does the Model Pick the Wrong Tool?

Here are the failure modes, ordered from the ones this bench measured to the ones it cannot:

1. The right tool was never loaded. Tool search returned five other tools, so the model chose among the wrong five.
2. A lookalike outranked it. Two tools share a name or vocabulary, and the wrong one matched the request better.
3. The name collided and one tool vanished. Two servers exposed get_issue, and a registry keyed by name kept only one.
4. Nothing was needed, but tools arrived anyway. A plain question returned candidate tools, which invites a call.
5. The right tool, the wrong arguments. Selection succeeded, then the model filled a parameter with the wrong kind of value.
6. A bad result steered the next pick. An opaque error made the model retry with a different, wrong tool.

The Bench: 57 Tools, 49 Tasks, Three Phrasings

The catalog models a common agent setup: nine services connected at once. It has 12 GitHub tools, 7 for Jira, 7 for Slack, 6 for Sentry, 4 for Postgres, 5 for Confluence, 6 for a local filesystem, 5 for a calendar and 5 for PagerDuty. The overlap is deliberate and realistic. GitHub, Jira and Sentry all have “issues”, Sentry and the calendar both have “events”, and Slack and Confluence both have a search tool.

Every tool carries three descriptions:

Field	Example for jira_get_issue
Terse	Get an issue.
Specific	Get one Jira issue by key, such as PAY-123, including status and assignee.
Hints, appended	Use when the user gives a ticket key like PAY-123 and wants its status or details.

The terse form is a deliberately weak baseline, close to what quickly written servers ship. The hints add the words users actually say, such as “ticket”, “PR”, “crash” and “ping”, and they name the neighboring tool to use instead.

Three Ways to Phrase the Same Task

Each of the 49 tasks has one correct tool and three phrasings:

Phrasing	Example for jira_search_issues
Literal	Search Jira issues assigned to me in the PAY project
Casual	Which tickets do I have open in PAY?
Held-out	what PAY tickets do I have on my plate?

There is a bias to declare here. The same author wrote the tool hints and the casual phrasings, so the hints may echo the casual wording. To measure that, a separate agent wrote the held-out set. It saw only the literal sentences and never the tool descriptions. Consequently, the held-out column is the honest one, and the gap between it and the casual column shows how much the author’s own phrasing flattered the hints.

Ten more requests need no tool at all, such as “What does HTTP 409 mean?” and “Explain the CAP theorem in two sentences”.

The Four Naming Policies

1. Bare names, terse descriptions, held in a registry keyed by name, so a later server overwrites an earlier one.
2. Prefixed names, terse descriptions: github_get_issue, jira_get_issue, sentry_get_issue.
3. Prefixed names, specific descriptions.
4. Prefixed names, specific descriptions plus hints.

The search indexes the tool name, the description, and every argument name and argument description. That matches the Anthropic tool search documentation, which states that “both tool search variants (regex and bm25) search tool names, descriptions, argument names, and argument descriptions.” It returns up to five tools, which is also that tool’s documented default. Anthropic’s server-side BM25 implementation is not public, so this bench uses its own, and the two will not rank identically.

Here is the scoring loop:

// bench.dart (excerpt): one probe, scored against the five tools search returned.
final hits = idx.search(phrasing, 5);
if (!registry.values.any((t) => t.id == want)) {
  unreachable++; // a same-named tool from another server replaced it
  continue;
}
final ids = hits.map((h) => h.$1).toList();
if (hits.isEmpty) none++;           // search returned nothing at all
if (ids.contains(want)) top5++;     // the model could have picked it
if (ids.isNotEmpty && ids.first == want) {
  first++;
} else if (ids.isNotEmpty) {
  lookalike++;                      // another tool outranked the right one
}

“In top-5” is the number that matters most. If the right tool is not in the five, no model can call it. “Ranked first” is weaker evidence, because a model reads all five definitions and does not always pick the top hit. Still, it shows how strongly a lookalike competed.

LLM Tool Calling Results: What Reached the Model

Here is the full run:

dart run bench.dart ../vocab/o200k_base.tiktoken

catalog: 57 tools, 9 services; probes: 49 tasks x 3 phrasings, 10 no-tool requests

bare names shared across services:
  search_issues  github, jira  -> registry keeps jira
  get_issue      github, jira, sentry  -> registry keeps sentry
  create_issue   github, jira  -> registry keeps jira
  update_issue   github, jira, sentry  -> registry keeps sentry

policy                        phrasing   in top-5  ranked 1st  lookalike 1st  unreachable  no match | defs tokens  mean top-5 tokens
bare names, terse             literal         45          41              4            4         0 |        2664                258
bare names, terse             casual          12           5             26            4        14 |        2664                132
bare names, terse             held-out        19          13             17            4        15 |        2664                110
prefixed names, terse         literal         49          45              4            0         0 |        3130                272
prefixed names, terse         casual          14           5             30            0        14 |        3130                145
prefixed names, terse         held-out        21          14             20            0        15 |        3130                129
prefixed, specific            literal         49          45              4            0         0 |        3605                316
prefixed, specific            casual          22          13             27            0         9 |        3605                198
prefixed, specific            held-out        28          18             23            0         8 |        3605                177
prefixed, specific + hints    literal         49          44              5            0         0 |        4542                410
prefixed, specific + hints    casual          45          35             12            0         2 |        4542                306
prefixed, specific + hints    held-out        39          28             17            0         4 |        4542                289

prefixed names, terse: wrong tool ranked first, same service 28, other service 26; most frequent:
  3x  jira_transition_issue <- jira_get_issue
  2x  calendar_list_events <- calendar_find_free_time
  2x  fs_move_file <- fs_search_files
  2x  jira_add_comment <- jira_get_issue
  2x  jira_search_issues <- jira_get_issue
  2x  pagerduty_acknowledge_incident <- confluence_get_page

prefixed, specific + hints: wrong tool ranked first, same service 20, other service 14; most frequent:
  3x  confluence_get_page <- confluence_search
  2x  calendar_create_event <- github_get_pull_request
  2x  calendar_list_events <- calendar_delete_event
  2x  github_add_issue_comment <- github_get_issue
  2x  jira_search_issues <- jira_get_issue
  2x  jira_transition_issue <- jira_get_issue

catalog size, the 11 GitHub tasks x 3 phrasings (33 probes):
servers                              tools  policy                      in top-5  ranked 1st
github                                  12  prefixed names, terse             21          15
github                                  12  prefixed, specific + hints        30          24
github, jira, sentry                    25  prefixed names, terse             20          14
github, jira, sentry                    25  prefixed, specific + hints        30          24
all 9                                   57  prefixed names, terse             20          14
all 9                                   57  prefixed, specific + hints        30          25

no-tool requests (10): how many still got candidate tools back
  bare names, terse             7 of 10 returned tools, 2.3 tools on average
  prefixed names, terse         7 of 10 returned tools, 2.3 tools on average
  prefixed, specific            7 of 10 returned tools, 2.5 tools on average
  prefixed, specific + hints    8 of 10 returned tools, 2.6 tools on average
  "Explain what an idempotency key is" -> postgres_explain_query (5.33), jira_get_issue (3.34), jira_add_comment (2.95)
  "Rewrite this sentence so it sounds less formal" -> slack_search_messages (3.16)
  "Summarize the trade-offs of optimistic locking" -> slack_get_channel_history (3.84)
  "Translate "the deploy is done" into German" -> jira_transition_issue (4.96), sentry_list_releases (3.43), slack_send_message (2.98)
  "How do I reverse a list in Python?" -> confluence_list_spaces (2.39), postgres_list_tables (2.36), fs_list_directory (2.33)
  "Suggest a good name for a retry helper function" -> github_search_code (3.41), slack_add_reaction (1.21), slack_get_user_profile (1.18)
  "Explain the CAP theorem in two sentences" -> postgres_explain_query (5.33)
  "Is it called a pull request or a merge request in GitLab?" -> github_merge_pull_request (13.02), github_create_pull_request (6.71), github_get_pull_request (6.65)

The same results as a comparison table, out of 49 tasks per phrasing:

Policy	Literal in top-5	Casual in top-5	Held-out in top-5	Held-out ranked 1st	Tokens, all 57 defs
Bare names, terse	45	12	19	13	2,664
Prefixed, terse	49	14	21	14	3,130
Prefixed, specific	49	22	28	18	3,605
Prefixed, specific + hints	49	45	39	28	4,542

Seven findings follow. The first and third matter most for anyone debugging wrong tool picks today.

Literal Test Prompts Hid the Whole Problem

With prefixed names, every policy found the right tool for all 49 literal requests, and even terse descriptions ranked it first 45 times. A test suite written in the tool’s own vocabulary, such as “Get Jira issue PAY-123”, passes everything. In contrast, the same terse catalog found the right tool for 14 of 49 casual requests and 21 of 49 held-out ones. For 15 held-out requests, search returned nothing at all, because a request such as “show me api PR 88” shares no indexed word with “Get a pull request.”

This is the most practical finding in the run. Engineers who write tools tend to test them with the tool’s own words. Real users say “ticket”, “PR”, “the crash” and “ack it”, and none of those appear in a description written from the API’s point of view.

Bare Names Made Four Tools Unreachable

Four tool names appeared in more than one service: search_issues, get_issue, create_issue and update_issue. In a registry keyed by name, the last server registered won. As a result, the GitHub versions of search, get and create disappeared, and so did Jira’s get_issue. Four of the 49 tasks could not succeed under any phrasing, and nothing raised an error.

The MCP 2025-11-25 tools specification only says names “SHOULD be unique within a server”. Uniqueness across servers is the client’s problem. Anthropic’s guidance on writing tools for agents recommends namespacing by service, “e.g., asana_search, jira_search“, and notes that “MCP clients sometimes do this by default.” Sometimes is the operative word. Prefixing alone recovered the four tasks, and it also raised literal recall from 45 to 49.

Descriptions in the User’s Words Doubled Held-Out Recall

Moving from terse to specific descriptions raised held-out recall from 21 to 28. Adding hints raised it to 39, and the right tool ranked first 28 times instead of 14. The no-match count fell from 15 to 4. No other change in the bench moved the numbers this much.

The casual column overstates the gain. With hints, casual recall was 45 while held-out was 39, and that gap of six is the author bias described earlier. Hints written by someone who has seen the test phrasings will match those phrasings better. Even so, the held-out improvement from 21 to 39 came from phrasings the hint writer never saw.

The cost was tokens. All 57 definitions grew from 3,130 to 4,542 tokens, about 45 percent more. That matters when every definition is loaded on every call, and much less with tool search, where the mean cost of the five loaded definitions went from 129 to 289 tokens per held-out request.

Most Wrong Picks Were the Right Service, Wrong Action

Under the hints policy, the tool ranked first was wrong 34 times across all phrasings. In 20 of those, the winner came from the correct service but performed the wrong action. The most common pattern was a “get” tool beating the action the user wanted:

[held-out] "move PAY-123 to done" want jira_transition_issue, got jira_get_issue, jira_transition_issue, github_get_issue
[held-out] "put the root cause on PAY-123" want jira_add_comment, got jira_get_issue, jira_add_comment, github_get_issue
[held-out] "drop a note on web #412 that the fix is live" want github_add_issue_comment, got github_get_issue, confluence_update_page, jira_add_comment

The jira_get_issue description contains an example key, “such as PAY-123”, so it matches every request that mentions a key. Admittedly, the bench reused the same key in its probes, which exaggerates the effect, since real traffic would carry other numbers. However, the project prefix “PAY” would still match. An identifier example in a description pulls in every request that contains that kind of identifier, whatever the user wants done with it.

Notably, the right tool was still in the top five in all three cases above. Here the search did its job, and the choice falls to the model, which has to read past the first definition. Anthropic’s advanced tool use post names exactly this as the main failure: “The most common failures are wrong tool selection and incorrect parameters, especially when tools have similar names like notification-send-user vs. notification-send-channel.”

Argument Names Are Searched Too

The strangest misses came from parameters. Slack’s reply_to_thread takes an argument named thread_ts, the timestamp of the parent message. The indexer splits that into “thread” and “ts”, so every request that mentioned a TypeScript file matched it:

[held-out] "mv old/utils.ts to lib/utils.ts" want fs_move_file, got slack_reply_to_thread, fs_search_files
[casual] "What does config.ts look like on main in the api repo?" want github_get_file_contents, got slack_reply_to_thread, github_get_issue, github_get_file_contents

Similarly, “book the design review Friday 2pm” returned github_get_pull_request first, because its description mentions “review state”. Likewise, “post in #deploys that the release shipped” returned sentry_list_releases before slack_send_message. Each of these is one rare word outscoring several common ones, which is how BM25 is supposed to work. The lesson is that every word in a definition is a potential match, including parameter names you never thought of as text.

Adding Servers Barely Moved the Search

This result contradicts the expected conclusion, so here it is plainly. The bench held the 11 GitHub tasks fixed and grew the catalog from 12 tools to 25 and then 57. Recall for those tasks stayed at 30 of 33 with hints and moved from 21 to 20 without them. The rank-first count with hints went from 24 to 25.

So in this bench, a larger catalog did not degrade the search. The well-known size problem lives in the model instead. Anthropic’s tool search documentation states that “Claude’s ability to pick the right tool degrades once you exceed 30–50 available tools”, and OpenAI’s function calling guide suggests that you “aim for fewer than 20 functions available at the start of a turn”, which it calls “a soft suggestion”. Tool search exists to keep the model’s choice small while the catalog grows. This bench supports the first half of that bargain, and it says nothing about the second.

Plain Questions Still Got Tools

Eight of ten no-tool requests returned candidate tools under the hints policy. The worst was “Is it called a pull request or a merge request in GitLab?”, which returned github_merge_pull_request with a score of 13.02, higher than most correct matches in the run. “Explain the CAP theorem in two sentences” returned postgres_explain_query, because both contain “explain”.

A search step returns its best matches, not a verdict on whether any tool applies. Whether the model then calls one depends on the model and on tool_choice. With OpenAI’s "auto", the default, the model can “call zero, one, or multiple functions”. With "required", it must call something, and here that something would be github_merge_pull_request.

How to Fix LLM Tool Calling Failures

Based on these measurements, a practical order of work looks like this:

1. Prefix every tool name with its server, and make registration fail on a duplicate instead of overwriting.
2. Write descriptions in the words users type, such as “ticket”, “PR” and “crash”, and name the neighboring tool to use instead.
3. Test selection with phrasings written by someone who has not read the descriptions.
4. Review argument names and identifier examples as search text, because they match requests too.
5. Keep the few most used tools always loaded, and defer the rest behind tool search.
6. Return errors that name the tool the model probably meant.
7. Leave tool_choice on auto for conversational agents, so a plain question can get a plain answer.

Steps 1, 2 and 3 accounted for every improvement in this bench. Steps 4 to 7 address failures it observed but did not score.

Register Tools Without Silent Overwrites

The first fix is a registry that namespaces by server and refuses to replace an existing tool. It also enforces the MCP name rules: “between 1 and 128 characters”, using only “A-Z, a-z”, digits, underscore, hyphen and dot.

// registry.dart: register tools from several servers without letting one
// silently replace another, and enforce the MCP 2025-11-25 name rules.

final _validName = RegExp(r'^[A-Za-z0-9_.\-]{1,128}$');

class ToolDef {
  final String server, name, description;
  const ToolDef(this.server, this.name, this.description);
}

class ToolRegistry {
  final _tools = <String, ToolDef>{};

  /// Registers [tool] under `server_name` and returns the exposed name.
  String register(ToolDef tool) {
    final exposed = '${tool.server}_${tool.name}';
    if (!_validName.hasMatch(exposed)) {
      throw ArgumentError.value(exposed, 'name', 'must be 1-128 chars of A-Z a-z 0-9 _ - .');
    }
    final existing = _tools[exposed];
    if (existing != null) {
      // Failing loudly here is the point: a silent overwrite removes a tool
      // the model can still be asked to use, and nothing errors downstream.
      throw StateError('$exposed is already registered by ${existing.server}');
    }
    _tools[exposed] = tool;
    return exposed;
  }

  Iterable<String> get names => _tools.keys;
}

void main() {
  final registry = ToolRegistry();
  const incoming = [
    ToolDef('github', 'get_issue', 'Get one GitHub issue by repository and number.'),
    ToolDef('jira', 'get_issue', 'Get one Jira issue by key, such as PAY-123.'),
    ToolDef('sentry', 'get_issue', 'Get one Sentry issue by ID.'),
    ToolDef('jira', 'get_issue', 'A second Jira server exposing the same tool.'),
    ToolDef('wiki', 'search pages', 'A name with a space in it.'),
  ];
  for (final tool in incoming) {
    try {
      print('registered ${registry.register(tool)}');
    } on Error catch (e) {
      print('rejected   ${tool.server}/${tool.name}: $e');
    }
  }
  print('exposed: ${registry.names.join(', ')}');
}

dart run registry.dart

registered github_get_issue
registered jira_get_issue
registered sentry_get_issue
rejected   jira/get_issue: Bad state: jira_get_issue is already registered by jira
rejected   wiki/search pages: Invalid argument (name): must be 1-128 chars of A-Z a-z 0-9 _ - .: "wiki_search pages"
exposed: github_get_issue, jira_get_issue, sentry_get_issue

Three get_issue tools now coexist, and a second Jira server fails at startup instead of quietly replacing the first. A startup failure is cheap to fix. By comparison, a missing tool surfaces weeks later as “the agent keeps looking up Sentry when I ask about Jira”.

Return Errors That Name the Right Tool

When the model does pick a lookalike, the tool result is your second chance. Anthropic’s tool-writing guide suggests that error responses “clearly communicate specific and actionable improvements, rather than opaque error codes or tracebacks.” For a misrouted call, the most actionable improvement is the name of the right tool:

// misroute.dart: when the model picks the wrong tool, say which tool it meant.
import 'dart:convert';

final _jiraKey = RegExp(r'^[A-Z][A-Z0-9]+-\d+$'); // PAY-123
final _sentryId = RegExp(r'^[A-Z]+-[0-9A-Z]+$'); // CHECKOUT-4F2
final _issueNumber = RegExp(r'^#?\d+$'); // 412 or #412

/// Validates input for github_get_issue. Returns null when valid, otherwise a
/// tool_result body that tells the model what to do next.
Map<String, Object>? checkGithubGetIssue(Map<String, Object?> input) {
  final number = '${input['number'] ?? ''}'.trim();
  if (_issueNumber.hasMatch(number)) return null;

  String? hint;
  if (_jiraKey.hasMatch(number)) {
    hint = '"$number" is a Jira issue key. Call jira_get_issue with issue_key "$number".';
  } else if (_sentryId.hasMatch(number)) {
    hint = '"$number" looks like a Sentry issue ID. Call sentry_get_issue with issue_id "$number".';
  }
  return {
    'is_error': true,
    'content': hint ?? 'number must be a GitHub issue number such as 412, got "$number".',
  };
}

void main() {
  for (final input in [
    {'repo': 'acme/web', 'number': '#412'},
    {'repo': 'acme/web', 'number': 'PAY-123'},
    {'repo': 'acme/checkout', 'number': 'CHECKOUT-4F2'},
    {'repo': 'acme/web', 'number': 'latest'},
  ]) {
    print('${input['number']}: ${jsonEncode(checkGithubGetIssue(input) ?? 'ok')}');
  }
}

dart run misroute.dart

#412: "ok"
PAY-123: {"is_error":true,"content":"\"PAY-123\" is a Jira issue key. Call jira_get_issue with issue_key \"PAY-123\"."}
CHECKOUT-4F2: {"is_error":true,"content":"\"CHECKOUT-4F2\" looks like a Sentry issue ID. Call sentry_get_issue with issue_id \"CHECKOUT-4F2\"."}
latest: {"is_error":true,"content":"number must be a GitHub issue number such as 412, got \"latest\"."}

The order of the checks matters, because “PAY-123” matches both patterns. Jira is checked first since its keys are digits only after the dash. The error never guesses silently, either. It returns is_error and lets the model make the corrected call itself, so the correction appears in the transcript where you can audit it.

Defer the Long Tail with Tool Search

Anthropic’s tool search sends every definition in the request but loads only the ones a search returns. You mark the long tail with defer_loading: true and add the search tool itself, which must never be deferred:

{
  "model": "claude-opus-5-5",
  "max_tokens": 2048,
  "tools": [
    { "type": "tool_search_tool_bm25_20251119", "name": "tool_search_tool_bm25" },
    {
      "name": "github_get_issue",
      "description": "Get one GitHub issue by repository and issue number, such as #412.",
      "input_schema": {
        "type": "object",
        "properties": {
          "repo": { "type": "string", "description": "Repository in owner/name form" },
          "number": { "type": "integer", "description": "Issue number" }
        },
        "required": ["repo", "number"]
      }
    },
    {
      "name": "jira_transition_issue",
      "description": "Move a Jira issue through its workflow, for example to In Progress or Done. Use to move a ticket to done, close it, or change its status.",
      "input_schema": {
        "type": "object",
        "properties": {
          "issue_key": { "type": "string", "description": "Issue key, for example PAY-123" },
          "transition": { "type": "string", "description": "Target status name" }
        },
        "required": ["issue_key", "transition"]
      },
      "defer_loading": true
    }
  ],
  "messages": [{ "role": "user", "content": "move PAY-123 to done" }]
}

The documentation advises keeping “your 3–5 most frequently used tools non-deferred so Claude can call them without searching first”, which is why github_get_issue above has no defer_loading. For tools that come through the MCP connector, you set deferral once per server on the mcp_toolset entry’s default_config, with per-tool exceptions in configs. If BM25 misses too many reworded requests in your own tests, the same docs describe returning tool_reference blocks from a custom search tool, for example one backed by embeddings. The site’s comparison of OpenAI, Voyage and Cohere embeddings is a starting point for that choice.

Notice the integer type on number in the first tool. A schema that only accepts a number would reject “PAY-123” before your handler even runs. OpenAI’s function calling guide makes the same point as a design rule, recommending enums and object structure so that invalid argument states cannot be represented at all.

Real-World Scenario: The Agent That Kept Reading Instead of Acting

Consider a small platform team that connects an internal assistant to GitHub, Jira, Sentry and Slack over a couple of weeks. Every server works in isolation, and the demo goes well, because the demo prompts read like the tool names: “get issue PAY-88”, “list open pull requests”. Then the team rolls it out, and the complaints are oddly specific. When someone asks the assistant to “move PAY-88 to done”, it replies with the ticket’s details instead. Similarly, a request to “drop a note on #412” summarizes the issue and stops.

The model is not ignoring the request. In fact, the transcripts show it calling get_issue every time, which is exactly the pattern the bench found. The get tool’s description contains the identifier format, so it outranks the action tool for any request that mentions a key. In addition, the team’s evaluation set was written by the engineers who wrote the tools, so it phrased everything the way the descriptions do.

The fix that works in this situation is cheap but takes discipline. First, the team rewrites the action tools’ descriptions with the verbs people use, such as “move”, “close”, “note” and “drop a comment”. Next, they remove the example key from the get tool’s description, since the schema already documents the format. Finally, they collect a few dozen real requests from the rollout channel and use them as the regression set. The trade-off is that descriptions get longer, and every tool still loaded on every call costs more tokens.

When to Use Tool Search for LLM Tool Calling

• Your agent connects to several servers and the full set passes 30 to 50 tools, the range where Anthropic documents selection degrading
• Tool definitions cost thousands of tokens per call, and most calls use two or three tools
• The catalog grows as teams add servers, and nobody owns the whole list
• You can afford a regression set of real user phrasings to test the search against
• Some tools are rarely used but must stay reachable, such as admin or incident tools

When NOT to Use Tool Search

• The agent has fewer than about ten tools, where loading everything is cheaper and removes a failure point
• Every task needs the same few tools, so there is nothing to defer
• Your descriptions are still terse, because search over “Get an issue.” found the right tool for only 21 of 49 held-out requests here
• Latency matters more than context, since a search step adds a model turn before the real call
• You cannot test with real phrasings, because literal tests will report near-perfect recall regardless

Common Mistakes with LLM Tool Calling

• Testing tool selection only with prompts that reuse the tool’s own words, which scored 49 of 49 here and hid every failure
• Exposing bare names from several servers, which made 4 of 49 tasks unreachable without a single error
• Writing descriptions from the API’s point of view instead of the user’s, with no “ticket”, “PR” or “crash” anywhere
• Putting identifier examples in a read tool’s description, which then outranks every action on that identifier
• Naming arguments without considering them as search text, as with thread_ts matching every .ts file
• Forcing tool_choice to required in a conversational agent, so “pull request or merge request?” triggers a merge tool
• Returning opaque errors from a misrouted call instead of naming the tool the model probably meant
• Blaming the model before checking whether the right tool was in the context at all

For the agent loop that these tools plug into, see the guide to building AI agents with tools, planning and execution. For how servers expose tools in the first place, see the MCP protocol explained.

What This Measurement Does Not Cover

No model ran in this bench. It measures which tools a BM25 search puts in front of the model, not which one the model calls. “Ranked first” is a proxy for how strongly a lookalike competed, not a prediction of the model’s choice. Anthropic’s own reported numbers cover that half: in its MCP evaluations, “Opus 4 improved from 49% to 74%, and Opus 4.5 improved from 79.5% to 88.1% with Tool Search Tool enabled.” Those are vendor figures, not measured here.

The search is also this bench’s own BM25, not Anthropic’s server-side implementation, and the regex variant, where the model writes the pattern, was not measured. Dense retrieval was not measured either. The catalog and descriptions were written for the bench, and the author bias in the hints is visible as the gap between casual and held-out recall. Argument filling, failure mode 5 above, and multi-step recovery, failure mode 6, were demonstrated with code but not scored. Token counts use one tokenizer, and Claude’s own tokenizer will count differently. The bench reruns in about a second, and it will be re-run on the next major change to the tool search features it cites, or at 12 months.

Conclusion

LLM tool calling failures often start before the model chooses. In this bench, tests written in the tools’ own vocabulary found the right tool every time, while real phrasings with terse descriptions found it for fewer than half the tasks. Prefixed names recovered four tools that a name collision had silently removed. Descriptions written in the user’s words raised held-out recall from 21 to 39 of 49, at about 45 percent more definition tokens.

Start today by collecting twenty real requests from your users and running them through whatever selects your tools, before touching the model or the prompt. Then check which tool came back first and what outranked the right one. Next, read the site’s guide to context engineering to see what those tool definitions cost once they share a long agent session with everything else.