Skip to content

# Add Support for Anthropic's Built-in Tools #8248

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

bleafman
Copy link

@bleafman bleafman commented May 23, 2025

Add Support for Anthropic's Built-in Tools

Adds support for Anthropic's built-in tools web search to bring feature parity with the Python implementation.

  • Added support for server_tool_use, web_search_tool_result, and web_search_result content blocks
  • Extended message formatting to handle server tool content blocks
  • Added test coverage

Fixes #8215

Twitter Handle: @BrandonLeafman

Copy link

vercel bot commented May 23, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
langchainjs-docs ✅ Ready (Inspect) Visit Preview Jun 8, 2025 7:54pm
1 Skipped Deployment
Name Status Preview Comments Updated (UTC)
langchainjs-api-refs ⬜️ Ignored (Inspect) Jun 8, 2025 7:54pm

@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. auto:enhancement A large net-new component, integration, or chain. Use sparingly. The largest features labels May 23, 2025
@bleafman
Copy link
Author

bleafman commented May 23, 2025

Here's some more background / rational for the changes:

Technical Details

Anthropic's built-in tools work differently from regular tools - they execute server-side and return specialized content blocks in API responses:

  • server_tool_use: When Claude uses a built-in tool (like web search)
  • web_search_tool_result: Container for web search results
  • web_search_result: Individual search result within the container

These content blocks need to be parsed otherwise you get a "Unsupported message content format" errors on subsequent turns after the built-in tool usage.

The @anthropic-ai/sdk package needed to be bumped to access the WebSearchTool20250305 type.

Changes made:

  1. types.ts: Added imports for the new content block types from @anthropic-ai/sdk
  2. message_inputs.ts:
    • Added the new types to toolTypes array in _formatContent() (line ~294)
    • Updated normalizeContent() union type to include server tool types (line ~578)
  3. message_outputs.ts: Enhanced streaming to handle server tool content blocks

I attempted to follows existing patterns - server tool content blocks get the same processing as regular tool blocks (index removal, input validation, etc.).

Python parity: The Python implementation already supported these via the Anthropic SDK's native types. This brings the JS version to the same level.

Testing: Added both unit tests (content block parsing) and integration tests (real API calls) to ensure reliability.

@dikaioai
Copy link

Please merge this!

… function from 'any' to 'unknown' for improved type safety
Copy link
Contributor

@benjamincburns benjamincburns left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for submitting this, @bleafman! I had some questions, and a few requested changes. Hopefully it's nothing too troublesome, but let me know if you need me to clarify anything.

Comment on lines 107 to 109
| Anthropic.Messages.ToolBash20250124
| Anthropic.Messages.ToolTextEditor20250124
| Anthropic.Messages.WebSearchTool20250305 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a union for built-in tools? If not, it would be good to create one in types.ts.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a export type ToolUnion = Tool | ToolBash20250124 | ToolTextEditor20250124 | WebSearchTool20250305; from the Anthropic SDK

Tool is for custom tools, so it really widens the typing and doesn't seem like a great fit to assert if a Tool is one of the built in tools (see typing below).

Creating one and putting it in types.ts seems best (will put more details in your comment about the type property).

// From @anthropic-ai/sdk's messages.ts 
export interface Tool {
  /**
   * [JSON schema](https://json-schema.org/draft/2020-12) for this tool's input.
   *
   * This defines the shape of the `input` that your tool accepts and that the model
   * will produce.
   */
  input_schema: Tool.InputSchema;

  /**
   * Name of the tool.
   *
   * This is how the tool will be called by the model and in `tool_use` blocks.
   */
  name: string;

  /**
   * Create a cache control breakpoint at this content block.
   */
  cache_control?: CacheControlEphemeral | null;

  /**
   * Description of what this tool does.
   *
   * Tool descriptions should be as detailed as possible. The more information that
   * the model has about what the tool is and how to use it, the better it will
   * perform. You can use natural language descriptions to reinforce important
   * aspects of the tool input JSON schema.
   */
  description?: string;

  type?: 'custom' | null;
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if export type AnthropicBuiltInTool = Anthropic.Messages.Tool & { type : null } wouldn't be enough to resolve AnthropicBuiltInTool to what we'd be after, without having to explicitly redefine. We like to derive types (and schema) from the provider SDK when possible, as it makes forward compatibility a bit easier when they make additive changes.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed up a change that uses:

export type AnthropicBuiltInToolUnion = Exclude<
  Anthropic.Messages.ToolUnion,
  Anthropic.Messages.Tool
>;

This separates out the custom tool and should make it forward compatible since it's just deriving from the SDK types (theoretically Anthropic could put something unexpected in the union, but this at least the type inference to be "Everything besides the custom tools").

Let me know what you think about that one.

export type AnthropicBuiltInTool = Anthropic.Messages.Tool & { type : null }`` doesn't work because the built-in tools all have string literals with they timestamp (ex. text_editor_20250124) for the their type. The built-in tools and the custom tools don't actually overlap on the type` property.

type actually would let you narrow, but typing "Any string that isn't literal 'custom'" is pretty gross/kluggy 😓

Comment on lines 102 to 109
{
type: "server_tool_use",
id: "toolu_01ABC123",
name: "web_search",
input: {
query: "latest AI developments",
},
},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you seen how the built-in tool responses are formatted by the OpenAI provider when using the responses API? Would it be possible to make this structure look similar to that? This will cut down on the number of breaking changes we'll need to make when we eventually go to standardize this type of output.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah! I didn't! That's helpful context. I'll see if I can get the structure to be similar.

Copy link
Author

@bleafman bleafman May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjamincburns I could use your confirmation/feedback before moving forward on how to split up / organize the data returned as part of the tool call.

You can see an example from Anthropic of a response with their built in web search tool here.

One of the big advantages of the built-in tool is that Claude will respond with text + citations side by side and you can use that render results (you're also required as part of the ToS to display the citations with any returned results).

Based on what's happening in the OpenAI provider, specifically in the _convertOpenAIResponseMessageToBaseMessage, it looks like we'd just append the citations as-is to an entry in the content array. Does that sound right?

It end up coming through like below:

 content: [
 {
      "citations": [
          {
              "type": "web_search_result_location",
              "cited_text": "Santolina, also known as cotton lavender, is a small, mounded shrub with silvery, aromatic foliage and button-like yellow flowers. It thrives in dry, ...",
              "url": "https://mygardeninspo.com/mediterranean-flowers/",
              "title": "26 Mediterranean Flowers",
              "encrypted_index": "..."
          }
      ],
      "type": "text",
      "text": "This is a small, mounded shrub with silvery, aromatic foliage and yellow button-like flowers. It's perfect for your needs as it thrives in dry, rocky soils and is often used for edging paths and borders. It prefers full sun and minimal watering"
  },
  // ...more entries
  ]

For the tool use itself, it seems like I should pass that into the tool_calls as a normal tool block, right?

I basically have the content working, the tool_calls are a little tricky since Anthropic expects a specific format in follow up turns but I have some ideas.

Just want to confirm before I go way down the wrong direction or I'm missing something.

Note: The OpenAI Provider does put some data into additional_kwargs.tool_calls but that field is marked as deprecated so I was planning on not replicating that behavior.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bleafman I would think that's OK. We have some opinions about how we want to standardize those citation/annotation outputs, but that can be for a later iteration.
(we're doing something similar with the annotations key for openai here)

@benjamincburns
Copy link
Contributor

Please also be sure to run yarn format && yarn lint && yarn build before committing.

Brandon Leafman added 2 commits May 28, 2025 10:32
…e correct 'str_replace_editor' name, enhance test cases for server tools
… responses, remove outdated integration tests for built-in tools and server tools
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Jun 8, 2025
@bleafman bleafman requested a review from benjamincburns June 8, 2025 19:42
@bleafman
Copy link
Author

bleafman commented Jun 8, 2025

Alright! I think this is ready for review again.

A few notes/questions:

  1. Limited down the scope to just the web_search tool, testing all the other tools was going to be too much and the other built-in tools are in beta anyway and may change (or be removed entirely).

  2. The isBuiltinTool currently allows the bash tool and the text editor tool. I can remove that if we want to avoid the risk around "all observable behaviors of your system will be depended on by somebody"

  3. When testing in my app, I noticed that the web search tool streams much more of the response in parallel* compared to non-server tool calls / regular Claude responses. This isn't an issue, more just a note that the behavior for the built in tool will output the same but feels a bit different when streaming vs. a standard bound tool since it's happening on the Anthropic server-side.

*This is speculation, but a recent breakdown I read about Claude Code seems to indicate that the search results are summarized by another LLM, so it'd make sense that it's streaming these responses in parallel under the hood.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto:enhancement A large net-new component, integration, or chain. Use sparingly. The largest features size:L This PR changes 100-499 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feature: support anthropic's new web search tool
4 participants