Skip to content

Commit 73dad70

Browse files
feat(mcp-adapters): support audio and standard content blocks (#8252)
1 parent 364cf5f commit 73dad70

File tree

5 files changed

+917
-348
lines changed

5 files changed

+917
-348
lines changed

libs/langchain-mcp-adapters/README.md

Lines changed: 73 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -223,6 +223,7 @@ When loading MCP tools either directly through `loadMcpTools` or via `MultiServe
223223
| `throwOnLoadError` | boolean | `true` | Whether to throw an error if a tool fails to load |
224224
| `prefixToolNameWithServerName` | boolean | `true` | If true, prefixes all tool names with the server name (e.g., `serverName__toolName`) |
225225
| `additionalToolNamePrefix` | string | `mcp` | Additional prefix to add to tool names (e.g., `prefix__serverName__toolName`) |
226+
| `useStandardContentBlocks` | boolean | `false` | If true, uses LangChain's standard multimodal content blocks. Defaults to false for backward compatibility; recommended to set true for new applications |
226227

227228
## Tool Timeout Configuration
228229

@@ -272,15 +273,34 @@ Timeouts can be configured using the following `RunnableConfig` fields:
272273
| `timeout` | number | 60000 | Timeout in milliseconds for the tool call |
273274
| `signal` | AbortSignal | undefined | An AbortSignal that, when asserted, will cancel the tool call |
274275

275-
## Response Handling
276+
## Reading Tool Outputs
276277

277-
MCP tools return results in the `content_and_artifact` format which can include:
278+
The tools returned by `client.getTools` and `loadMcpTools` are LangChain tools that return ordinary LangChain `ToolMessage` objects. See the table below for the different types of tool output supported by MCP, and how we map them into the LangChain `ToolMessage` object:
278279

279-
- **Text content**: Plain text responses
280-
- **Image content**: Base64-encoded images with MIME type
281-
- **Embedded resources**: Files, structured data, or other resources
280+
| MCP Tool Output Type | LangChain Mapping | Notes |
281+
| -------------------- | ----------------- | ----- |
282+
| **Text content** | Added to `ToolMessage.content` | See [Content Block Formats](#content-block-formats) for format details |
283+
| **Image content** | Added to `ToolMessage.content` | See [Content Block Formats](#content-block-formats) for format details |
284+
| **Audio content** | Added to `ToolMessage.content` | See [Content Block Formats](#content-block-formats) for format details |
285+
| **Embedded resources** | Added to `ToolMessage.artifact` array | Embedded resources are not transformed in any way before adding them to the arfifact array |
282286

283-
Example for handling different content types:
287+
### Content Block Formats
288+
289+
The `useStandardContentBlocks` option controls how content blocks returned by tools are formatted in the `ToolMessage.content` field. This option defaults to `false` for backward compatibility with existing applications, but **new applications should set this to `true`** to use LangChain's standard multimodal content blocks.
290+
291+
**When `useStandardContentBlocks` is `false` (default for backward compatibility):**
292+
- **Images**: Returned as [`MessageContentImageUrl`](https://v03.api.js.langchain.com/types/_langchain_core.messages.MessageContentImageUrl.html) objects with base64 data URLs (`data:image/png;base64,<data>`)
293+
- **Audio**: Returned as [`StandardAudioBlock`](https://v03.api.js.langchain.com/types/_langchain_core.messages.StandardAudioBlock.html) objects.
294+
- **Text**: Returned as [`MessageContentText`](https://v03.api.js.langchain.com/types/_langchain_core.messages.MessageContentText.html) objects.
295+
296+
**When `useStandardContentBlocks` is `true` (recommended for new applications):**
297+
- **Images**: Returned as base64 [`StandardImageBlock`](https://v03.api.js.langchain.com/types/_langchain_core.messages.StandardImageBlock.html) objects.
298+
- **Audio**: Returned as base64 [`StandardAudioBlock`](https://v03.api.js.langchain.com/types/_langchain_core.messages.StandardAudioBlock.html) objects.
299+
- **Text**: Returned as [`StandardTextBlock`](https://v03.api.js.langchain.com/types/_langchain_core.messages.StandardTextBlock.html) objects.
300+
301+
**Note**: The `useStandardContentBlocks` does not impact embedded resources. Embedded resources are always assigned to `ToolMessage.artifact` as an array of MCP `EmbeddedResource` objects, regardless of whether their MIME type indicates one of the formats specified above.
302+
303+
### Example Usage
284304

285305
```ts
286306
const tool = tools.find((t) => t.name === "mcp__math__calculate");
@@ -296,12 +316,19 @@ const [textContent, artifacts] = result;
296316
if (typeof textContent === "string") {
297317
console.log("Result:", textContent);
298318
} else {
299-
// Handle complex content (text + images)
319+
// Handle complex content (text + images/audio)
300320
textContent.forEach((item) => {
301321
if (item.type === "text") {
302322
console.log("Text:", item.text);
303323
} else if (item.type === "image_url") {
324+
// Legacy format (useStandardContentBlocks: false)
304325
console.log("Image URL:", item.image_url.url);
326+
} else if (item.type === "image") {
327+
// Standard format (useStandardContentBlocks: true)
328+
console.log("Image data:", item.data, "MIME:", item.mime_type);
329+
} else if (item.type === "audio") {
330+
// Audio always uses standard format
331+
console.log("Audio data:", item.data, "MIME:", item.mime_type);
305332
}
306333
});
307334
}
@@ -312,6 +339,45 @@ if (artifacts.length > 0) {
312339
}
313340
```
314341

342+
### Response Format Examples
343+
344+
**Legacy format** (`useStandardContentBlocks: false`):
345+
```ts
346+
const [content, artifacts] = await imageTool.invoke({ prompt: "a cat" });
347+
// content structure:
348+
[
349+
{
350+
type: "text",
351+
text: "Generated an image of a cat"
352+
},
353+
{
354+
type: "image_url",
355+
image_url: {
356+
url: "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA..."
357+
}
358+
}
359+
]
360+
```
361+
362+
**Standard format** (`useStandardContentBlocks: true`):
363+
```ts
364+
const [content, artifacts] = await imageTool.invoke({ prompt: "a cat" });
365+
// content structure:
366+
[
367+
{
368+
type: "text",
369+
source_type: "text",
370+
text: "Generated an image of a cat"
371+
},
372+
{
373+
type: "image",
374+
source_type: "base64",
375+
data: "iVBORw0KGgoAAAANSUhEUgAA...",
376+
mime_type: "image/png"
377+
}
378+
]
379+
```
380+
315381
## OAuth 2.0 Authentication
316382

317383
For secure MCP servers that require OAuth 2.0 authentication, you can use the `authProvider` option instead of manually managing headers. This provides automatic token refresh, error handling, and standards-compliant OAuth flows.

0 commit comments

Comments
 (0)