AWS Bedrock Converse API Prompt Caching not supported

[isaac-j-miller]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain.js documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain.js rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

the following code:

import { BedrockRuntimeClient } from '@aws-sdk/client-bedrock-runtime';
import { ChatBedrockConverse } from '@langchain/aws';
import {
    SystemMessage,
    HumanMessage
} from '@langchain/core/messages';

const bedrock = new BedrockRuntimeClient({region:"us-east-1"});
const chatBedrockConverse = new ChatBedrockConverse({
    model: "us.amazon.nova-lite-v1:0",
    client: bedrock
});

const messages = [
  new SystemMessage({content:[
    {
        type: "text",
        text: "my system prompt"
    },
    {
        cachePoint: {
        type: "default"
        }
    }
  ]
}),
new HumanMessage("foo bar")
]

await chatBedrockConverse.invoke(messages);
// throws "System message content must be either a string, or a content array containing a single text object."
// from https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-aws/src/common.ts:355

Error Message and Stack Trace (if applicable)

"System message content must be either a string, or a content array containing a single text object."
from https://github.com/langchain-ai/langchainjs/blob/main/libs/langchain-aws/src/common.ts:355

Description

I'm trying to use prompt caching, per https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html, with the Converse API. It works with langchain in python.

System Info

langchain@0.3.24 | MIT | deps: 12 | versions: 319
Typescript bindings for langchain
https://github.com/langchain-ai/langchainjs/tree/main/langchain/

keywords: llm, ai, gpt3, chain, prompt, prompt engineering, chatgpt, machine learning, ml, openai, embeddings, vectorstores

dist
.tarball: https://registry.npmjs.org/langchain/-/langchain-0.3.24.tgz
.shasum: 24c5e16f7a43be8ce54d32079ea8dc8087c56777
.integrity: sha512-BTjiYkUCpWFAmufK8J5zMqc5aUs4eEnAXPWtPe2+R4ZPP+U7bXJSBHAcrB40rQ3VeTdRgMvgDjekOOgCMWut6Q==
.unpackedSize: 4.8 MB

dependencies:
@langchain/openai: >=0.1.0 <0.6.0 js-yaml: ^4.1.0 openapi-types: ^12.1.3 yaml: ^2.2.1
@langchain/textsplitters: >=0.0.0 <0.2.0 jsonpointer: ^5.0.1 p-retry: 4 zod-to-json-schema: ^3.22.3
js-tiktoken: ^1.0.12 langsmith: ^0.3.16 uuid: ^10.0.0 zod: ^3.22.4

maintainers:

• nfcampos nuno@boringbits.io
• jacoblee93 jacoblee93@gmail.com
• andrewnguonly andrewnguonly@gmail.com
• benjamincburns benjamin.c.burns@gmail.com
• davidduong david@duong.cz
• hwchase17 hw.chase.17@gmail.com
• basproul braceasproul@gmail.com

dist-tags:
latest: 0.3.24 next: 0.3.2-rc.0 tag-for-publishing-older-releases: 0.2.20

published 2 weeks ago by basproul braceasproul@gmail.com

Using Node.js v22, on Fedora

[kishan0725]

Any update on this? @benjamincburns

[rohitkf]

I thought ChatBedrockConverse API now supports it, as per #326

[kishan0725]

@rohitkf I think it does support python but not js version

[rohitkf]

Can someone help me with Prompt Caching ?
I'm trying to figure out how to do it but if failing desperately.

Goal :

• Use ChatBedrockConverse
• Use Pydantic Structured Output ( using llm.with_structured_output() )
• Use Prompt Caching on the SystemMessage (with ChatPromptTemplate.from_messages())
• Get usage metadata to see the token count, cache read/write details

from langchain_aws import ChatBedrockConverse
from langchain_core.messages import HumanMessage, AIMessage, SystemMessage
from langchain_core.prompts import ChatPromptTemplate
from pydantic import BaseModel, Field
from typing import Literal
import boto3
import json
from strenum import StrEnum
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

LlmMaxTokens = Literal[250, 512]
MAX_TOKENS: LlmMaxTokens = 250
# this value was raised from 250 to 512 as it was causing errors in the TerminationType
#   the TT response was being cut off causing formatting errors.
TT_MAX_TOKENS: LlmMaxTokens = 512
TEMPERATURE = 0.0
REGION = "us-east-1"

def get_llm(model_id, max_tokens: LlmMaxTokens = MAX_TOKENS, thinking_enabled: bool = False):
    """get_llm
    Returns an instance of an llm
    Returns:
        _type_: BedrockConverse
    """
    additional_params = {
        "thinking": {
            "type": "enabled",
            "budget_tokens": 2000
        },
        "top_k": 20
    } if thinking_enabled else {"top_k": 20}

    bedrock_client = boto3.client("bedrock-runtime", REGION)
    return ChatBedrockConverse(
        model=model_id,
        region_name=REGION,
        client=bedrock_client,
        max_tokens=max_tokens,
        temperature=TEMPERATURE,
        top_p=0.999,
        additional_model_request_fields=additional_params,
    )

# Initialize the LLM
llm = get_llm("anthropic.claude-3-5-sonnet-20240620-v1:0")

# Define the prompt template
user_prompt = "what is the capital of {input}"

prompt = ChatPromptTemplate.from_messages(
    [
        SystemMessage(content="You are a helpful assistant.", additional_kwargs={"cachePoint": "default"}),
        HumanMessage(content=user_prompt)
    ]
)

# Define a specific Pydantic model for structured output
class ResponseModel(BaseModel):
    answer: str = Field(description="The answer to the question")

# Create a chain using LangChain's pipe syntax
# This follows the standard LangChain pattern for creating chains
chain = (
    {"input": RunnablePassthrough()}
    | prompt 
    | llm.with_structured_output(ResponseModel)
)

# Execute the chain
response = chain.invoke({"input": "France"})

# Display the response
response