Structured output and tool calling not working for Nova

[PabloElgueta623]

Testing both structured output and tool calling provide the same error: An error occurred (ModelErrorException) when calling the Converse operation: The system encountered an unexpected error during processing. Try your request again. This doesn't occur when using normal generation, and it works without using Langchain.

Code to replicate:

from langchain_core.messages import SystemMessage, HumanMessage
from pydantic import Field, BaseModel
from models import OutputGuardrail

client = client(
    service_name='bedrock-runtime',
    region_name='eu-west-1',

)
additionalModelRequestFields={"inferenceConfig": {"topK": 1}}

nova_micro = ChatBedrockConverse(
    model="eu.amazon.nova-micro-v1:0",
    client=client,
    region_name="eu-west-1",
    temperature=1,
    top_p=1,
    additional_model_request_fields=additionalModelRequestFields,
)
class TestModel(BaseModel):
    """A test model to check the tool use conversion."""
    name: str = Field(description="The name of the user.")
    age: int = Field(description="The age of the user.")

model = nova_micro.bind_tools([TestModel], tool_choice="any")
messages = [SystemMessage(content="You are a chatbot assistant."), HumanMessage(content="Hi")]
response = model.invoke(messages)
print(response)```



Update: I've been testing this functionality with the Converse API directly, and believe it's a problem with that, not the LangChain implementation. Whenever I try to use 'any' or named tool parameter, it gives the error above.

[3coins]

@PabloElgueta623
Thanks for reporting this issue. I am not able to replicate the issue with us.amazon.nova-micro-v1:0 and us-west-2 region. Note that the sample code included in description is not an example of structured output. For that use-case you should use with_structured_output rather than bind_tools directly. Here is a slightly modified version that I tested.

from langchain_core.messages import SystemMessage, HumanMessage
from pydantic import Field, BaseModel

from langchain_aws import ChatBedrockConverse


additionalModelRequestFields = {"inferenceConfig": {"topK": 1}}

nova_micro = ChatBedrockConverse(
    model="us.amazon.nova-micro-v1:0",
    region_name="us-west-2",
    temperature=1,
    top_p=1,
    additional_model_request_fields=additionalModelRequestFields,
)


class TestModel(BaseModel):
    """A test model to check the tool use conversion."""

    name: str = Field(description="The name of the user.")
    age: int = Field(description="The age of the user.")


if __name__ == "__main__":
    model = nova_micro.with_structured_output(TestModel)
    messages = [
        SystemMessage(content="You are a chatbot assistant."),
        HumanMessage(content="Hi"),
    ]
    response = model.invoke(messages)
    print(response)

   # This produces name='User' age=0, since no relevant info was passed
   # Passing in a message as "User's name is John and their age is 15", would
   # produce name='John' age=15

[ficiverson]

Trying you exact example:

from langchain_core.messages import SystemMessage, HumanMessage
from pydantic import BaseModel, Field
from langchain_aws import ChatBedrockConverse

additionalModelRequestFields = {"inferenceConfig": {"topK": 1}}

llm_model = ChatBedrockConverse(
    model_id="eu.amazon.nova-lite-v1:0",
    region_name="eu-west-1",
    temperature=1,
    top_p=1.0,
    credentials_profile_name="my_profile",
    additional_model_request_fields=additionalModelRequestFields,
)


class TestModel(BaseModel):
    """A test model to check the tool use conversion."""

    name: str = Field(description="The name of the user.")
    age: int = Field(description="The age of the user.")

model = llm_model.with_structured_output(TestModel)
messages = [
        SystemMessage(content="You are a chatbot assistant."),
        HumanMessage(content="Hi"),
    ]
response = model.invoke(messages)
print(response)

I am getting the same error as @PabloElgueta623

ModelErrorException: An error occurred (ModelErrorException) when calling the Converse operation: The system encountered an unexpected error during processing. Try your request again.

Could be the regional model different? I see mine and @PabloElgueta623 are models from eu and yours from us region. Do we need to configure something in the AWS console?

Thanks!

[ficiverson]

Update: It seems Nova model doesnt support with_structured_output method. I created this small piece to parse content and now its working.

def manual_nova_parser(model: TBaseModel, query: str):
    # Set up a parser
    parser = PydanticOutputParser(pydantic_object=model)

    try:
        # Prompt
        prompt = ChatPromptTemplate.from_messages(
            [
                (
                    "system",
                    "Answer the user query. Wrap the output in `json` tags\n{format_instructions}",
                ),
                ("human", "{query}"),
            ]
        ).partial(format_instructions=parser.get_format_instructions())
        chain = prompt | llm_model | parser
        base_response = chain.invoke(query)
    
        return base_response
        
    except Exception as e:
        print(f"Error occurred: {type(e).__name__}")
        print(f"Error message: {str(e)}")

result = manual_nova_parser(TestModel,"I am Fernando and I am X years old")

[3coins]

@ficiverson
Thanks for verifying that. I can replicate the issue by changing to eu-west-1 and eu.amazon.nova-micro-v1:0, looking further into with_structured_output to debug the issue.

[3coins]

It seems like the problem is at the service end, where nova model in eu region doesn't work with any or <tool-name> toolChoice values. This is perhaps a gap in the current documentation or a delay in propagation of this feature in the eu region.

One solution would be to adjust the supports_tool_choice_values for eu region to only auto or None, so these errors are not seen. For reference, the code worked for me in eu region when I hardcoded the tool_choice as auto or None.

[3coins]

I closed the PR that I had opened initially to add a fix, but felt that it might cause other problems. A rather simple workaround would be to pass in the supports_tool_choice_values directly while creating the ChatBedrockConverse class.

nova_micro = ChatBedrockConverse(
    model="eu.amazon.nova-micro-v1:0",
    region_name="eu-west-1",
    temperature=0,
    top_p=1,
    additional_model_request_fields=additionalModelRequestFields,
    supports_tool_choice_values=()
)

I will follow up with the Bedrock team and find out more about tool choice support in different regions, until then this solution should suffice.

[TobiasMaehl-pIX]

In region eu-central-1, when passing supports_tool_choice_values=() the model response is always None

[3coins]

In region eu-central-1, when passing supports_tool_choice_values=() the model response is always None

That is expected for an input of "Hi", that has no relevant data to convert to the model output, providing an input like "User's name is John and their age is 15" will provide a valid model output.

[TobiasMaehl-pIX]

That's clear, we provided sufficient input for the schema.
Further experiments showed that in eu-central-1, eu.amazon.nova-pro-v1:0 was able to provide a structured output, while eu.amazon.nova-micro-v1:0 did not. For Nova Pro, we did not have to override supports_tool_choice_values.

[3coins]

Confirmed with the Bedrock team, that the expanded tool choice feature has only been enabled in some regions; an authoritative list hasn't been published, but seems like it was originally enabled only in

• us-east-1
• us-east-2
• ap-northeast-1
• eu-north-1
• us-iso-east-1
• us-gov-west-1

Since the feature is GA now, this will be enabled in all regions (where model is available) in the coming few months. Will keep this issue open until then, so users can find the discussion here useful.