ChatBedrockConverse vs ChatBedrock (difference in preparation of prompts?)

[Elanmarkowitz]

It seems like there is a difference between ChatBedrockConverse and ChatBedrock regarding the preparation of prompts.
ChatBedrock uses ChatPromptAdapter while ChatBedrockConverse does not.

Wondering if this is desired behavior or not?

[michaelnchin]

The ChatBedrockConverse uses Bedrock's new Converse API, which automatically applies prompt templating specific to the model used.

The older InvokeModel API that ChatBedrock uses does not provide this functionality, so the ChatPromptAdapter is needed for prompt preprocessing.

[3coins]

The older InvokeModel API that ChatBedrock uses does not provide this functionality, so the ChatPromptAdapter is needed for prompt preprocessing.

A small correction, InvokeModel API is just a different API that puts the onus on the developer to handle the native formatting for the model, hence the need for ChatPromptAdaptor here. Converse on the other handles the native formatting on the API backend, so simpler to use for most use cases. We (langchain-aws) have had better success with Converse, since it is easier to maintain for newer models, it is also much more complete for other use cases such as tool calling and langgraph.

Let us know if you have any other questions.

[Elanmarkowitz]

Thank you for the responses and clarification. That makes sense.

If I were to want to use the Bedrock batch inference, I could use the langchain ChatPromptAdapter and then send the batch requests in via AWS?

Something like this:

prompt = ChatPromptAdapter.convert_messages_to_prompt(bedrock_provider, [HumanMessage(content=prompt)], model)

and then add those to the s3 records file?

Or alternatively, does batch inference have an equivalent of Converse where there is a unified API?

[3coins]

I believe batch inference is not supported in the same way as converse, it is in-fact a totally different API via the bedrock client, not the bedrock-runtime as for converse and invoke. It's implementation is also quite different from the existing chat implementations, for example it needs an S3 bucket, and is submitted as an async job, rather than a sync API that can be implemented directly with LangChain. Feel free to discuss more about batch in #250.