Stream with agentWithChatHistory

[KeenanFernandes2000]

How do I use streaming with agentWithChatHistory, I have tried

const streamRes = await agentWithChatHistory.stream(
      {
        input: message,
      },
      {
        // callbacks: [tracer],
        configurable: {
          sessionId: sessionID,
        },
      }
    );

however it is only outputting the final piece once it is generated

[dosubot[bot]]

🤖

Based on the context provided, it seems like the issue you're facing is related to the handling of streaming in the LangChain framework. The _generate method of the OpenAI class waits for all the data to be received from the OpenAI API before resolving the Promise, which is why handleLLMNewToken receives all tokens at once.

To get the output as it's being generated, you need to listen for changes to the streamed_output_str array and concatenate the chunks as they're added. Here's a simplified example of how you might do this:

let currentOutput = '';

agentWithChatHistory.stream().subscribe({
  next: (update) => {
    if (update.ops[0].op === 'add' && update.ops[0].path.startsWith('/logs/ChatOpenAI:2/streamed_output_str')) {
      currentOutput += update.ops[0].value;
      console.log(currentOutput);
    }
  },
  error: (err) => console.error(err),
  complete: () => console.log('Stream completed'),
});

In this example, we're subscribing to the stream and checking each update. If the update is an 'add' operation to the streamed_output_str array, we append the new chunk to currentOutput and log it. This way, you can see the output as it's being generated, rather than waiting for the final output.

You can find more details in the examples/src/agents/stream_log.ts file in the LangChainJS repository.

Additionally, it's worth noting that the .stream() method does not work properly for ConversationChain and LLMChain subclasses, and the old callback approach should be used instead. This was mentioned in the Streaming broken for ChatOpenAI in version 0.0.132 (and before) issue.

Lastly, one user mentioned that the issue was resolved by updating to Bun 0.8, which implies that the problem might have been related to the runtime environment rather than the LangChain framework itself. However, this comment did not receive any reactions, so it should be considered with lower precedence.

I hope this helps! If you have any other questions or need further clarification, feel free to ask.

Sources

• Streaming broken for ChatOpenAI in version 0.0.132 (and before)
• examples/src/agents/stream_log.ts
• examples/src/agents/streaming.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[KeenanFernandes2000]

@dosu-bot is there an easier way to implement streaming that would allow me to continue using sessions as well

[dosubot[bot]]

🤖

Based on the issues you've encountered and the context provided, it seems that the streaming issue with agentWithChatHistory might be related to how the LangChain framework handles streaming. In the issue #2365, it was suggested that the _generate method of the OpenAI class waits for all the data to be received from the OpenAI API before resolving the Promise, which is why handleLLMNewToken receives all tokens at once. The suggested solution is to modify the code to resolve the Promise or handle the data as soon as each token is received, rather than waiting for the [DONE] message.

However, modifying the _generate method might not be the best approach as it could potentially break other parts of the code. Instead, you might want to consider using the createStructuredChatAgent function provided by the LangChain framework. This function returns a runnable sequence representing an agent and supports streaming. Here is an example of how you can use it:

import { AgentExecutor, createStructuredChatAgent } from "langchain/agents";
import { pull } from "langchain/hub";
import type { ChatPromptTemplate } from "@langchain/core/prompts";
import { AIMessage, HumanMessage } from "@langchain/core/messages";

import { ChatOpenAI } from "@langchain/openai";

// Define the tools the agent will have access to.
const tools = [...];

// Get the prompt to use - you can modify this!
const prompt = await pull<ChatPromptTemplate>(
  "hwchase17/structured-chat-agent"
);

const llm = new ChatOpenAI({
  temperature: 0,
  modelName: "gpt-3.5-turbo-1106",
});

const agent = await createStructuredChatAgent({
  llm,
  tools,
  prompt,
});

const agentExecutor = new AgentExecutor({
  agent,
  tools,
});

const result = await agentExecutor.invoke({
  input: "what is LangChain?",
});

// With chat history
const result2 = await agentExecutor.invoke({
  input: "what's my name?",
  chat_history: [
    new HumanMessage("hi! my name is cob"),
    new AIMessage("Hello Cob! How can I assist you today?"),
  ],
});

In this example, the createStructuredChatAgent function is used to create an agent that supports streaming. The agent is then invoked with an input and a chat history. The result is either an AgentAction or AgentFinish. You can find more details in the langchain/src/agents/structured_chat/index.ts file.

As for the agentWithChatHistory.stream function, I'm afraid I can't provide an exact description of its functionality as it's not explicitly mentioned in the provided context. If you could provide more details about this function or the context in which it's used, I might be able to provide a more accurate answer.

Sources

• Streaming broken for ChatOpenAI in version 0.0.132 (and before)
• langchain/src/agents/structured_chat/index.ts
• examples/src/models/chat/chat_streaming.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.