Langchain, async streaming with memory.

[sorrentinopasquale]

Hi everyone,

I'm working on a Node.js application that involves streaming with memory management. While the app functions correctly during the first conversation, I'm encountering issues with asynchronous operations and a need to restart the system to make it work for subsequent API calls for new users. I would greatly appreciate any guidance or suggestions to resolve these issues.

Issue Summary:

The app works fine for the first conversation.
Subsequent conversations require a system restart for the app to function.
The streaming is not asynchronous, leading to performance issues.

All in
router.post('/myAPI', async (req, res) => {

// Initializing memory with chat history I get from my DB
const memory = new BufferMemory({
chatHistory: new ChatMessageHistory(pastMessages),
returnMessages: true, memoryKey: "history"
});

// OpenAI LLM options configuration
const openAILLMOptions = {
modelName: modelname,
cache: true,
openAIApiKey: 'mykey',
temperature: temperature,
streaming: true,
verbose: true,
};

// Chat prompt setup
const chatPrompt = ChatPromptTemplate.fromPromptMessages([
SystemMessagePromptTemplate.fromTemplate("The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know." + modifiedContent),
new MessagesPlaceholder("history"),
HumanMessagePromptTemplate.fromTemplate("{input}"),
]);

// Model and streaming chain initialization
const chatModel = new ChatOpenAI(openAILLMOptions);
const chain = new LLMChain({
memory: memory,
prompt: chatPrompt,
llm: chatModel,
});

// Streaming function
const stream = await chain.stream({
handleLLMNewToken(token: any) {
res.write(data: ${JSON.stringify(sentence)}\n\n);
},
handleLLMEnd() {
res.end();
}
});

[dosubot[bot]]

🤖

Based on your description, it seems like you're encountering issues with memory management and asynchronous operations when making multiple API calls. To handle this, you can use the stream method provided by the LangChain framework. This method allows you to handle the response in a streaming manner, which is beneficial for memory management as it doesn't require storing the entire response in memory.

Here is an example of how you can use the stream method:

const chain = prompt
  .pipe(model)
  .pipe(new JsonOutputFunctionsParser({ diff: true }));

const stream = await chain.stream({
  foo: "bears",
});

// Stream a diff as JSON patch operations
for await (const chunk of stream) {
  console.log(chunk);
}

const chain2 = prompt.pipe(model).pipe(new JsonOutputFunctionsParser());

const stream2 = await chain2.stream({
  foo: "beets",
});

// Stream the entire aggregated JSON object
for await (const chunk of stream2) {
  console.log(chunk);
}

In this code, the stream method is used to handle the response from the chain object in a streaming manner. For each new chunk received from the stream, the chunk is logged to the console. You can modify this to handle the chunk in a way that suits your application's needs.

For memory management, LangChain uses the BufferMemory class in conjunction with the ConversationChain class. The BufferMemory class is responsible for managing the memory of the conversation history. It stores the conversation history in a buffer and returns the messages when needed. This is particularly useful when dealing with large conversation histories, as it allows the history to be stored efficiently without requiring a large amount of memory.

Here is an example of how you can use the BufferMemory class:

const chain = new ConversationChain({
  memory: new BufferMemory({ returnMessages: true, memoryKey: "history" }),
  prompt: chatPrompt,
  llm: chat,
});

In this code, chain is an instance of the ConversationChain class, which uses an instance of the BufferMemory class to manage the memory of the conversation history. The call method is then used to process the conversation, with the conversation history being stored in the buffer memory and returned as needed.

I hope this helps! If you have any further questions, feel free to ask.

Sources

• examples/src/models/chat/chat_streaming.ts
• examples/src/chains/llm_chain_stream.ts
• examples/src/models/llm/llm_streaming_stdout.ts
• examples/src/chains/llm_chain_cancellation.ts
• examples/src/prompts/json_structured_output_parser_streaming.ts
• examples/src/models/chat/chat_streaming_stream_method.ts
• examples/src/chat/memory.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[sorrentinopasquale]

Can someone give a more specific advise?

[rasert]

This example shows how to work with streaming, but not streaming with memory. How can I do that?

https://js.langchain.com/docs/modules/chains/popular/chat_vector_db#streaming