Stream answer from MultiPromptChain

[thijssdaniels]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

import { ChatOpenAI } from '@langchain/openai';
import { PromptTemplate } from '@langchain/core/prompts';
import { RouterOutputParser } from 'langchain/output_parsers';
import { BaseChain, LLMChain, LLMRouterChain, MultiPromptChain } from 'langchain/chains';
import promptInfos from '../utils/promptFactory';
import { PromptStore } from '../utils/promptStore';
import { z } from "zod";

export const RoutedGenerator = async (
  prompt: string
) => {

  let llm = new ChatOpenAI({
    verbose: true,
    modelName: "gpt-4-1106-preview",
    temperature: 1,
    streaming: true,
  });

  let routerLLM = new ChatOpenAI({
    verbose: true,
    modelName: "gpt-4",
    temperature: 1,
    streaming: false,
  });

  let templates = promptInfos;

  let destinationChains: {[key: string]: BaseChain } = {};

  for ( const item of templates ) {
    let prompt = new PromptTemplate({template: item.prompt_template, inputVariables: ['input']});
    let chain = new LLMChain({llm: llm, prompt: prompt});
    destinationChains[item.name] = chain;
  }

  let destinations = templates.map(item => (item.name + ': ' + item.description)).join('\n');

  let defaultPrompt = new PromptTemplate({template: PromptStore.qaTemplate, inputVariables: ['input']});

  let defaultChain = new LLMChain({llm: llm, prompt: defaultPrompt});

  let routerTemplate = PromptStore.routerTemplate;

  routerTemplate = routerTemplate.replace('{destinations}', destinations);

  let routerParser = RouterOutputParser.fromZodSchema(z.object({
      destination: z
          .string()
          .describe('name of the prompt to use or "DEFAULT"'),
      next_inputs: z.object({
          input: z
              .string()
              .describe('a potentially modified version of the original input')
      })
  }))
  let routerFormat = routerParser.getFormatInstructions();

  let routerPrompt = new PromptTemplate({
    template: routerTemplate,
      inputVariables: ['input'],
      outputParser: routerParser,
      partialVariables: {
          format_instructions: routerFormat
      }
  });

  let routerChain = LLMRouterChain.fromLLM(routerLLM, routerPrompt);

  let multiPromptChain = new MultiPromptChain({
    routerChain,
    destinationChains,
    defaultChain, 
    verbose: true
  });

  const response = await multiPromptChain.invoke({input: prompt});

  return response.text;
};

Description

I'm building a web application where I want the LLM choose among different chains. I've successfully implement this with MultiPromptChain, however, I was wondering if I can stream the answers from the chains instead of providing the responses directly.

System Info

Node = v20.11.1
Yarn = v1.22.21
@langchain/core = 0.1.42

[dosubot[bot]]

Hey @thijssdaniels, great to see you diving into new challenges with LangChain JS! How's everything going on your end?

Based on the code you've provided, it seems you're trying to use the MultiPromptChain class to route prompts to different chains and you're interested in streaming the responses from these chains.

Currently, the MultiPromptChain class in the LangChain JS framework does not directly support streaming responses. The invoke method of MultiPromptChain waits for all the promises to resolve before returning the responses.

However, the ChatOpenAI class, which is a part of the LLMChain subclass, does support streaming responses. This is evidenced by the streaming: true option in the constructor of the ChatOpenAI instance and the implementation of a callback function (handleLLMNewToken) that processes each new token received from the stream. You can refer to this example for more details.

To achieve streaming responses from the chains in a MultiPromptChain implementation, you would need to handle each promise individually rather than waiting for all of them with Promise.all. This way, as each individual prompt is resolved, you can process and potentially stream the response immediately. Here's a conceptual approach:

const handleResponse = async (promise, name) => {
  const { text: result } = await promise;
  console.log(`${name}: ${result}`);
};

handleResponse(testPromise1, "Physics");
handleResponse(testPromise2, "Math");
handleResponse(testPromise3, "History");

This approach would allow each response to be processed and output as soon as it's available, which is closer to streaming the responses. However, for a true streaming implementation, especially if the responses themselves are large and can be streamed in parts, you would need to ensure that the underlying language model client (ChatOpenAI in this case) supports streaming responses and that MultiPromptChain is capable of handling these streamed responses appropriately.

I hope this helps! If you have any further questions, feel free to ask.

Sources

• Is it possible to use streaming for Custom LLM in chains ?
• examples/src/chains/multi_prompt.ts
• langchain/src/chains/router/tests/multi_prompt.test.ts
• examples/src/models/chat/chat_streaming_stdout.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.