Integrating Function Calling APIs to LLM interaction

[Jkrish1011]

Hello team langchain, I am a newbie programmer and I need some guidance here.
Context:
I am using langchain to build a multi-model LLM pipe with the LLM querying an api and return the results based on the api information. For the same I wish to integrate the function api calling (https://js.langchain.com/docs/modules/chains/additional/openai_functions/openapi ) as discussed in this blog here.

Aim: As you all see the code based that I am using currently to chat with the LLM, how do I integrate the function api calling functionality into the code. I cannot find any reference to do that using Runnable Sequence.?

Question:
1/ As discussed in that blog and also this one (https://js.langchain.com/docs/integrations/chat/openai), how do I build the functionality where my model talks to a api of my choice, and then answer the question prompt.
2/ Should I write a function to call the api, or will the internal langchain function will make the call for LLM processing?
3/ Should I make a function call separately and add the result as context for LLM interactions.

any help or guidance would be appreciated.

Code

import { ChatOpenAI } from '@langchain/openai';
import { PromptTemplate } from '@langchain/core/prompts';
import {
  RunnableSequence,
  RunnablePassthrough,
} from '@langchain/core/runnables';
import {
  BytesOutputParser,
  StringOutputParser,
} from '@langchain/core/output_parsers';

export async function POST(req: NextRequest) {
const sample_standaloneQuestionTemplate = `
Given some conversation history (if any) and a question, covert the question to a standalone question.
question: {question}
standalone question:`;

const sample_answerTemplate = `You are a helpful assistant that can access external functions.
The responses from these function calls will be appended to this dialogue. Please provide responses based on the information from these function calls.
Use the following pieces of context to answer the question at the end and the conversation history.
Try to find the answer from the conversation context.
If you don't know the answer, just say you don't know. DO NOT try to make up an answer.
If the question is not related to the context, politely respond that you are tuned to only answer questions that are related to the context.

question: {question}
Conversation History: {chat_history}
answer: `;

const sample_prompt = PromptTemplate.fromTemplate(
  sample_standaloneQuestionTemplate,
);

const sample_answer = PromptTemplate.fromTemplate(sample_answerTemplate);

const model = new ChatOpenAI({
      modelName: 'mistralai/Mixtral-8x7B-Instruct-v0.1',
      temperature: 0,
      configuration: {
        apiKey:
          process.env.TOGETHER_AI_API_KEY,
        baseURL: 'https://api.together.xyz/v1',
      },
    });

const _standaloneQuestionChain = RunnableSequence.from([
      sample_prompt,
      model,
      new StringOutputParser(),
    ]);

    const _answerChain = RunnableSequence.from([
      {
        question: ({ original_input }) => original_input.question,
        chat_history: ({ original_input }) => original_input.chat_history,
      },
      sample_answer,
      model,
      new StringOutputParser(),
    ]);

    const _chain = RunnableSequence.from([
      {
        question: _standaloneQuestionChain,
        original_input: new RunnablePassthrough(),
      },
      _answerChain,
    ]);

    const stream1 = await _chain.stream({
      question: currentMessageContent,
      chat_history: formattedPreviousMessages.join('\n'),
    });
    return new StreamingTextResponse(stream1, {
      headers: {
        'x-message-index': (formattedPreviousMessages.length + 1).toString(),
        
      },
    });

}

[dosubot[bot]]

🤖

Hey there @Jkrish1011! 🎉 Great to see you back diving into the depths of langchainjs! How's the coding journey going?

To integrate the function API calling functionality into your existing langchainjs code, you can follow these steps:

1. Define a function that makes the API call and returns the result. This function should be asynchronous and return a Promise. Here's an example:

async function callApi() {
  // Replace with your actual API call
  const response = await fetch('https://api.example.com/data');
  const data = await response.json();
  return data;
}

2. Add this function to your RunnableSequence. The result of the function call will be added to the context for the next Runnable in the sequence. Here's how you can do it:

const _chain = RunnableSequence.from([
  {
    question: _standaloneQuestionChain,
    original_input: new RunnablePassthrough(),
    api_data: callApi, // Add your function here
  },
  _answerChain,
]);

3. In your answer template, you can now access the result of the API call using {api_data}. For example:

const sample_answerTemplate = `You are a helpful assistant that can access external functions.
The responses from these function calls will be appended to this dialogue. Please provide responses based on the information from these function calls.
Use the following pieces of context to answer the question at the end and the conversation history.
Try to find the answer from the conversation context.
If you don't know the answer, just say you don't know. DO NOT try to make up an answer.
If the question is not related to the context, politely respond that you are tuned to only answer questions that are related to the context.

question: {question}
Conversation History: {chat_history}
API Data: {api_data} // Access the API data here
answer: `;

This way, your language model will be able to interact with the API and respond to prompts based on the API information.

Please note that this is a simplified example and you might need to adjust it according to your specific needs and the structure of your API data.

For more information, you can refer to the LangChainJS documentation and the RunnableSequence API documentation.

I hope this helps! If you have any more questions, feel free to ask.

-Dosu

Sources

• examples/src/models/llm/openai_basePath.ts
• libs/langchain-openai/src/azure/chat_models.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Jkrish1011]

@dosu-bot
The RunnableSequence where api call is being made is not waiting for the api to return the data, thus the llm fails to get the context of the api call. How do we await for the function calling the api to complete before the LLM proceeds to curate a reply for the question.?

[Jkrish1011]

@dosu-bot
I got your perspective. Let me try out this approach and see if it work's out for the given scenario. Thanks.

[Jkrish1011]

@dosu-bot
The API call that I want to make is a POST request which requires 2 parameters, say Name and City.
I am expecting these 2 values be present in the user query that has been prompted. So I need to parse this and pass it to the callapi function.
Any direction for this?

[dosubot[bot]]

🤖

To integrate the function API calling functionality into your existing langchain code, you can create a new class that extends the StructuredTool class. This new class should implement the _call method, which will be responsible for making the API call and returning the result. Here's an example of how you can do this:

import axios from 'axios'; // You can use any library to make HTTP requests

class APITool extends StructuredTool {
  schema = z.object({
    endpoint: z.string().describe("The API endpoint"),
    // Add other parameters as needed
  });

  name = "api_tool";

  description = "Make a request to an API";

  constructor() {
    super(...arguments);
  }

  async _call(input: { endpoint: string }) {
    const { endpoint } = input;
    try {
      const response = await axios.get(endpoint); // Replace with the appropriate HTTP method
      return JSON.stringify(response.data);
    } catch (error) {
      console.error(error);
      return JSON.stringify({ error: "An error occurred while making the API request" });
    }
  }
}

You can then add an instance of this class to the tools array when creating an OpenAIAssistantRunnable:

const tools = [new WeatherTool(), new HumanReadableChecker(), new APITool()];
const agent = await OpenAIAssistantRunnable.createAssistant({
  model: "gpt-3.5-turbo-1106",
  instructions: "You are a weather bot. Use the provided functions to answer questions.",
  name: "Weather Assistant",
  tools,
  asAgent: true,
});

Now, your language model can interact with the API of your choice by using the api_tool tool. The _call method of the APITool class will be invoked when the language model uses this tool, and the result of the API call will be returned to the language model.

Regarding your question about RunnableSequence and RunnablePassthrough, these are classes in the langchainjs codebase used to manage and control the execution of a sequence of runnables and a mapping of runnables respectively.

The RunnableSequence class is used to manage a sequence of runnables. It has methods like invoke, batch, _streamIterator, pipe, isRunnableSequence, and from. The invoke method is used to execute the sequence of runnables one after the other, passing the output of one as the input to the next.

The RunnablePassthrough class is a special type of runnable that simply passes its input to its output without making any changes. This can be useful in situations where you want to preserve the original input while also transforming it in some way.

I hope this helps! Let me know if you have any other questions.

Sources

• langchain/src/experimental/openai_assistant/tests/openai_assistant.int.test.ts
• langchain-core/src/runnables/base.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

🤖

To make a POST request with parameters extracted from the user's query, you can create a function that parses the user's query and extracts the required parameters. This function can then be used to make the API call. Here's an example of how you can do this:

function parseUserQuery(query) {
  // Parse the user's query to extract the required parameters
  // This is a simple example and may need to be adjusted based on the actual format of the user's query
  const [name, city] = query.split(' ').slice(-2);
  return { name, city };
}

async function callApi(query) {
  const { name, city } = parseUserQuery(query);

  // Make the API call
  const response = await fetch('https://api.example.com', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ name, city }),
  });

  const data = await response.json();
  return data;
}

In your _chain definition, you can add a new step that calls this function and adds the result to the context:

const _chain = RunnableSequence.from([
  {
    question: _standaloneQuestionChain,
    original_input: new RunnablePassthrough(),
  },
  async (context) => {
    const apiData = await callApi(context.question);
    return { ...context, apiData };
  },
  _answerChain,
]);

In this example, callApi is a function that makes a POST request to an API with parameters extracted from the user's query. The result of the API call is added to the context under the apiData key. This data can then be used in the _answerChain to generate the AI's response.

Please note that this is a simplified example and you may need to adjust the parseUserQuery and callApi functions based on the actual format of the user's query and the requirements of the API you're calling.

I hope this helps! If you have any further questions, feel free to ask.

Sources

• libs/langchain-community/src/chat_models/cloudflare_workersai.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Jkrish1011]

@dosu-bot,
I tried all the options. It's not working.

Issues - I want the LLM to understand the user query and extract the name and city from the prompt. Then make the API calls and respond with the result of the api as it's context of answer.

Current state - The api calls are being made, but the input parameters required are not extracted properly thus the LLM outputing gibberish.

[dosubot[bot]]

🤖

I understand your issue. You want to extract specific parameters from the user's query, use these parameters to make an API call, and then use the API response as context for the model's answer. Here's how you can achieve this:

First, you need to create a custom Runnable that will parse the user's query and extract the desired parameters. This Runnable can then be added to your RunnableSequence. Here's an example:

import { Runnable } from "@langchain/core/runnables";

class ParameterExtractor extends Runnable {
  constructor() {
    super();
  }

  async run(input: any) {
    // Assuming the input is a string, we can use regular expressions or other string parsing methods to extract the desired parameters.
    const nameMatch = input.match(/name is (\w+)/i);
    const cityMatch = input.match(/from (\w+)/i);

    const name = nameMatch ? nameMatch[1] : null;
    const city = cityMatch ? cityMatch[1] : null;

    return { name, city };
  }
}

// Then, you can add this to your RunnableSequence like so:

const parameterExtractionChain = RunnableSequence.from([
  new ParameterExtractor(),
  // ...other Runnables
]);

In this example, the ParameterExtractor Runnable is looking for phrases like "name is John" and "from New York" in the user's input. It then extracts the name and city from these phrases and returns them. You would need to adjust the regular expressions to match the specific format of your user's queries.

Next, you need to create a new tool that will handle the API call and provide the response to the agent. Here's an example:

import { Tool } from "@langchain/core/tools";
import axios from "axios";

class WeatherAPI extends Tool {
  async invoke(input: { name: string, city: string }) {
    // Make the API call
    const response = await axios.post(`https://your-api-url.com`, input);

    // Return the API response
    return response.data;
  }
}

// Add the new tool to the tools array
const tools = [
  new WeatherAPI(),
  // ...other tools
];

In this example, a new tool WeatherAPI is created. This tool has an invoke method that is called when the tool is used by the agent. The invoke method makes an API call to your API and returns the response. The new tool is then added to the tools array so it can be used by the agent.

Please replace https://your-api-url.com with your actual API URL. Also, make sure that the invoke method accepts an object with name and city properties, as these are the parameters that will be passed from the ParameterExtractor Runnable.

I hope this helps! If you have any other questions, feel free to ask.

Sources

• examples/src/tools/exa_search.ts
• examples/src/guides/expression_language/cookbook_conversational_retrieval.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Jkrish1011]

@dosu-bot,
In the example shown, why use the regular expression? Can't we use the LLM itself to extract the name and city from the prompt. ? Like using a separate RunnableSequence with a question prompt and answer template to understand the user's input and get the information we require out of it?

[dosubot[bot]]

🤖

To extract specific parameters like name and city from a user's query, you can use a Named Entity Recognition (NER) model. In JavaScript, libraries like compromise or natural can be used for this purpose. Here's a simplified example of how you can do this:

import nlp from 'compromise';

const extractParameters = (input: string) => {
  const doc = nlp(input);
  const name = doc.people().text();
  const city = doc.cities().text();
  return { name, city };
};

const { name, city } = extractParameters(conversationalRetrievalQAChainInput.question);

In this example, compromise is used to extract the name and city from the user's query. The people().text() and cities().text() methods are used to extract the name and city respectively.

Once you have the name and city, you can pass them to your API call function. Here's an example of how you can do this:

const apiResponse = await makeApiCall(name, city);

In this example, makeApiCall is a hypothetical function that makes the API call using the name and city as parameters. You would replace this with your actual function that makes the API call.

Finally, you can use the result of the API call as the context for the language model. Here's an example of how you can do this:

const result = await conversationalRetrievalQAChain.invoke({
  question: conversationalRetrievalQAChainInput.question,
  chat_history: conversationalRetrievalQAChainInput.chat_history,
  context: apiResponse,  // Use the API response as the context
});

In this example, conversationalRetrievalQAChain.invoke is a hypothetical method that invokes the language model. The context parameter is set to the result of the API call.

Please note that this is a simplified example and you may need to adjust it according to your specific requirements and the structure of your API response. Also, the NER model may not always correctly extract the parameters, especially if the input text is complex or ambiguous. You may need to train the NER model with more data or use a more advanced NER model for better accuracy.

Regarding your question about using a separate RunnableSequence with a question prompt and answer template to understand the user's input and get the information we require out of it, yes, you can do that. However, it might be more efficient to use a NER model to extract the parameters directly from the user's query, as shown in the example above.

I hope this helps! If you have any more questions, feel free to ask.

Sources

• examples/src/guides/expression_language/cookbook_conversational_retrieval.ts
• examples/src/use_cases/sql/query_checking.ts

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[dosubot[bot]]

🤖

In the LangChain JavaScript framework, the Runnable sequence is designed to handle asynchronous operations. The invoke method of a Runnable returns a Promise, which means it's already set up to handle asynchronous operations like API calls.

In your provided context, the createOpenAIFnRunnable function returns a Runnable sequence that includes the llm (language model), which is bound with the llmKwargs (parameters for the language model). This sequence is then piped through the outputParser.

When you call the invoke method on the Runnable returned by createOpenAIFnRunnable, it will wait for the llm to complete its operation (which could be an asynchronous API call) before it proceeds to the outputParser.

Here's the relevant code:

const llmWithKwargs = (llm as Runnable).bind(llmKwargs);
return prompt.pipe(llmWithKwargs).pipe(outputParser);

In this code, llmWithKwargs is a Runnable that represents the language model operation. The pipe method is used to create a sequence of operations. The invoke method on the final Runnable will wait for each operation in the sequence to complete before moving on to the next one.

So, to ensure that your RunnableSequence waits for the asynchronous callApi function to complete and return data before the language model proceeds to generate a reply, you just need to make sure that the callApi function is part of the Runnable sequence. If it's the llm operation, then the RunnableSequence will already wait for it to complete before proceeding.

You can find more details about this in the langchain/src/chains/openai_functions/base.ts file.

As for the RunnableSequence.from method, it is implemented in the langchainjs codebase as a static method that takes an array of RunnableLike objects and an optional name as arguments. It returns a new instance of RunnableSequence. This method creates a new RunnableSequence instance by taking the first RunnableLike object as the first runnable, the last RunnableLike object as the last runnable, and all the RunnableLike objects in between as the middle runnables. It uses the _coerceToRunnable function to convert RunnableLike objects into Runnable objects.

Yes, it does support asynchronous operations. The Runnable class, which RunnableSequence extends, has methods like invoke, transform, stream, and batch that return Promises or AsyncGenerators, indicating that they are designed to handle asynchronous operations.

You can find more details about this in the langchain-core/src/runnables/base.ts file.

Sources

• langchain/src/chains/openai_functions/base.ts
• langchain-core/src/runnables/base.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.