Can I call a custom tool from ConversationalRetrievalQAChain?

[Originn]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

export const makeChain = (vectorstore: PineconeStore, onTokenStream: (token: string) => void) => {
  const streamingModel = new OpenAI({
    streaming: true,
    modelName: MODEL_NAME,
    temperature: TEMPRATURE,
    callbacks: [
      {
        handleLLMNewToken: (token) => {
          onTokenStream(token); // Forward the streamed token to the front-end
        },
      },
    ],
  });

  // Non-streaming model setup
  const nonStreamingModel = new OpenAI({
    modelName: 'gpt-3.5-turbo',
    temperature: TEMPRATURE
  });

  const translationModel = new OpenAI({
    modelName: 'gpt-4',
    temperature: TEMPRATURE
  });

  function generateUniqueId(): string {
    return uuidv4();
  }
  return {
    call: async (question: string, Documents: MyDocument[], roomId: string, session: any) => {
      const userEmail = session?.user?.email || 'unknown';
      const qaId = generateUniqueId();

      if (!roomMemories[roomId]) {
        roomMemories[roomId] = new BufferMemory({
          memoryKey: "chat_history",
          inputKey: "question",
          outputKey: "text",
        });
      }
      let chat_history = roomMemories[roomId];
      const language = await detectLanguageWithOpenAI(question, nonStreamingModel);

      if (language !== 'English') {
        question = await translateToEnglish(question, translationModel);
      }

      const formattedPrompt = QA_PROMPT
        .replace('{language}', language);
      
      // Since the first user question in not going through the CONDENSE_PROMPT and therefor abbriviations cannot be changed, i have added a generic user message.
      if ((chat_history.chatHistory as any).messages.length === 0) {
        const initialHumanMessage = new HumanMessage({
          content: "Hi",
          name: "Human",
      });
      chat_history.chatHistory.addMessage(initialHumanMessage); // Hypothetical method to add message
      }

      const customRetriever = new CustomRetriever(vectorstore);
      // Use the specific room memory for the chain
      const chain = ConversationalRetrievalQAChain.fromLLM(
        streamingModel,
        customRetriever,
        {
          memory: chat_history,
          questionGeneratorChainOptions: {
            llm: nonStreamingModel,
          },
          qaTemplate: formattedPrompt,
          questionGeneratorTemplate: CONDENSE_PROMPT,
          returnSourceDocuments: true,
          verbose: true
        }
      );

      const responseText = (await (async () => {
        const response = await chain.invoke({
            question: question,
        });
        return {
          text: response.text,
          sourceDocuments: response.sourceDocuments
        };
      })());


My custom tool looks like this

```typescript
import { Tool, ToolParams } from "@langchain/core/tools"; // Adjust the import path as needed
import axios from 'axios';

export class CustomCalculator extends Tool {
    static lc_name() {
        return "CustomCalculator";
    }

    name = "CustomCalculator"; // Adjusted to match the pattern

    description = "Performs the API operation Calculate() command based on the provided operation number.";

    constructor(config?: ToolParams) {
        super(config);
    }

    async _call(operationIndex : any) {
        const opIndex = parseInt(operationIndex);
        if (isNaN(opIndex)) {
            throw new Error('Operation number must be a valid integer');
        }
    
        const url = `http://localhost:8080?op=${opIndex}&action=calculate`;
    
        try {
            const response = await axios.get(url);
            console.log(response.data); 
            return response.data;
        } catch (error) {
            console.error('Error making the request:', error);
            throw new Error('Failed to perform calculation');
        }
    }
}

### Description

I am trying to find a way to invoke or pass the tools objects after I add `const tools = [new CustomCalculator];`

so that the LLM will be able to decide if to call it or not, but I need my implementation to stay as is, preferably to pass it to ConversationalRetrievalQAChain. Is this possible? 

### System Info

platform (windows)
Node version v20.9.0
npm version 10.1.0

[dosubot[bot]]

Hey there @Originn! Great to see you diving into another intriguing challenge with LangChain JS. Hope you've been doing awesome since our last chat! 🚀😄

Based on the information provided, it seems you want to integrate a custom tool into the ConversationalRetrievalQAChain in LangChain JS. This is indeed possible, and you can follow the pattern demonstrated in the provided context with the DynamicTool and DynamicStructuredTool classes.

Here's a step-by-step guide based on the context:

1. Define Your Custom Tool: Create a new instance of DynamicTool or DynamicStructuredTool depending on whether your tool requires structured input. For a simple tool that doesn't require structured input, use DynamicTool. For tools that require structured input (like the "random-number-generator"), use DynamicStructuredTool.

2. Tool Configuration:

   • For DynamicTool, you need to provide a name, a description, and a func which is the asynchronous function that will be executed when the tool is called.
   • For DynamicStructuredTool, in addition to name, description, and func, you also need to define a schema using zod for input validation.

3. Implement the Tool Function: The func property is an asynchronous function that implements the logic of your tool. It receives input (if any) and returns a string. For DynamicStructuredTool, the input will be validated against the provided schema.

4. Add the Tool to the Tools Array: Once your custom tool is defined, add it to the tools array. This array is then passed to the createOpenAIFunctionsAgent function, which integrates your tools with the agent.

5. Create the Agent: Use the createOpenAIFunctionsAgent function to create an agent. This function requires the LLM instance (llm), the tools array (tools), and a prompt template (prompt). The prompt template can be customized or pulled from an external source as shown.

6. Execute the Agent with Your Tool: Use the AgentExecutor class to execute the agent. Create an instance of AgentExecutor by passing an object with the agent and tools properties, and optionally verbose for logging. Then, invoke the invoke method on the AgentExecutor instance with the desired input to use your custom tool.

Here's a simplified example of adding a custom tool that returns the current date:

const currentDateTool = new DynamicTool({
  name: "current-date",
  description: "Gets the current date.",
  func: async () => new Date().toDateString(),
});

// Add the currentDateTool to the tools array
tools.push(currentDateTool);

// Then follow the same steps to create the agent and execute it

This example demonstrates how to define a simple tool that returns the current date as a string. You can adapt this pattern to integrate any custom tool into the ConversationalRetrievalQAChain by defining the tool, adding it to the tools array, and using it through the agent execution process as shown in the context.

For more details, you can refer to the source code of the DynamicTool and DynamicStructuredTool classes in the langchainjs repository.

I hope this helps! If you have any further questions, feel free to ask.

Sources

• Do Conversational Retrieval QA Chain support custom tool?
• ConversationalRetrievalQAChain using Tools.
• examples/src/chains/conversational_qa_built_in_memory_legacy.ts
• examples/src/agents/custom_tool.ts
• examples/src/retrievers/custom.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Originn]

@dosu-bot I still don't understand how your answer fulfill the question of how to call the tool from ConversationalRetrievalQAChain as you suggested to use AgentExecutor

[dosubot[bot]]

To integrate your CustomCalculator tool into the ConversationalRetrievalQAChain and call it using AgentExecutor, you need to follow these steps:

1. Define Your Custom Tool: You've already done this with your CustomCalculator class.

2. Create an Agent with Tools: Use createOpenAIFunctionsAgent to create an agent that includes your custom tools. This agent is configured with the LLM (Language Model) and the tools you've defined.

3. Use AgentExecutor for Invocation: The AgentExecutor is then used to manage the invocation of the agent, passing in user inputs and receiving outputs.

Here's a simplified example of how you can achieve this:

import { createOpenAIFunctionsAgent, AgentExecutor } from "@langchain/core/agents";
import { DynamicTool } from "@langchain/core/tools";
import { CustomCalculator } from "./CustomCalculator"; // Import your custom tool

// Create an instance of your custom tool
const customCalculator = new CustomCalculator();

// Create an agent with your custom tool
const agent = createOpenAIFunctionsAgent({
  llm: streamingModel, // Your language model
  tools: [customCalculator], // Your custom tool
  prompt: "Perform a calculation using the CustomCalculator tool.", // Your prompt
});

// Create an instance of AgentExecutor with your agent and custom tool
const agentExecutor = new AgentExecutor({
  agent,
  tools: [customCalculator],
  verbose: true,
});

// Now you can use the AgentExecutor to invoke your custom tool
const result = await agentExecutor.invoke({ input: "Calculate the operation with index 5" });
console.log(result);

In this example, createOpenAIFunctionsAgent is used to create an agent that includes your CustomCalculator tool. The AgentExecutor is then used to manage the invocation of the agent, passing in user inputs and receiving outputs.

Please note that you need to adjust the prompt and input to match your specific use case. The prompt should be a string that instructs the language model to use your custom tool, and the input should be a string that describes the operation you want to perform with your custom tool.

Regarding the integration of the AgentExecutor into the ConversationalRetrievalQAChain, you would need to adjust the makeChain function to include the AgentExecutor and your custom tool. However, without more details about your specific use case, it's hard to provide a more concrete example. If you could provide more information about how you want to use your custom tool within the ConversationalRetrievalQAChain, I could provide a more detailed guide.

For more information about the AgentExecutor and how to use it with custom tools, you can refer to the AgentExecutor documentation and the custom tool example in the LangChain JS repository.

Sources

• examples/src/agents/custom_tool.ts
• langchain/src/agents/executor.ts
• examples/src/use_cases/tool_use/agents.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Originn]

@dosu-bot the way i want to implement it is that I would like the user to hold the conversation using the existing ConversationalRetrievalQAChain with the chat history, but I want it to also detect when the user, during the conversation want to perform a function that calculate something, then the LLM will recognize it and call the customCalculator function and return some text that it completed the operation. Then the user can continue with the chat or ask another function calling....

[dosubot[bot]]

To integrate your CustomCalculator tool into the ConversationalRetrievalQAChain, you need to modify the chain to recognize when a calculation operation is required and then call your custom tool. Here's a step-by-step guide on how to achieve this:

1. Define a function to detect calculation operations: You need a function that can analyze a user's question and determine if it implies a calculation. This function could use regular expressions to identify patterns that suggest a calculation is needed. Here's a simple example:

function isCalculation(question: string): boolean {
  // This is a very basic check. You might need a more sophisticated method.
  return /\d+/.test(question) && /[\+\-\*\/]/.test(question);
}

2. Modify the ConversationalRetrievalQAChain to call your custom tool: You need to modify the call method of your ConversationalRetrievalQAChain to call your CustomCalculator tool when a calculation operation is detected. Here's how you can do it:

// Import your custom tool
import { CustomCalculator } from './path/to/your/custom_calculator';

// Instantiate your custom tool
const customCalculator = new CustomCalculator();

// Modify the `call` method
export const makeChain = (vectorstore: PineconeStore, onTokenStream: (token: string) => void) => {
  // ... existing code ...

  return {
    call: async (question: string, Documents: MyDocument[], roomId: string, session: any) => {
      // ... existing code ...

      // Check if the question implies a calculation
      if (isCalculation(question)) {
        // If it does, call your custom tool and return its result
        const result = await customCalculator._call(question);
        return {
          text: `The result of the calculation is ${result}.`,
          sourceDocuments: []
        };
      }

      // ... existing code ...
    }
  };
};

In this modification, the call method first checks if the user's question implies a calculation by calling the isCalculation function. If it does, it calls your CustomCalculator tool with the question as the operation index and returns its result. If it doesn't, it proceeds with the existing logic.

Please note that this is a basic example and might not cover all your needs. You might need to adjust the isCalculation function to accurately detect calculation operations based on your requirements. Also, you might need to handle errors and edge cases in the call method when calling your custom tool.

I hope this helps! If you have any further questions, feel free to ask.

Sources

• examples/src/models/chat/integration_togetherai_tools.ts
• langchain/src/chains/tests/conversational_retrieval_chain.int.test.ts
• examples/src/agents/custom_tool.ts

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[Originn]

My temporary solution is to call agentExecutor prior to ConversationalRetrievalQAChain and set a CustomTemplate to return NO if no tool was executed. If the output from agentExecutor is NO then ConversationalRetrievalQAChain is called, but if something else then do some logic with the agentExecutor response.

If there is a better solution please let me know.

Cheers.