How to test custom OpenAI tools with FakeStreamingChatModel

[JakubKontra]

const fakeLLM = new FakeStreamingChatModel({
        responses: [
          new AIMessage({
            content: 'Hello',
          }),
        ],
      })

      const conversation = [
        new SystemMessage("Age: 19"),
        new AIMessage(""),
        new HumanMessage('I want to change my age from 19 to 25'),
      ]

      let tools: StructuredTool[] = []
      tools = [changeNameTool]
      const model = fakeLLM
      const ret = await model.bindTools?.(tools).invoke(conversation)

Description

I'm trying to invoke a tool on a test input via FakeStreamingChatModel. Is it possible to set the required properties to trigger the tool call? Thanks.

System Info

windows

[dosubot[bot]]

To test custom OpenAI tools with FakeStreamingChatModel in TypeScript, you can follow the pattern used in the provided tests for ChatOpenAI. Here is an example of how you can set up and test custom tools using FakeStreamingChatModel:

import { test, expect, jest } from "@jest/globals";
import { AIMessage, HumanMessage, ToolMessage } from "@langchain/core/messages";
import { FakeStreamingChatModel } from "@langchain/core/models";
import { InMemoryCache } from "@langchain/core/caches";

test("Test FakeStreamingChatModel tool calling", async () => {
  function getCurrentWeather(location: string) {
    if (location.toLowerCase().includes("tokyo")) {
      return JSON.stringify({ location, temperature: "10", unit: "celsius" });
    } else if (location.toLowerCase().includes("san francisco")) {
      return JSON.stringify({
        location,
        temperature: "72",
        unit: "fahrenheit",
      });
    } else {
      return JSON.stringify({ location, temperature: "22", unit: "celsius" });
    }
  }

  const chat = new FakeStreamingChatModel({
    modelName: "gpt-3.5-turbo-1106",
    maxTokens: 128,
  }).bind({
    tools: [
      {
        type: "function",
        function: {
          name: "get_current_weather",
          description: "Get the current weather in a given location",
          parameters: {
            type: "object",
            properties: {
              location: {
                type: "string",
                description: "The city and state, e.g. San Francisco, CA",
              },
              unit: { type: "string", enum: ["celsius", "fahrenheit"] },
            },
            required: ["location"],
          },
        },
      },
    ],
    tool_choice: "auto",
  });

  const res = await chat.invoke([
    ["human", "What's the weather like in San Francisco, Tokyo, and Paris?"],
  ]);
  console.log(JSON.stringify(res));
  expect(res.additional_kwargs.tool_calls?.length).toBeGreaterThan(1);

  const toolMessages = res.additional_kwargs.tool_calls!.map(
    (toolCall) =>
      new ToolMessage({
        tool_call_id: toolCall.id,
        name: toolCall.function.name,
        content: getCurrentWeather(
          JSON.parse(toolCall.function.arguments).location
        ),
      })
  );

  const finalResponse = await chat.invoke([
    ["human", "What's the weather like in San Francisco, Tokyo, and Paris?"],
    res,
    ...toolMessages,
  ]);
  console.log(finalResponse);
});

This code sets up a test for FakeStreamingChatModel with a custom tool for getting the current weather. It follows the pattern used in the provided tests for ChatOpenAI, including invoking the model with a human message, processing tool calls, and verifying the results. The test also includes the use of jest for mocking and spying, similar to the tests in the langchain-azure-openai library ^[1][2].

Additionally, you can refer to the ChatOpenAI tests for more examples of how to handle tool calls and streaming responses. For instance, the ChatOpenAI tests demonstrate how to bind tools, handle streaming responses, and verify tool calls, which can be adapted for use with FakeStreamingChatModel ^[1].

To continue talking to Dosu, mention @dosu.

[JakubKontra]

@dosu

It doesn't work as expected, although I need langchain-openai instead of langchain-azure-openai.

[dosubot[bot]]

To test custom OpenAI tools with FakeStreamingChatModel using the langchain-openai library, you can follow this example:

import { test, expect, jest } from "@jest/globals";
import { AIMessage, HumanMessage, ToolMessage } from "@langchain/core/messages";
import { FakeStreamingChatModel } from "@langchain/core/models";
import { StructuredTool } from "@langchain/core/tools";

test("Test FakeStreamingChatModel tool calling", async () => {
  function changeNameTool(newName: string) {
    return JSON.stringify({ newName });
  }

  const fakeLLM = new FakeStreamingChatModel({
    responses: [
      new AIMessage({
        content: 'Hello',
      }),
    ],
  });

  const conversation = [
    new HumanMessage('I want to change my age from 19 to 25'),
  ];

  const tools: StructuredTool[] = [
    {
      type: "function",
      function: {
        name: "change_name",
        description: "Change the name of a person",
        parameters: {
          type: "object",
          properties: {
            newName: {
              type: "string",
              description: "The new name to set",
            },
          },
          required: ["newName"],
        },
      },
    },
  ];

  const model = fakeLLM.bind({ tools });

  const res = await model.invoke(conversation);
  console.log(JSON.stringify(res));
  expect(res.additional_kwargs.tool_calls?.length).toBeGreaterThan(0);

  const toolMessages = res.additional_kwargs.tool_calls!.map(
    (toolCall) =>
      new ToolMessage({
        tool_call_id: toolCall.id,
        name: toolCall.function.name,
        content: changeNameTool(
          JSON.parse(toolCall.function.arguments).newName
        ),
      })
  );

  const finalResponse = await model.invoke([
    ...conversation,
    res,
    ...toolMessages,
  ]);
  console.log(finalResponse);
});

This example sets up a test for FakeStreamingChatModel with a custom tool for changing a name. It follows the pattern used in the provided tests for ChatOpenAI, including invoking the model with a human message, processing tool calls, and verifying the results. The test also includes the use of jest for mocking and spying, similar to the tests in the langchain-openai library ^[1][2].

To continue talking to Dosu, mention @dosu.