Dealing with Streaming Chat Completion: possible blocking function?

[HunterProduction]

Hello!

Still me, from the #358, here for something else!
I am using your very good API port to realize an higher level wrapper library to implement AI-based feature in my projects. When testing the Streamed Chat Completion feature, I'm experiencing a kind of freeze in the app execution, of unpredictable duration (sometimes is a really long delay).

I took a snapshot of the profiler which shows that spike of about 60 seconds:

Now I'll describe my implementation so that you can have a context.

I have a sort of "service" class I call Processor that directly manages your OpenAI Client. Here there are these 2 methods:

public async Task<(string, AudioClip)> StreamRespondTo(string speakerName, string message, Action<ChatResponse> partialResponseHandler)
{
    Conversation.AppendMessage(new Message(Role.User, message, speakerName));
    var chatRequest = new ChatRequest(Conversation.Messages, model: Model, audioConfig: new AudioConfig(Voice, AudioFormat.Pcm16));
    var response = await StreamChatCompletion(chatRequest, partialResponseHandler);

    return (response, response.AudioOutput.AudioClip);
}

protected virtual async Task<Message> StreamChatCompletion(ChatRequest chatRequest, Action<ChatResponse> partialResultHandler)
{
    if (!Authenticated)
    {
        Log($"[{GetType().Name}] LanguageProcessor not ready");
        return new Message(Role.Assistant, "");
    }

    Log($"[{GetType().Name}] Responding...");

    var start = DateTime.Now;

    var response = await Api.ChatEndpoint.StreamCompletionAsync(chatRequest, partialResultHandler);

    var latency = DateTime.Now.Subtract(start).TotalMilliseconds;

    var choice = response.FirstChoice.Message;

    Log($"[{GetType().Name}] Request latency: {latency:0.0}ms | Finish reason: {response.FirstChoice.FinishReason}\nResponse: {choice}");

    // Add new response to the history
    Conversation.AppendMessage(choice);

    return choice;
}

Then, in my desired MonoBehaviour I instantiate that processor and use it at higher level to perform the specific task (example: SimpleChatAssistant MonoBehaviour uses an AIConversationalProcessor to retrieve an ai-based response to a user message). Here is the method in the MonoBehaviour:

public async void StreamRespond()
{
    var message = inputTextField.text;
    onRequestReceived?.Invoke(message);

    var (answer, audioClip) = await _languageProcessor.StreamRespondTo("User", message, partialResponse =>
    {
        var choice = partialResponse?.FirstChoice;
        if (choice?.Delta == null)
        {
            return;
        }
        answerTextField.text += choice.Delta.ToString();

        // Audio Section: null reference exception
        /*
        var samples = choice.Delta.AudioOutput?.AudioSamples;
        if (samples != null)
        {
            isAudioResponseInProgress = true;
            streamAudioSource.BufferCallback(samples);
        }
        else
        {
            isAudioResponseInProgress = false;
        }
        */
    });

    onRequestCompleted?.Invoke(answer);
    answerTextField.text = answer;
}

Now, I have the same twin methods using non-streamed chat completion, and the execution does not causes that freeze. Am I missing something? I wonder if there could be any possible blocking function call in the asynchronous pipeline of the streamed chat completion.

To mention another problem I'm facing, as you can see in the commented section of the partial response handler, I tried mimicing your audio delta management as you did in the RealtimeBehaviour sample using your custom StreamAudioSource, but the API throws me a sequence of null pointer exceptions even with those explicit null checks. I think I am wrong on something, just let me know if you have an answer for this.

[StephenHodgson]

@HunterProduction can you dig into that profiler hierarchy and show me exactly what is taking so long?

you only show the top level item. Feel free to DM me the profiler data on Discord as well

[HunterProduction]

@StephenHodgson I leave you the screenshot here if it's ok for you (I don't directly use Discord while working on this activity).

As you can see I didn't showed you the full hierarchy before because it's not so talkative. It regards Coroutines but does not specify the source of it. Even with Raw Hierarchy mode I can't see nothing important rather than those explicit 300+ GC allocations, which seems a bit anomalous considering they are allocating around 70Kb in a peak of 4.6 secs.

I tried some debug with breakpoints to dig a little bit lower in the call stack, but as far as I could see I didn't entered any explicit Coroutine handling or any accessible synchronous code section that causes a freeze.

[StephenHodgson]

Yeah from the screenshot I'm still not entirely sure what that is from exactly.

I use this library all the time but don't see this kind of behavior.

Are you using chat, real-time, or assistant endpoints?

Can you also try running just the sample scenes and profile those and see if you have the same problem?

[StephenHodgson]

@HunterProduction I think I found the cause, I believe it is a regression from the async or rest utils library.

I'll look into it and see if I can resolve this in the base library

[StephenHodgson]

@HunterProduction still looking into this. but I am unable to devote as much time as I'd like to it.

I was able to find the source of the bottle neck but no solution yet.