Multi turn conversation with user

[HGF1KOR]

I have a question: What are the possible ways to enable multi-turn conversations between a user and a multi-agent system, such that the system is aware of the previous questions asked?

[withsmilo]

@HGF1KOR
It's definitely possible with AutoGen. You can refer various tutorials and how to manage team state in official doc.

[HGF1KOR]

can you share any examples, i went through most of the documentation, i couldnot find one .
The best i got was to add memory to the agent , so that it remembers the past conversation .

[SongChiYoung]

AutoGen provides built-in support for multi-turn conversations by automatically preserving message history.

When using a UserProxyAgent, the LLM receives all prior messages in the conversation thread. So no additional configuration is needed.
👉 See the Human-in-the-loop tutorial for a minimal example. : https://microsoft.github.io/autogen/stable//user-guide/agentchat-user-guide/tutorial/human-in-the-loop.html

If you’re using a multi-agent structure and want to share context across different teams or sessions, then yes -you’re right- that’s where the Memory module becomes important:

• AutoGen’s Memory module allows storing and retrieving conversation history.
• It can be shared across agents to maintain context continuity beyond a single chat loop.

Alternatively, if you want to enable multi-team coordination without using Memory, you can use the SocietyOfMindAgent.

The SocietyOfMindAgent encapsulates an inner team but still behaves as a regular agent.
This allows you to include it in an outer team structure -for example, alongside a UserProxyAgent- and enables chatting with multiple teams as a unified interface.

[HGF1KOR]

@SongChiYoung Thanks for providing examples along with all the possibilities of how to enable multi-turn conversation.

However, I still see several issues with these designs:

1. Human-in-the-loop – We can't really call it true multi-turn, as we don’t have low-level control over the previous messages exchanged by the multi-agent system. Suppose we want to remove or delete some parts of the chat (to reduce token usage) — I don't think we have that kind of control in the human-in-the-loop setup.

2. Memory Module – Yes, we are currently using this, and it works fine sometimes. But other times, it fails to catch follow-up questions properly. So, I believe this also isn’t a complete solution for multi-turn conversations.

Don't we have any options similar to https://openai.github.io/openai-agents-python/results/ result.to_input_list() in the OpenAI Agents SDK? That approach feels more intuitive for handling multi-turn interactions, as it allows direct manipulation of the conversation list — offering fine-grained control over what to keep or discard.

[SongChiYoung]

@HGF1KOR

Great questions
And I think what you’re asking points directly toward the model_context feature in AutoGen.

By default, AutoGen does offer low-level control over what context is passed to the LLM via the model_context argument.
You can read more about it here:
👉 model_context – Core User Guide : https://microsoft.github.io/autogen/dev//user-guide/core-user-guide/components/model-context.html

🧩 Want full control like result.to_input_list() in OpenAI Agents?

Then you might want to try a user-defined model_context, where you can implement your own filtering, slicing, rewriting, or summarization strategies.
For that, I recommend this 3rd-party extension I built:
👉 autogen-contextplus : https://github.com/SongChiYoung/autogen-contextplus

It lets you plug in custom logic into the context selection flow, giving you the kind of precise control you’re describing.

Let me know if you’d like an example - happy to share a working setup!

[HGF1KOR]

@SongChiYoung

Sorry to bother you again, but I have a few doubts.

If I understand model_context correctly, it's not meant for a Selector Group Chat or a multi-agent system. It seems to work with a single agent or model client—that’s what I gathered from the example. Please correct me if I’m wrong.

With the current output of the Selector Group Chat (i.e., TaskResult), we’re only using that to create a condensed message and storing it in memory. However, this TaskResult should ideally allow us to also include the user message and pass it back to the Selector Group Chat for multi-turn interactions—something along those lines.

In short, what I’m trying to say is that we should be able to pass a format like this to a multi-agent system as well:

messages = [
    {"user": "question", 
     "assistant": "thoughts or exchanged messages", 
     "assistant": "response to 1st question"},
    
    {"user": "2nd question"}
]

This format should align with what most LLM providers (like Claude, OpenAI, etc.) use. If mapped to a model client, it should handle follow-up questions much better than simply relying on memory attached to a Multi Agent system.

[SongChiYoung]

@HGF1KOR
I think it might be more appropriate to have a separate model_context for each agent.
Each agent may have different roles or purposes, and therefore might need different ways of summarizing or managing its context (I’ll call this “summarization” for now).

And you’re right, SelectorGroupChat does use history when selecting the next agent, and that history also needs to be summarized or managed properly.
There’s actually an open issue that’s currently addressing this:
👉 Issue #6301

From what I understood, it sounds like you’re suggesting having a separate context system for each SDK. Personally, I feel that might lead to fragmentation, since we already have a shared model_context structure that can be customized for different use cases.

Let me know if I’ve misunderstood anything — I’d love to hear your thoughts!

---

Also, just to clarify one point — I think there might be a small misunderstanding.
model_context isn’t just about Memory. As you mentioned, it also controls how the summarized messages are actually passed to the LLM itself. 😊

[HGF1KOR]

@SongChiYoung Thanks i will try and check whether its working or not :)

[Ethan0456]

Based on other messages in this discussion, I think you need the following two functionalities:

1. Multi-turn conversation with user, with previous chat history retained.
2. Ability to manipulate messages in the chat history (before follow-up) to manage token limits and control the context passed to agents.

Here are a couple of approaches that might help:

1. Simple Follow-up with Context

In my experience, SelectorGroupChat usually preserves the chat state across .run or .run_stream calls. So if you're doing something like this:

team.run_stream(task="first task")

# Output: TaskResult with some context

team.run_stream(task="follow up")  # You just provide the follow-up user message as the new task

...the context is typically preserved internally, and the agents can continue the conversation just fine.

You also mentioned in one of your earlier messages that you're using memory to retain context—if so, you're probably already covered on this front. But this built-in behavior can be handy even without custom memory logic.

2. Manual Manipulation of Messages (for token savings and fine-grained control)

This is more useful when you want tighter control over the context—like pruning unnecessary messages, summarizing older content, or just being more token-efficient.

As @SongChiYoung suggested earlier, this points to using the model_context. AssistantAgent already supports this, and we might consider extending similar control to SelectorGroupChat.

In the meantime, there's a workaround using save_state and load_state. Here's a step-by-step you can follow:

team.run_stream(task="first task")

state = await team.save_state()

# The `state` object now contains a detailed snapshot of the chat history:
# {
#   'type': 'TeamState',
#   'version': '1.0.0',
#   'agent_states': {
#     'agent_1': {
#       'type': 'ChatAgentContainerState',
#       'version': '1.0.0',
#       'agent_state': {
#         'type': 'AssistantAgentState',
#         'version': '1.0.0',
#         'llm_context': {
#           'messages': [
#             {'type': 'UserMessage', 'source': 'user', 'content': 'Let's have a conversation around quantum physics.'},
#             {'type': 'UserMessage', 'source': 'agent_3', 'content': '...'},
#             {'type': 'AssistantMessage', 'source': 'agent_2', 'content': '...'}
#           ]
#         }
#       }
#     }
#   }
# }

# Now if you're planning to continue the session later, and want to reduce token usage...

# Step 1: Reset the team if you’re not disposing the running process
await team.reset()

# Step 2: Modify the state however you need — trim messages, summarize parts, etc.
# You can also save this to disk if needed:
# with open("team_state.json", "w") as f:
#     json.dump(state, f)

# Step 3: Later (after N minutes/hours), when the user sends a follow-up:
# - Re-initialize your agents and team
team = ...  # however you originally constructed it

# - Load the modified state back in
await team.load_state(state)  # or load from disk

# Step 4: Start the next task using the follow-up
await team.run_stream(task="user follow-up")

This gives you full control over what the agents see in their context window, and is super helpful if you’re running into token limits or want to be selective about historical messages.

Let me know if I misunderstood any part of your question or if there's a specific detail you'd like me to revisit.

[HGF1KOR]

@Ethan0456 Thanks , i will check if this works
just few more doubts ,If the team preserves chat state and supose it hits token limit , then what happens ( how is it handled ) ?

[Ethan0456]

To support this, we're introducing a model_context parameter to SelectorGroupChatManager. This allows you to implement custom logic—such as retrieving the last n messages or the last n messages from a specific agent—and pass it to the selector group chat. You can also customize the prompt variable to guide the model in selecting the appropriate next speaker.

In the meantime, you can update the state using your own logic as outlined in Step 2.

[SongChiYoung]

@HGF1KOR @Ethan0456
And If you using 3-Party extention autogen-contextplus, this type of context has trigger -TokenUsageCondition.

https://github.com/SongChiYoung/autogen-contextplus/blob/1ec2bf5452573a46652bbec1e67be35e35e6995e/autogen_contextplus/conditions/_conditions.py#L161

class TokenUsageCondition(ContextPlusCondition, Component[TokenUsageConditionConfig]):
    """(Experimental) A token based chat completion context maintains a view of the context up to a token limit.

    .. note::

        Added in v0.5.10(at AutoGen). This is an experimental component and may change in the future.

    Args:
        model_client (ChatCompletionClient): The model client to use for token counting.
            The model client must implement the :meth:`~autogen_core.models.ChatCompletionClient.count_tokens`
            and :meth:`~autogen_core.models.ChatCompletionClient.remaining_tokens` methods.
        token_limit (int | None): The maximum number of tokens to keep in the context
            using the :meth:`~autogen_core.models.ChatCompletionClient.count_tokens` method.
            If None, the context will be limited by the model client using the
            :meth:`~autogen_core.models.ChatCompletionClient.remaining_tokens` method.
        tool_schema (List[ToolSchema] | None): A list of tool schema to use in the context.
        initial_messages (List[LLMMessage] | None): A list of initial messages to include in the context.
    """

You could try it when using Agent now.

Check the example : https://github.com/SongChiYoung/autogen-contextplus/blob/main/examples/simple_example.py

And, after merge #6330, you could using it at SelectorGroupChat too.