Python: Why eliminate duplicate AG-UI messages?

[SogoKato]

Within the function converting AG-UI messages to Microsoft Agent Framework messages, there is logic that removes duplicate messages contained within the request body.

I believe this implementation carries the risk of removing simple messages like "yes," potentially causing the conversation history to be improperly rewritten.

I would like to understand the intent behind this implementation.

agent-framework/python/packages/ag-ui/agent_framework_ag_ui/_message_adapters.py

Lines 471 to 489 in fcdaaff

	def normalize_agui_input_messages(
	messages: list[dict[str, Any]],
	*,
	sanitize_tool_history: bool = True,
	) -> tuple[list[Message], list[dict[str, Any]]]:
	"""Normalize raw AG-UI messages into provider and snapshot formats.
	Args:
	messages: Raw AG-UI messages.
	sanitize_tool_history: Apply agent-run specific tool history repair logic.
	Keep enabled for standard agent runs; disable for native workflow runs
	where pending-request responses must come explicitly from interrupt resume.
	"""
	provider_messages = agui_messages_to_agent_framework(messages)
	if sanitize_tool_history:
	provider_messages = _sanitize_tool_history(provider_messages)
	provider_messages = _deduplicate_messages(provider_messages)
	snapshot_messages = agui_messages_to_snapshot_format(messages)
	return provider_messages, snapshot_messages

agent-framework/python/packages/ag-ui/agent_framework_ag_ui/_message_adapters.py

Lines 200 to 255 in fcdaaff

	def _deduplicate_messages(messages: list[Message]) -> list[Message]:
	"""Remove duplicate messages while preserving order."""
	seen_keys: dict[Any, int] = {}
	unique_messages: list[Message] = []
	for idx, msg in enumerate(messages):
	role_value = get_role_value(msg)
	if role_value == "tool" and msg.contents and msg.contents[0].type == "function_result":
	call_id = str(msg.contents[0].call_id)
	key: Any = (role_value, call_id)
	if key in seen_keys:
	existing_idx = seen_keys[key]
	existing_msg = unique_messages[existing_idx]
	existing_result = None
	if existing_msg.contents and existing_msg.contents[0].type == "function_result":
	existing_result = existing_msg.contents[0].result
	new_result = msg.contents[0].result
	if (not existing_result or existing_result == "") and new_result:
	logger.info(f"Replacing empty tool result at index {existing_idx} with data from index {idx}")
	unique_messages[existing_idx] = msg
	else:
	logger.info(f"Skipping duplicate tool result at index {idx}: call_id={call_id}")
	continue
	seen_keys[key] = len(unique_messages)
	unique_messages.append(msg)
	elif role_value == "assistant" and msg.contents and any(c.type == "function_call" for c in msg.contents):
	tool_call_ids = tuple(
	sorted(str(c.call_id) for c in msg.contents if c.type == "function_call" and c.call_id)
	)
	key = (role_value, tool_call_ids)
	if key in seen_keys:
	logger.info(f"Skipping duplicate assistant tool call at index {idx}")
	continue
	seen_keys[key] = len(unique_messages)
	unique_messages.append(msg)
	else:
	content_str = str([str(c) for c in msg.contents]) if msg.contents else ""
	key = (role_value, hash(content_str))
	if key in seen_keys:
	logger.info(f"Skipping duplicate message at index {idx}: role={role_value}")
	continue
	seen_keys[key] = len(unique_messages)
	unique_messages.append(msg)
	return unique_messages

[moonbox3]

Good catch. The deduplication exists because AG-UI clients send the full conversation history on every request, which can produce duplicates — particularly around tool call flows (duplicate tool results with the same call_id, duplicate assistant tool call messages). The first two branches of _deduplicate_messages handle those specific cases and are necessary to avoid API errors.

The catch-all branch (lines 244-253) that deduplicates by (role, hash(content)) is where the risk you identified exists — a user saying "yes" twice in a conversation would have the second one dropped. We should scope that branch more tightly or remove it. I'll create an issue to track this.

[moonbox3]

Tracking in #4682. Closing this item.