Multi Agent Architectures - Agents as Tools vs Handoffs

Multi-agent systems are rapidly becoming a dominant architectural pattern in applied AI. In these systems, multiple agents coordinate and cooperate to solve a task. While these systems gradually move from simple chatbots to complex task execution engines, architecture decisions directly impact their capabilities, performance, and user experience.

In this article I'll do a deep dive between two common approaches of building multi agent systems, agent as tools and agent handoffs, comparing their pros and cons.

Before going through the technical aspects, I wanted to preface that I tested both approaches as part of a real application that I've developed using Pydantic AI: it is a resume builder and editor that allows users to input their professional experience, render it in different templates, and chat with AI agents that can provide feedback and directly edit the resume content. You can find all the details in the project's repository: ResuMate. While frameworks differ in implementation details, the architectural trade-offs discussed here apply regardless of tooling.

What is a multi-agent system?

Agents represent a structural evolution beyond traditional prompt-response LLM applications. In addition to the knowledge they can tap into, agents can be equipped with tools that allow them to interact with other systems (like databases, applications, etc). An agent is able to autonomously use these tools to fulfill the user request. This is a very powerful paradigm and any modern LLM platforms expose built-in capabilities such as web search or code execution, even if we might not know about them.

A multi-agent system is a system that is composed of multiple agents that interact with each other to achieve a common goal. Each agent has its own capabilities, knowledge, and tools, and they can communicate and collaborate to solve complex tasks that might be beyond the capabilities of a single agent.

Agents as Tools

An agent can be equipped with any tool that can be useful to fulfill the user request. A tool can be anything, from a calculator function to a way to access the web, and of course it can also be another agent. In this pattern the process flows like this:

1. The supervisor agent receives a request from the user.
2. The supervisor agent decides if it needs to call another agent to fulfill the request.
3. If the answer is yes, the supervisor will call the agent with the required context and wait for feedback. The agent that is called by the supervisor obviously can have its own tools and use them to perform the necessary actions.
4. When the supervisor receives the agent feedback it can decide whether to answer to user or repeat the process if more information or actions are needed.

This approach is particularly effective when:

• Tasks can be easily decomposed into sub-tasks that can be handled by specialized agents.
• The supervisor agent can be prompted in a way that allows it to collect all necessary information from the user in one go.
• The sub-agents have one dedicated task that does not require a lot of input from the user.

The main advantage of this approach is that it is generally easier to implement in most agentic frameworks, but the main downside is that context management can be an issue, since the supervisor is responsible for managing the conversation state and for routing the information between agents. If the supervisor is not prompted well or if the task requires a lot of back and forth between the user and the agents, this can lead to a lot of friction in the user experience.

Operationally, this approach sees the context accumulating in the supervisor agent, which can lead to higher costs if the conversation is long and complex, since the supervisor needs to process all the context every time it needs to call an agent. Additionally the supervisor can become the single point of failure of the system, but on the other hand it can also be easier to track and debug since all the information passes through it.

Some examples of applications that work well with this architecture are:

• A travel planning assistant, where the supervisor agent can call different agents for booking flights, hotels, and activities.

I initially prototyped my resume builder application using this architecture, but I found that the user experience was not very good, since the supervisor agent had to do a lot of back and forth with the user to collect all the necessary information and to route it to the right agents, which made the conversation feel very unnatural and clunky.

Agent Handoffs

In this paradigm there is no longer a strict supervisor–sub-agent hierarchy, but all agents are considered as first class citizens of a network. Each agent has access to its own tools and domain knowledge, and it is given the ability to perform a handoff: when an agent determines that a task is outside of its scope, it can transfer control, along with the relevant context, to a more appropriate agent.

The process typically looks like this:

1. An agent receives a request from the user.
2. The agent evaluates whether the request falls within its capabilities.
3. If it does, the agent proceeds and answers directly.
4. If it does not, the agent performs a handoff to the appropriate agent.
5. The receiving agent continues the interaction and provides the answer.

In this architecture there is no central orchestrator that owns the conversation state, but the responsibility of managing the context and executing the task shifts dynamically between agents. This approach works particularly well when:

• Tasks naturally span multiple domains.
• Agents need to ask follow up questions directly to the user.
• Conversations are long lived and evolve organically.

The main advantage of this approach is that it allows for a more natural conversational flow, since the user interacts directly with the agent that is currently handling the task. It can be better for complex, evolving tasks, where agents can progressively refine the interaction without waiting for approval from a supervisor.

The trade-off here is that it is generally more complex to implement, since you need to build a robust handoff mechanism (with routing logic and state tracking, and possibly context compression) and you need to be careful in defining what context gets passed during handoffs in order to avoid losing important information. Most of the popular agentic frameworks do not (yet) support this kind of architecture out of the box, so you will need to implement a lot of the logic by yourself.

Operationally, this approach can be more scalable since context can be managed by the active agent, potentially reducing costs, but it requires careful management to avoid losing important information during handoffs. Additionally, it can be more resilient since there is no single point of failure, but there is a risk of agent ping-pong if boundaries are not clear. It can also be harder to track and debug since the control is distributed across agents, but with good logging and monitoring it is still possible to have good observability.

Some examples of applications that work well with this architecture are:

• A customer support assistant where billing, technical, and account agents can transfer ownership of a conversation as the issue evolves.

For my resume builder application, I opted for the agent handoffs architecture, since I wanted to experiment with a more natural conversational flow and with agents that can ask follow up questions directly to the user. However, since my framework of choice did not support this architecture out of the box, I had to implement a lot of the handoff logic by myself, which was a fun challenge.

Detailed Comparison

Now that we have explored both paradigms, let's compare them across key dimensions.

A final note on hybrid approaches

Of course, these two paradigms are not mutually exclusive and real world applications could potentially benefit from a hybrid approach with:

• A high-level supervisor ensures guardrails and policy enforcement.
• Specific domains where agents can handoff to each other for more natural interactions.
• Some agents exposed as tools for specific tasks.

Conclusion

Both the architectural patterns explored in this article have their own advantages and trade-offs, and the best choice depends on the specific requirements and constraints of your project. If you prioritize ease of implementation, observability and predictable flows, agents as tools might be the way to go. If you prioritize natural conversational flow, scalability and autonomy, agent handoffs might be the better choice. The key question to ask yourself is: how complex and evolving are the tasks that your agents will need to handle and where should the control and context management responsibility lie in order to provide the best user experience?

For most teams building their first multi-agent system, starting with the agent as tools paradigm is a good way to get familiar with the concepts and capabilities of agents: it is easier to implement to debug and to explain to stakeholders. Once you understand where the friction is, you'll know whether handoffs are worth the added complexity.