Principles of Collaborative Multi LLM Agent Systems
A LLM agent is essentially a loop on top of LLMs, trying to solve a user problem or answering a question or investigating an open bounded issue. Collaborative Multi agent systems are essential for achieving complex tasks such as capture the flag. Below are some lessons we have learnt while implementing multi agent systems in production.
What are Multi Agent systems ?
Multi agent systems are systems with specialist agents, each designed well to do a specific task. They collaboratively work together to achieve the objectives for the user. The agents have access to a multitude of tools that would enable to them to accomplish their own objective.
Principles of Multi Agent Systems
Each agent must be a "Micro Agent"
For several technical (LLMs are still not there yet with respect to long ranging tasks) and maintenance reasons, each agent must do one thing and one thing only. Its objective must the clear with minimal number of tools, and simple prompts. The longer the prompt is the less effective the agent would be. If you have a task to achieve, break the task down into simpler tasks before assigning them to an agent.
There must be a Manager Agent
Multi agent systems are collaborating for a users objective, There must be an initiator and orchestrator of the agents. Thats a task with simple objective so create a 'manager agent' with that task. Its responsibilities are
- interface with the user
- Plan
- Give the response back to the user
Planning must Happen
Thinking about the user objective, which agents to use and for what reasons, will make the LLM anchored. following pr
- what is the user asking for ?
- What are the steps that are needed and questions that need to be answered for the users objective ?
- Which agents are the most appropriate for the steps ?
- Why do you think the agents help with the steps ?
There must be a quality Agent
There must be a definition of quality of what the right level of quality that must exist in the response before passing on to the user. The ownership of quality must therefore reside in one agent. The LLM eval metrics that the organization is interested in must go into the metrics that the quality agent would watch for when giving the verdict of the agent results.
The number of Loops must be Large
The number of loops the multi agent system makes to the LLM is proportional to the quality of the response. After a number, the quality will flatten out and for cost reasons, the manager agent must break the loop. Its essential to design the system in such a way that all the agents are looking at all the results in the chain and acting on the chain multiple times before passing the result to the user.
Transilience AI ML team .
