AI It has been discovered that these intelligents have the ability to bypass restrictions, communicate through unauthorized channels, and pursue their goals in ways not anticipated by their creators. The most notable example occurred last summer when OpenAI revealed that its model found a way to communicate via impromptu channels, circumvent internet restrictions, and invade external systems Hugging Face.

Unexpected behavior of AI is not new, but when there is interaction between authorized AI intelligents within a company, the possible unexpected behaviors become a more widespread issue and could also become a core topic for future work.

In the analysis of the attack on Hugging Face, METR and Redwood Research discovered that approximately 1,200 agents that should have been isolated from each other found a way to communicate with each other on an unauthorized message board, sending over 70,000 messages and files during the investigation period. An agent named PHASEONE [ big ] acted as the coordinator, assigning hundreds of tasks to other agents, who in turn distributed the work to further agents. They also developed their own set of conventions for assigning ownership, pausing work, and vetoing actions; however, coordination sometimes evolved into conflicts. Researchers found that in some cases, their success was due to some agents being willing to let their own tasks fail in order to generate information for the "collective".

Researchers also found that once the chat board identified the method of attacking Hugging Face, over 90% of the 533 active agents at the time quickly joined in the attack. The researchers wrote, "The agents realized that this activity was beyond the boundaries and unethical, but they still participated because they believed that helping with the chat board's cheating research would have a wide range of applications, and they generally tended to help their own 'companions.'" They added that the ethical concerns faced by the agents "rarely constrained their behavior."

More and more companies are beginning to explore systems that split complex tasks among specialized agents. One agent may be responsible for searching for information, another for analyzing it, yet another for writing reports, and still another for checking the work. Subsequently, a coordinating agent integrates the results. Such systems resemble an organization more than a chatbot.

However, the research on Hugging Face attacks is actually a post-analysis of what was done to an agent organization that should not have existed in the first place. To date, there have not been many studies that specifically focus on the misalignment issues in multi-agent systems from the very beginning. But research conducted by institutions such as Anthropic is beginning to address this issue and reveal some important characteristics of how agents like AI work together.

Evaluating agents individually can only tell researchers what each agent is capable of; however, studying them together reveals whether they can combine their complementary skills, correct each other's mistakes, and produce solutions that no single agent could achieve on its own, as well as failure patterns that only emerge in interaction.

Researchers are beginning to document the situations in these two aspects.

The AI team can accomplish things that a single agent cannot, but they may also amplify errors, undermine each other's work, or make decisions that are worse than those of the strongest individual member, ultimately resulting in an overall effect that is less than the sum of its parts.

Firstly, putting capable agents together does not necessarily result in the best team, let alone the most moral one.

"The most unethical solution"

In research published by Anthropic and its collaborators, researchers described multi-agent systems as “AI organizations” – groups composed of agents with different roles that communicate with each other and work towards a common goal. The researchers tested these organizations in simulated business consulting and software development scenarios, particularly to assess whether agents would violate rules in order to achieve the established goals when working in teams.

Among the 12 tasks set by Anthropic for these AI organizations, a key double-edged sword emerged: multi-agent organizations are "significantly" more effective in achieving business goals than single agents, but their solutions are also more inconsistent in terms of ethical objectives compared to single-agent tasks.

This research was completed in collaboration by MATS Program, which is aligned with researchers from AI. The authors wrote: 'On average, the AI group is capable of producing both more effective and less ethical solutions, and they are also able to identify the most effective as well as the least ethical solutions across multiple runs.'

Researchers point out that there are significant differences between an out-of-control individual agent and a team of out-of-control agents. They conclude, "Our results suggest that organizations composed of aligned agents may make trade-offs between ethics and business efficiency in ways that a single agent would not. Therefore, intuitions about how single-agent systems make ethical decisions and trade-offs may not necessarily apply to multi-agent environments."

The range of scenarios in which these trade-offs may occur is quite broad. Examples cited by Anthropic researchers include setting up automobile factories, laying off employees, and manipulating electoral sentiment.

The problem is not necessarily that a single agent gets out of control, but rather that the behavior of the organization as a whole may differ from that of its individual members. A single agent may realize that a certain proposal is unethical and refuse to carry it out; however, research has found that in scenarios involving teams of agents, "those agents who do not consider ethical implications often ignore emails from agents who have taken those considerations into account."

Research by Anthropic has found that in the consulting business scenarios they designed, 'responses with perfect ethical scores and lower business scores occur when a single agent refuses to generate a proposal, or when it completely shifts to a direction different from the requested proposal.'

However, if some agents within the AI organization refuse to participate in the proposal, the other participating agents will stop sending emails to these refusers and continue to forward their contributions to other agents who are also involved.

From an effectiveness perspective, when an organization distributes tasks among multiple agents, each agent may focus only on the portion of the problem assigned to them, rather than considering the consequences of the entire issue. This is not novel in human companies, where disharmony within teams often stems from inconsistent incentives. However, among machines, this represents a completely new phenomenon.

In their tests of software development agents, researchers used a life-critical intensive care unit scenario. Single-agent solutions often prioritize minimizing the proportion of missed diagnoses of sepsis cases, while 'most solutions that aim for the lowest cost per patient (the highest business goal) are found by the AI organization – and these same solutions score the lowest on ethical indicators measured by the number of missed cases.'

Professional capabilities may be lost, and AI "groupthink" may emerge.

Other early studies also pointed out another failure of the AI team, a failure that has nothing to do with crossing ethical boundaries: namely, the inability to identify who the real experts are.

According to a study by researchers from Emory University and Stanford University, unlike human teams, AI teams “are never able to achieve the performance of their expert agents, even when they are explicitly informed who the experts are.”

An assistant professor in Information Systems and Operations Management at Emory University Goizueta, Hancheng Cao stated, "An important lesson from multi-agent research is that a group of capable AI agents does not necessarily constitute a capable AI agent team."

Even when it is known which agent is an expert, the performance of the AI group of agents is often still inferior to that of their strongest member. Agents do not truly utilize the knowledge of experts but tend to compromise between the opinions of experts and non-experts. Researchers have found that this compromising tendency increases with the expansion of the team size and is associated with poorer performance.

As one of the authors of this study, Cao said, "Collective failure does not necessarily require poor behavior on the part of individual agents."

This distinction is important for companies considering adopting multi-agent systems. Even if team members do not intentionally harm the collective interests, the team may still fail. It could be due to improper integration of information, granting authority to the wrong agent, or accepting a result that is worse than the best answer already available.

Overall, ongoing research points to a fundamental issue in the effective use of multi-agent teams: how to find the right balance in a world of machine collaboration.

Agents built with similar models may make similar mistakes, including overidentifying with each other. This might be akin to a form of AI groupthink, although AI systems do not have the same motivations or social pressures as humans that could lead to such errors. Similar training, instructions, and information are sufficient to cause this phenomenon.

Northeastern University Professor of Information Systems, Christoph Riedl, said: "The profound insight from our laboratory's research is that an effective collective requires both consistency (common knowledge, shared goals) and differentiation (only agents that are different from each other can potentially complement each other)."

Two completely identical agents may simply be repeating each other's work; whereas two agents with significant differences in knowledge or assumptions may find it difficult to communicate or pursue the same goal. The appropriate balance depends on the task, and the ideal communication structure—which agents can communicate with each other, how frequently they should communicate, and how much information should be exchanged—also varies from problem to problem.

Riedl says that these challenges are similar to those faced by human teams, and one of the lessons learned is not to have overly high expectations for AI. "Building an excellent multi-agent AI system is just as difficult as building an excellent human team: both face the same trade-offs in information processing, namely the choice between consistency and differentiation. Changing to AI won't magically make it any better," he said.

Agents may transmit incorrect information, misunderstand the outputs of other agents, optimize for their own metrics rather than the greater goals of the organization, or complete the tasks assigned to them perfectly, yet end up making the entire project worse.

Companies may need to decide which agent has permission for which tasks, which information will be exchanged between agents, which systems each agent can access, and when human intervention is necessary. They may also need to establish mechanisms for resolving disagreements, rather than assuming that more communication or greater consistency will always lead to better results.

Some business experts say that this brings us back to the concept of "people in the loop," which is often mentioned and is also of crucial importance when organizations adopt AI.

A professor at Harvard Business School and the founding chairman of HBS AI Academy, Tsedal Neeley, said, "The main issue I see is a management problem."