Automated Design of Agentic Systems
TL;DR
In this work, we describe a newly forming research area Automated Design of Agentic Systems (ADAS), which aims to automatically create powerful agentic system designs, including inventing novel building blocks and/or combining them in new ways.
We present a simple yet effective ADAS algorithm named Meta Agent Search to demonstrate that agents can invent novel and powerful agent designs by programming in code.
In Meta Agent Search, we instruct the "meta" agent to iteratively program new agents, test their performance on tasks, add them to an archive of discovered agents, and use this archive to inform the meta agent in subsequent iterations.
Examples of Discovered Agents
def forward(self, taskInfo):
# Step 1: Generate initial candidate solutions using multiple FM Modules
initial_instruction = 'Please think step by step and then solve the task by writing the code.'
num_candidates = 5 # Number of initial candidates
initial_module = [FM_Module(['thinking', 'code'], 'Initial Solution', temperature=0.8) for _ in range(num_candidates)]
initial_solutions = []
for i in range(num_candidates):
thoughts = initial_module[i]([taskInfo], initial_instruction)
thinking, code = thoughts[0], thoughts[1]
feedback, correct_examples, wrong_examples = self.run_examples_and_get_feedback(code)
if len(correct_examples) > 0: # Only consider solutions that passed at least one example
initial_solutions.append({'thinking': thinking, 'code': code, 'feedback': feedback, 'correct_count': len(correct_examples)})
# Step 2: Simulate human-like feedback for each candidate solution
human_like_feedback_module = FM_Module(['thinking', 'feedback'], 'Human-like Feedback', temperature=0.5)
human_feedback_instruction = 'Please provide human-like feedback for the code, focusing on common mistakes, heuristic corrections, and best practices.'
for sol in initial_solutions:
thoughts = human_like_feedback_module([taskInfo, sol['thinking'], sol['code']], human_feedback_instruction)
human_thinking, human_feedback = thoughts[0], thoughts[1]
sol['human_feedback'] = human_feedback
# Step 3: Assign expert advisors to evaluate and provide targeted feedback
expert_roles = ['Efficiency Expert', 'Readability Expert', 'Simplicity Expert']
expert_advisors = [FM_Module(['thinking', 'feedback'], role, temperature=0.6) for role in expert_roles]
expert_instruction = 'Please evaluate the given code and provide targeted feedback for improvement.'
for sol in initial_solutions:
sol_feedback = {}
for advisor in expert_advisors:
thoughts = advisor([taskInfo, sol['thinking'], sol['code']], expert_instruction)
thinking, feedback = thoughts[0], thoughts[1]
sol_feedback[advisor.role] = feedback
sol['expert_feedback'] = sol_feedback
# Step 4: Parse and structure the feedback to avoid redundancy and refine the solutions iteratively
max_refinement_iterations = 3
refinement_module = FM_Module(['thinking', 'code'], 'Refinement Module', temperature=0.5)
refined_solutions = []
for sol in initial_solutions:
for i in range(max_refinement_iterations):
combined_feedback = sol['feedback'].content + sol['human_feedback'].content + ''.join([fb.content for fb in sol['expert_feedback'].values()])
structured_feedback = ' '.join(set(combined_feedback.split())) # Avoid redundancy
refinement_instruction = 'Using the structured feedback, refine the solution to improve its performance.'
thoughts = refinement_module([taskInfo, sol['thinking'], sol['code'], Info('feedback', 'Structured Feedback', structured_feedback, i)], refinement_instruction, i)
refinement_thinking, refined_code = thoughts[0], thoughts[1]
feedback, correct_examples, wrong_examples = self.run_examples_and_get_feedback(refined_code)
if len(correct_examples) > 0:
sol.update({'thinking': refinement_thinking, 'code': refined_code, 'feedback': feedback, 'correct_count': len(correct_examples)})
refined_solutions.append(sol)
# Step 5: Select the best-performing solutions and make a final decision using an ensemble approach
sorted_solutions = sorted(refined_solutions, key=lambda x: x['correct_count'], reverse=True)
top_solutions = sorted_solutions[:3] # Select the top 3 solutions
final_decision_instruction = 'Given all the above solutions, reason over them carefully and provide a final answer by writing the code.'
final_decision_module = refinement_module(['thinking', 'code'], 'Final Decision Module', temperature=0.1)
final_inputs = [taskInfo] + [item for solution in top_solutions for item in [solution['thinking'], solution['code'], solution['feedback']]]
final_thoughts = final_decision_module(final_inputs, final_decision_instruction)
final_thinking, final_code = final_thoughts[0], final_thoughts[1]
answer = self.get_test_output_from_code(final_code)
return answer
def forward(self, taskInfo):
initial_instruction = "Please think step by step and then solve the task."
critique_instruction = "Please review the answer above and provide feedback on where it might be wrong. If you are absolutely sure it is correct, output 'True' in 'correct'."
refine_instruction = "Given previous attempts and feedback, carefully consider where you could go wrong in your latest attempt. Using insights from previous attempts, try to solve the task better."
final_decision_instruction = "Given all the above thinking and answers, reason over them carefully and provide a final answer."
FM_modules = [FM_module(['thinking', 'answer'], 'FM Module', role=role) for role in ['Physics Expert', 'Chemistry Expert', 'Biology Expert', 'Science Generalist']]
critic_modules = [FM_module(['feedback', 'correct'], 'Critic', role=role) for role in ['Physics Critic', 'Chemistry Critic', 'Biology Critic', 'General Critic']]
final_decision_module = FM_module(['thinking', 'answer'], 'Final Decision', temperature=0.1)
all_thinking = [[] for _ in range(len(FM_modules))]
all_answer = [[] for _ in range(len(FM_modules))]
all_feedback = [[] for _ in range(len(FM_modules))]
for i in range(len(FM_modules)):
thinking, answer = FM_modules[i]([taskInfo], initial_instruction)
all_thinking[i].append(thinking)
all_answer[i].append(answer)
for i in range(len(FM_modules)):
for j in range(len(FM_modules)):
if i != j:
feedback, correct = critic_modules[j]([taskInfo, all_thinking[i][0], all_answer[i][0]], critique_instruction)
all_feedback[i].append(feedback)
for i in range(len(FM_modules)):
refine_inputs = [taskInfo, all_thinking[i][0], all_answer[i][0]] + all_feedback[i]
thinking, answer = FM_modules[i](refine_inputs, refine_instruction)
all_thinking[i].append(thinking)
all_answer[i].append(answer)
final_inputs = [taskInfo] + [all_thinking[i][1] for i in range(len(FM_modules))] + [all_answer[i][1] for i in range(len(FM_modules))]
thinking, answer = final_decision_module(final_inputs, final_decision_instruction)
return answer
def forward(self, taskInfo):
# Step 1: Decompose the problem into sub-problems
decomposition_instruction = "Please decompose the problem into smaller, manageable sub-problems. List each sub-problem clearly."
decomposition_module = FM_Module(['thinking', 'sub_problems'], 'Decomposition Module')
# Step 2: Assign each sub-problem to a specialized expert
sub_problem_instruction = "Please think step by step and then solve the sub-problem."
specialized_experts = [FM_Module(['thinking', 'sub_solution'], 'Specialized Expert', role=role) for role in ['Physics Expert', 'Chemistry Expert', 'Biology Expert', 'General Expert']]
# Step 3: Integrate the sub-problem solutions into the final answer
integration_instruction = "Given the solutions to the sub-problems, integrate them to provide a final answer to the original problem."
integration_module = FM_Module(['thinking', 'answer'], 'Integration Module', temperature=0.1)
# Decompose the problem
thinking, sub_problems = decomposition_module([taskInfo], decomposition_instruction)
# Ensure sub_problems is a string and split into individual sub-problems
sub_problems_list = sub_problems.content.split('\n') if isinstance(sub_problems.content, str) else []
# Solve each sub-problem
sub_solutions = []
for i, sub_problem in enumerate(sub_problems_list):
sub_problem_info = Info('sub_problem', decomposition_module.__repr__(), sub_problem, i)
sub_thinking, sub_solution = specialized_experts[i % len(specialized_experts)]([sub_problem_info], sub_problem_instruction)
sub_solutions.append(sub_solution)
# Integrate the sub-problem solutions
integration_inputs = [taskInfo] + sub_solutions
thinking, answer = integration_module(integration_inputs, integration_instruction)
return answer
def forward(self, taskInfo):
# Instruction for generating visual representation of the problem
visual_instruction = "Please create a visual representation (e.g., diagram, graph) of the given problem."
# Instruction for verifying the visual representation
verification_instruction = "Please verify the accuracy and relevance of the visual representation. Provide feedback and suggestions for improvement if necessary."
# Instruction for solving the problem using the verified visual aid
cot_instruction = "Using the provided visual representation, think step by step and solve the problem."
# Instantiate the visual representation module, verification module, and Chain-of-Thought module
visual_module = FM_Module(['visual'], 'Visual Representation Module')
verification_module = FM_Module(['feedback', 'verified_visual'], 'Verification Module')
cot_module = FM_Module(['thinking', 'answer'], 'Chain-of-Thought Module')
# Generate the visual representation of the problem
visual_output = visual_module([taskInfo], visual_instruction)
visual_representation = visual_output[0] # Using Info object directly
# Verify the visual representation
feedback, verified_visual = verification_module([taskInfo, visual_representation], verification_instruction)
# Use the verified visual representation to solve the problem
thinking, answer = cot_module([taskInfo, verified_visual], cot_instruction)
return answer
New Research Area: Automated Design of Agentic Systems (ADAS)
The three key components of ADAS. The search space determines which agentic systems can be represented in ADAS. The search algorithm specifies how the ADAS method explores the search space. The evaluation function defines how to evaluate a candidate agent on target objectives such as performance.
Experiments
The results of Meta Agent Search on the ARC challenge. (a) Meta Agent Search progressively discovers high-performance agents based on an ever-growing archive of previous discoveries. We report the median accuracy and the 95% bootstrap confidence interval on a held-out test set by evaluating agents five times. (b) The visualization of the best agent discovered by Meta Agent Search on the ARC challenge.
Performance comparison between Meta Agent Search and state-of-the-art hand-designed agents across multiple domains. Meta Agent Search discovers superior agents compared to the baselines in every domain. We report the test accuracy and the 95% bootstrap confidence interval on held-out test sets. The search is conducted independently for each domain.
Importantly, we consistently observe the surprising result that agents invented by Meta Agent Search maintain superior performance even when transferred across domains and models, demonstrating their robustness and generality. Here, we present performance across multiple domains when transferring top agents from the Math (MGSM) domain to non-math domains. Agents discovered by Meta Agent Search in the math domain can outperform or match the performance of baselines after being transferred to domains beyond math. We report the test accuracy and the 95% bootstrap confidence interval. All results on transferability are available in the paper.
Citation
Acknowledgements
This work was supported by the Vector Institute, the Canada CIFAR AI Chairs program, grants from Schmidt Futures and Open Philanthropy, an NSERC Discovery Grant, and a generous donation from Rafael Cosman. We thank Jenny Zhang, Rach Pradhan, Ruiyu Gou, Nicholas Ioannidis, and Eunjeong Hwang for insightful discussions and feedback.
The website template was borrowed from Jon Barron.