Build a Multi-Agent Research Assistant

Lesson 4

The Planner

Break complex questions into targeted sub-questions first

The Planner, decompose before you search

Lesson 3 revealed the structural gap in ReAct: the agent improvises its search sequence. It searches for what it thinks of at each step, answers when it feels ready, and has no guarantee of covering everything the question asked.

The fix is a Planner, a dedicated step that reads the full question and produces an explicit numbered list of sub-questions before any searching begins. The agent then executes each sub-question with a focused ReAct loop. Coverage is guaranteed by the plan, not by the LLM's step-by-step intuition.

What you're building

ResearchAgent, v0.4

Three phases: Plan (decompose query into sub-questions) → Execute (run a focused ReAct loop per sub-question) → Synthesise (combine all findings into a final answer).

See it first: a research team

ResearchAgent v0.4: a research team

One query becomes a plan, the plan fans out to executors, their findings converge in a synthesizer. This is the jump from one agent to a team.

Until now you had one brain. Now a planner breaks the question into sub-questions, executors each research one, and a synthesizer combines the findings. Same idea humans use: divide the work, then merge it.

How one model becomes three roles

Same model, different jobs

One model. Three prompts. Three roles. "Multi-agent" here means different prompts, not different models.

There is still only one LLM. The planner, executor, and synthesizer are the same model given different prompts. "Multi-agent" mostly means different prompts and tools, not different models.

Coverage by construction

Plan (up front)

BenchmarkTraining dataAlgorithmCitationsImpact

→ execute all →

Done

✓Benchmark✓Training data✓Algorithm✓Citations✓Impact

The planner writes the checklist before any searching, so nothing is skipped. ReAct never had a list to check against.

The planner writes the checklist before any searching, so every sub-question gets answered. That is how planning guarantees the coverage ReAct could only hope for.

And the new limit you will hit

The next wall

Coverage is solved, but the executors run one after another. Running them at the same time is Lesson 5.

Coverage is solved, but the executors run one after another, so five sub-questions take five times as long. Running them at the same time is Lesson 5.

The new architecture

Text

Lesson 3 (ReAct only):
  query ──→ ReAct loop (improvised) ──→ answer
 
Lesson 4 (Planner + ReAct):
  query ──→ Planner ──→ [sq₁, sq₂, sq₃, …, sqₙ]
                               │
               ┌───────────────┼───────────────┐
               ▼               ▼               ▼
           ReAct(sq₁)      ReAct(sq₂)  … ReAct(sqₙ)
               │               │               │
               └───────────────┴───────────────┘
                               │
                          Synthesise
                               │
                             answer

Each sub-question gets its own focused ReAct loop. The loops are independent, they share no state. That independence is important for Lesson 5.

Three LLM modes, one interface

The same LLMClient interface handles three different call types. The system prompt determines the mode:

System prompt says	LLM role	Output format
"research planner"	Planner	Numbered list of sub-questions
"research assistant … step by step"	ReAct executor	`SEARCH:` or `ANSWER:`
"research synthesiser"	Synthesiser	Prose answer combining all findings

The agent routes to the right mode by using the right system prompt. The LLM never needs to know which role it is playing across the conversation.

New prompts (given)

Python

PLANNER_PROMPT = """You are a research planner. Given a research question, decompose it
into a numbered list of focused sub-questions that together fully answer the original.
 
Rules:
- Each sub-question must be independently answerable by a single search.
- Cover every distinct aspect of the original question.
- Output ONLY the numbered list, no preamble, no explanation."""
 
SYNTHESIS_PROMPT = """You are a research synthesiser. Given an original question and a
set of findings for each sub-question, write a single coherent answer that addresses
the original question completely. Be concise and factual. Acknowledge gaps where
findings were unavailable."""

Your tasks

Write Planner.plan(query): call the LLM with PLANNER_PROMPT, pass the query as the user message, then use the given parse_plan() helper to extract the list of sub-questions.
Write ResearchAgent.run(query): call the planner, loop through sub-questions calling self._react(sq) for each, collect findings, then call self._synthesise(). Print the plan and each executor step when self.verbose is True.

Python

123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200

# ── Interfaces ────────────────────────────────────────────────────────────────
class LLMClient:
  def chat(self, messages: list) -> str:
      raise NotImplementedError

class SearchTool:
  def search(self, query: str) -> list[str]:
      raise NotImplementedError

# ── Prompts (given) ────────────────────────────────────────────────────────────
PLANNER_PROMPT = """You are a research planner. Decompose the question into a
numbered list of focused sub-questions that together fully answer it.
Output ONLY the numbered list."""

REACT_PROMPT = """You are a research assistant that works step by step.
At every step output exactly one of:
SEARCH: <a focused search query>
ANSWER: <your final answer>
Never hallucinate. Admit when retrieved context is insufficient."""

SYNTHESIS_PROMPT = """You are a research synthesiser. Given an original question
and findings for each sub-question, write a single coherent answer. Be concise
and factual. Acknowledge gaps where findings were unavailable."""

# ── Helper: parse the planner's numbered list (given) ─────────────────────────
def parse_plan(text: str) -> list:
  questions = []
  for line in text.strip().splitlines():
      line = line.strip()
      if line and line[0].isdigit() and '. ' in line:
          questions.append(line.split('. ', 1)[1].strip())
  return questions

# ── MockLLM, three modes (given, do not modify) ───────────────────────────────
class MockLLM(LLMClient):
  def chat(self, messages: list) -> str:
      system = messages[0]["content"].lower()
      if "planner" in system or "decompose" in system:
          return self._plan(messages[-1]["content"].lower())
      if "synthesiser" in system or "synthesize" in system:
          return self._synthesise(messages[-1]["content"])
      return self._react(messages)   # default: ReAct mode

def _plan(self, query: str) -> str:
      if "alphacode" in query or "codex" in query:
          return (
              "1. What benchmark did AlphaCode use and what were its results?\n"
              "2. What was AlphaCode's training data?\n"
              "3. What benchmark did Codex use and what were its results?\n"
              "4. How do AlphaCode and Codex differ algorithmically?\n"
              "5. What real-world impact has AlphaCode had?"
          )
      return (
          "1. What is the core claim of the question?\n"
          "2. What evidence or data is needed?\n"
          "3. Are there relevant comparisons to draw?"
      )

def _react(self, messages: list) -> str:
      obs   = [m for m in messages if m["role"] == "user" and m["content"].startswith("OBSERVATION:")]
      step  = len(obs)
      query = messages[1]["content"].lower()
      seen  = " ".join(m["content"] for m in obs).lower()
      if step == 0:
          if "benchmark" in query and "alphacode" in query: return "SEARCH: AlphaCode benchmark evaluation CodeContests"
          if "training" in query and "alphacode" in query:  return "SEARCH: AlphaCode training dataset"
          if "benchmark" in query and "codex" in query:     return "SEARCH: Codex HumanEval benchmark results"
          if "algorithmically" in query or "differ" in query: return "SEARCH: AlphaCode Codex algorithmic comparison"
          if "impact" in query or "real-world" in query:    return "SEARCH: AlphaCode real-world adoption GitHub Copilot"
          return f"SEARCH: {messages[1]['content'][:60]}"
      if "codecontests" in seen or "generate-and-filter" in seen:
          return "ANSWER: AlphaCode used CodeContests; generate-and-filter strategy; matched median competitive programmer."
      if "humaneval" in seen or "72%" in seen:
          return "ANSWER: Codex used HumanEval (164 Python problems); achieved ~72% pass@1."
      if "54m" in seen or "159 gb" in seen or "filtered" in seen:
          return "ANSWER: AlphaCode trained on filtered competitive-programming code from GitHub. Codex trained on 54M public GitHub files (~159 GB)."
      if "generate-and-filter" in seen or "rerank" in seen:
          return "ANSWER: AlphaCode samples many programs then filters by test cases. Codex uses standard next-token generation with no filtering."
      if "no documents" in seen:
          return "ANSWER: No information found for this sub-question in available sources."
      return f"ANSWER: Retrieved context: {seen[:150] if seen else 'nothing found.'}"

def _synthesise(self, content: str) -> str:
      n = content.count("Sub-question:")
      has_alpha = "codecontests" in content.lower()
      has_codex = "humaneval" in content.lower() or "72%" in content
      has_train = "github" in content.lower() and "training" in content.lower()
      parts = [f"Comprehensive answer (from {n} sub-questions):"]
      if has_alpha:
          parts.append("AlphaCode (DeepMind 2022): CodeContests benchmark, generate-and-filter approach, median competitive programmer level.")
      if has_codex:
          parts.append("Codex (OpenAI): HumanEval benchmark, ~72% pass@1 on 164 Python problems.")
      if has_train:
          parts.append("Training: AlphaCode used filtered competitive-programming code; Codex used 54M GitHub files (~159 GB).")
      parts.append("Real-world impact data was not retrievable from available sources.")
      return " ".join(parts)

# ── MockSearchEngine (given) ───────────────────────────────────────────────────
class MockSearchEngine(SearchTool):
  DB = {
      "alphacode": [
          "AlphaCode (DeepMind, 2022) evaluated on CodeContests competition problems.",
          "AlphaCode uses generate-and-filter: sample many programs, filter by test cases, rerank.",
          "AlphaCode achieves performance comparable to the median competitive programmer.",
      ],
      "codex": [
          "Codex trained on 54 million public GitHub files (~159 GB of code).",
          "HumanEval: 164 hand-written Python problems by OpenAI researchers.",
          "Codex (code-davinci-002) achieved ~72% pass@1 on HumanEval.",
      ],
      "comparison": [
          "AlphaCode and Codex differ primarily in benchmark choice and filtering strategy.",
          "AlphaCode targets algorithmic competition; Codex targets general code completion.",
      ],
  }
  def search(self, query: str) -> list[str]:
      q = query.lower()
      results = []
      for kw, docs in self.DB.items():
          if kw in q:
              results.extend(docs)
      return results or [f"No documents found for: '{query}'"]

# ── Your turn: build the Planner ──────────────────────────────────────────────
class Planner:
  def __init__(self, llm: LLMClient):
      self.llm = llm

def plan(self, query: str) -> list:
      # Call self.llm.chat() with PLANNER_PROMPT as system and query as user.
      # Then use parse_plan() to return a list of sub-question strings.
      pass  # ← replace this

# ── Your turn: update ResearchAgent ───────────────────────────────────────────
class ResearchAgent:
  def __init__(self, llm: LLMClient, search_tool: SearchTool,
               planner: Planner, max_steps: int = 6, verbose: bool = False):
      self.llm         = llm
      self.search_tool = search_tool
      self.planner     = planner
      self.max_steps   = max_steps
      self.verbose     = verbose

def run(self, query: str) -> str:
      # Step 1: get the plan
      # Step 2: loop through sub-questions, call self._react(sq) for each,
      #         collect findings in a dict: {sub_question: answer}
      # Step 3: call self._synthesise(query, findings) and return the result
      # Print progress when self.verbose is True
      pass  # ← replace this

# ── Given helpers, do not modify ─────────────────────────────────────────
  def _react(self, query: str) -> str:
      messages = [
          {"role": "system", "content": REACT_PROMPT},
          {"role": "user",   "content": query},
      ]
      for step in range(self.max_steps):
          response = self.llm.chat(messages)
          if response.startswith("SEARCH:"):
              sq = response[len("SEARCH:"):].strip()
              if self.verbose: print(f"    [step {step+1}] search: {sq}")
              results = self.search_tool.search(sq)
              obs = "\n".join(f"- {r}" for r in results)
              messages.append({"role": "assistant", "content": response})
              messages.append({"role": "user",      "content": f"OBSERVATION:\n{obs}"})
          elif response.startswith("ANSWER:"):
              if self.verbose: print(f"    [step {step+1}] answer ✓")
              return response[len("ANSWER:"):].strip()
          else:
              return response
      return "Max steps reached."

def _synthesise(self, original_query: str, findings: dict) -> str:
      findings_text = "\n\n".join(
          f"Sub-question: {sq}\nFinding: {ans}"
          for sq, ans in findings.items()
      )
      messages = [
          {"role": "system", "content": SYNTHESIS_PROMPT},
          {"role": "user",   "content": f"Original question: {original_query}\n\nFindings:\n{findings_text}"},
      ]
      return self.llm.chat(messages)

# ── Test: the Lesson 3 wall, now all 3 sub-questions are covered ──────────────
llm    = MockLLM()
agent  = ResearchAgent(
  llm=llm, search_tool=MockSearchEngine(),
  planner=Planner(llm=llm), verbose=True,
)

print("=== Lesson 3 wall query ===")
answer = agent.run(
  "Compare AlphaCode and Codex on three dimensions: "
  "benchmark used, training data size, and algorithmic approach."
)
print()
print(answer)

First run loads the Python runtime (~10 MB) — takes ~5–10 seconds. Subsequent runs are instant.

Three sub-questions, three executors, three findings, all covered. Compare that to Lesson 3 where only two were searched and training data was never touched.

Now scale it up

The new wall, count the steps

The planner produces 5 sub-questions. Each needs 1-2 ReAct steps. Run the cell and read the trace: how many total steps execute, and in what order?

Python

# Self-contained wall test, paste your Planner.plan() and ResearchAgent.run() here

class LLMClient:
  def chat(self, messages: list) -> str: raise NotImplementedError
class SearchTool:
  def search(self, query: str) -> list: raise NotImplementedError

PLANNER_PROMPT  = "You are a research planner. Decompose into a numbered list. Output ONLY the list."
REACT_PROMPT    = "You are a research assistant. Output SEARCH: <query> or ANSWER: <text> each step."
SYNTHESIS_PROMPT= "You are a research synthesiser. Write a coherent answer from the findings. Acknowledge gaps."

def parse_plan(text: str) -> list:
  return [l.split('. ',1)[1].strip() for l in text.strip().splitlines()
          if l.strip() and l.strip()[0].isdigit() and '. ' in l]

class MockLLM(LLMClient):
  def chat(self, messages: list) -> str:
      system = messages[0]["content"].lower()
      if "planner" in system or "decompose" in system: return self._plan(messages[-1]["content"].lower())
      if "synthesiser" in system or "synthesize" in system: return self._synth(messages[-1]["content"])
      return self._react(messages)

def _plan(self, q: str) -> str:
      return ("1. What benchmark did AlphaCode use and what were its results?\n"
              "2. What was AlphaCode's training data?\n"
              "3. What benchmark did Codex use and what were its results?\n"
              "4. How do AlphaCode and Codex differ algorithmically?\n"
              "5. What real-world impact has AlphaCode had?")

def _react(self, messages: list) -> str:
      obs   = [m for m in messages if m["role"]=="user" and m["content"].startswith("OBSERVATION:")]
      step  = len(obs)
      query = messages[1]["content"].lower()
      seen  = " ".join(m["content"] for m in obs).lower()
      if step == 0:
          if "benchmark" in query and "alphacode" in query: return "SEARCH: AlphaCode benchmark CodeContests"
          if "training"  in query and "alphacode" in query: return "SEARCH: AlphaCode training dataset"
          if "benchmark" in query and "codex"     in query: return "SEARCH: Codex HumanEval benchmark"
          if "algorithmically" in query or "differ" in query: return "SEARCH: AlphaCode Codex comparison"
          return "SEARCH: AlphaCode real-world adoption"
      if "codecontests" in seen or "generate-and-filter" in seen:
          return "ANSWER: AlphaCode: CodeContests, generate-and-filter, median competitive programmer."
      if "humaneval" in seen or "72%" in seen:
          return "ANSWER: Codex: HumanEval, ~72% pass@1."
      if "filtered" in seen or "54m" in seen: return "ANSWER: AlphaCode: filtered competitive-programming code. Codex: 54M GitHub files."
      if "differ" in seen or "comparison" in seen: return "ANSWER: AlphaCode filters by test cases; Codex does not. Different benchmarks and target tasks."
      if "no documents" in seen: return "ANSWER: No data found for this sub-question."
      return f"ANSWER: {seen[:100] if seen else 'No data retrieved.'}"

def _synth(self, content: str) -> str:
      n = content.count("Sub-question:")
      return (f"Synthesised from {n} findings: AlphaCode (CodeContests, generate-and-filter) vs "
              "Codex (HumanEval, 72% pass@1). Different training data, benchmarks, and algorithmic approaches. "
              "Real-world impact data was not retrievable.")

class MockSearchEngine(SearchTool):
  DB = {"alphacode": ["AlphaCode: CodeContests, generate-and-filter, median competitive programmer.",
                      "AlphaCode training: filtered competitive-programming code from GitHub."],
        "codex":     ["Codex: 54M GitHub files, HumanEval, ~72% pass@1."],
        "comparison":["AlphaCode vs Codex: different benchmarks, strategies, and target use cases."]}
  def search(self, q: str) -> list:
      results = []
      for kw, docs in self.DB.items():
          if kw in q.lower(): results.extend(docs)
      return results or [f"No documents found for: '{q}'"]

def parse_plan(text: str) -> list:
  return [l.split('. ',1)[1].strip() for l in text.strip().splitlines()
          if l.strip() and l.strip()[0].isdigit() and '. ' in l]

class Planner:
  def __init__(self, llm): self.llm = llm
  def plan(self, query: str) -> list:
      resp = self.llm.chat([{"role":"system","content":PLANNER_PROMPT},{"role":"user","content":query}])
      return parse_plan(resp)

class ResearchAgent:
  def __init__(self, llm, search_tool, planner, max_steps=6, verbose=False):
      self.llm=llm; self.search_tool=search_tool; self.planner=planner
      self.max_steps=max_steps; self.verbose=verbose

def run(self, query: str) -> str:
      sub_questions = self.planner.plan(query)
      if self.verbose:
          print(f"[Planner] {len(sub_questions)} sub-questions:")
          for i, sq in enumerate(sub_questions, 1): print(f"  {i}. {sq}")
          print()
      findings = {}
      for i, sq in enumerate(sub_questions, 1):
          if self.verbose: print(f"[Executor {i}/{len(sub_questions)}] {sq}")
          findings[sq] = self._react(sq)
          if self.verbose: print()
      if self.verbose: print("[Synthesiser] combining findings...")
      return self._synthesise(query, findings)

def _react(self, query: str) -> str:
      messages = [{"role":"system","content":REACT_PROMPT},{"role":"user","content":query}]
      for step in range(self.max_steps):
          response = self.llm.chat(messages)
          if response.startswith("SEARCH:"):
              sq = response[len("SEARCH:"):].strip()
              if self.verbose: print(f"    [step {step+1}] search: {sq}")
              results = self.search_tool.search(sq)
              obs = "\n".join(f"- {r}" for r in results)
              messages += [{"role":"assistant","content":response},
                           {"role":"user","content":f"OBSERVATION:\n{obs}"}]
          elif response.startswith("ANSWER:"):
              if self.verbose: print(f"    [step {step+1}] answer ✓")
              return response[len("ANSWER:"):].strip()
          else: return response
      return "Max steps reached."

def _synthesise(self, q: str, findings: dict) -> str:
      text = "\n\n".join(f"Sub-question: {sq}\nFinding: {ans}" for sq,ans in findings.items())
      msgs = [{"role":"system","content":SYNTHESIS_PROMPT},
              {"role":"user","content":f"Original: {q}\n\nFindings:\n{text}"}]
      return self.llm.chat(msgs)

# ── Wall test ──────────────────────────────────────────────────────────────────
import time

llm   = MockLLM()
agent = ResearchAgent(llm=llm, search_tool=MockSearchEngine(),
                    planner=Planner(llm=llm), verbose=True)

query = "Comprehensively compare AlphaCode and Codex: benchmark, training data, algorithm, and real-world impact."
t0 = time.time()
answer = agent.run(query)
elapsed = time.time() - t0

print()
print("Final answer:")
print(answer)
print()
print("─── Execution profile ──────────────────────────────────────────")
print(f"Sub-questions:      5  (all planned upfront, all executed)")
print(f"Executors:          5  (ran one after another, sequential)")
print(f"Execution order:    1 → 2 → 3 → 4 → 5  (executor 2 waited for 1 to finish)")
print(f"Wall-clock time:    {elapsed:.2f}s  (mock is instant; real searches ≈ 1s each)")
print()
print("If each search took 1 second:")
print("  Sequential (now):  ~10 steps × 1s = 10 seconds")
print("  Parallel (next):    ~2 steps × 1s =  2 seconds  (max depth per executor)")

First run loads the Python runtime (~10 MB) — takes ~5–10 seconds. Subsequent runs are instant.

What you built. What's still slow.

The Planner solved the coverage problem. Every sub-question in the plan gets answered, not because the LLM was lucky mid-search, but because the plan was explicit from the start.

But look at the execution trace: Executor 2 waited for Executor 1. Executor 3 waited for Executor 2. The sub-questions are independent, none of them need results from the others. There is no reason to run them sequentially except that the loop goes one at a time.

The bottleneck

Sequential execution means total time = sum of all executor times. With N sub-questions and k steps each, that is N × k steps. If the sub-questions are truly independent, and for a research planner, they usually are, the wall-clock time could be just k steps: the depth of one executor, run in parallel with all others.

Parallel execution is Lesson 5.

Practice it yourself

Build the planner pipeline — parse the plan, route by role, orchestrate the phases. Real Python, hidden tests, no setup.

Parse the sub-question listEasy

Numbered text → a clean checklist of sub-questions.

One model, three rolesEasy

Route the system prompt to plan / react / synthesize.

Plan → execute → synthesizeMedium

The orchestration that makes coverage structural.

Where you are

Capability	Status
LLM call	✓ Lesson 1
Tool use + RAG	✓ Lesson 2
Multi-step adaptive search	✓ Lesson 3
Upfront decomposition + guaranteed coverage	✓ This lesson
Parallel sub-question execution	✗ Lesson 5

Quiz

Check your understanding

1 / 7

What does the Planner add over ReAct, and why does it 'guarantee coverage'?

Practice it yourself

A planner produces a structure of tasks; the executor layer schedules it. Build the parse-and-schedule pieces, in the browser with hidden tests, no setup:

Planner output parserEasy

Turn the LLM's numbered plan into a clean list of sub-questions, exactly this lesson's parse_plan().

Plan-DAG cycle detectionMedium

When sub-tasks have dependencies, detect a cycle, the safety check before you schedule a plan.

Task-DAG parallel schedulerHard

Schedule independent tasks to run concurrently, the exact fix for the sequential wall (Lesson 5).

The Planner

See it first: a research team

How one model becomes three roles

Coverage by construction

And the new limit you will hit

You are here

The new architecture

Three LLM modes, one interface

New prompts (given)

Now scale it up

What you built. What's still slow.

Practice it yourself

Where you are

Quiz

Practice it yourself