Part 4 built the tasks layer: structured plan generation with Pydantic, human-in-the-loop approval before anything runs, serial execution with dependency resolution, an LLM-as-judge verifying each task's output, and crash recovery that resets incomplete tasks on restart. ClearCode could take a goal, break it into steps, and implement it autonomously.

Part 5 is about making it fast. And about the harder problem that fast creates.

The problem with asking the same question twice

Every /ask query goes through the same pipeline: embed the question, retrieve relevant chunks, invoke the agent with that context, stream tool calls until an answer emerges, return the result. For a complex question, that might be four or five tool calls and two LLM completions. For a simple one, it is still at least one embedding call and one completion.

If you ask the same question twice — or a slightly different version of the same question — you pay the full price both times.

This compounds quickly in practice. You ask the same types of questions about a codebase repeatedly. "Where is X defined?", "What does Y do?", "How is Z called?" drift in phrasing but converge on the same answer. The agent does the same work, produces the same output, and charges you again.

The fix is a cache. But an exact-match cache fails the moment the phrasing changes. "What does index_codebase do?" and "Can you explain what index_codebase does?" should hit the same entry. They will not under an exact-match scheme.

The right cache for a language model application is a semantic cache: one that measures similarity between queries in embedding space and returns the stored answer if the new question is close enough to one already answered.

1. Semantic cache: why cosine similarity, not exact match

The cache stores (question_embedding, answer) pairs. On a new query, it embeds the question, finds the nearest stored embedding by cosine similarity, and returns the associated answer if the similarity exceeds a configurable threshold. Below the threshold, it falls through to the agent.

async def get(self, query: str, domain: str, model: str) -> str | None:
    hit = await self._top_match(query, domain=domain, model=model)
    if hit is None:
        return None
    similarity = 1 - float(hit["vector_distance"])
    if similarity >= self.threshold:
        return hit["response"]
    return None

The threshold is the key tuning parameter. Too high and you only hit on near-exact phrasing — the cache becomes useless. Too low and you start returning cached answers for questions that are only superficially similar, which is worse than no cache at all. 0.85 cosine similarity is the default, chosen empirically. It catches rephrasings and synonym variations while rejecting questions that are genuinely different.

The embeddings come from the same model used for retrieval — text-embedding-3-small by default. This is intentional. The cache and the retrieval layer operate in the same vector space. Switching embedding models requires rebuilding both, but they are always consistent with each other.

2. Redis + RedisVL

The backing store is Redis with a RedisVL HNSW vector index. Redis was the natural choice: fast, widely deployed, and RedisVL gives it native approximate nearest-neighbour search without requiring a separate vector database.

The index schema is defined once at startup:

def _build_index_schema(dims: int) -> dict:
    return {
        "index": {"name": "semantic_cache", "prefix": "cache:"},
        "fields": [
            {
                "name": "query_vector",
                "type": "vector",
                "attrs": {
                    "dims": dims,
                    "algorithm": "HNSW",
                    "distance_metric": "cosine",
                    "datatype": "float32",
                },
            },
            {"name": "response",   "type": "text"},
            {"name": "query_text", "type": "text"},
            {"name": "domain",     "type": "tag"},
            {"name": "model",      "type": "tag"},
            {"name": "created_at", "type": "numeric"},
        ],
    }

overwrite=False on index.create() means the index is reused across restarts rather than wiped. Cache entries survive a REPL restart — you do not pay embedding costs again on the first query of a new session if you already asked the same question before.

One correctness detail worth naming: embed_query on LangChain's embedder is a synchronous function. Calling it directly from an async method blocks the event loop. The fix is asyncio.to_thread:

async def _embed(self, text: str) -> list[float]:
    return await asyncio.to_thread(self.embedder.embed_query, text)

This runs the sync call in a threadpool executor and awaits the result without blocking. Missing this produces a TypeError and a confusing error message that does not point at the actual cause.

3. Domain and model isolation

A single Redis instance can serve multiple repos. Without isolation, cached answers from one codebase would contaminate queries about another — both happening to use the same question phrasing.

Each repo gets a domain: the first 16 characters of the SHA-256 hash of its absolute path.

def get_repo_domain(repo_path: str) -> str:
    return hashlib.sha256(repo_path.encode()).hexdigest()[:16]

Every cache entry is tagged with both domain and model. The VectorQuery filters on both before scoring similarity:

q.set_filter((Tag("domain") == domain) & (Tag("model") == model))

Two repos on the same Redis instance never share entries. And switching from gpt-4o to claude-opus-4 does not cause the cache to serve answers generated by a different model. Different models have different capabilities and different failure modes. An answer correct for one may not be for another.

4. The watcher → cache bridge

A cached answer that is no longer correct is worse than no cache at all.

Ask "What does process_payment do?", get an accurate summary, cache it. Then a /plan task rewrites process_payment completely. The next time you ask, the cache returns the old answer without checking the codebase. You are now confidently wrong.

The solution is to invalidate the cache whenever the codebase changes. ClearCode already has a filesystem watcher that fires on create, modify, rename, or delete. The cache invalidation just needs to be wired to that callback.

The wiring has one complication: the watchdog observer runs in a background daemon thread. The asyncio event loop — which owns the Redis client — runs in the main thread. You cannot call an async function from a background thread directly.

The bridge is asyncio.run_coroutine_threadsafe:

loop = asyncio.get_running_loop()

def invalidate_cache_on_change():
    if semantic_cache is not None:
        asyncio.run_coroutine_threadsafe(
            semantic_cache.invalidate_domain(cache_domain), loop
        )

This submits the coroutine to the event loop from the watcher thread and returns a Future. The event loop picks it up when it is next free. The callback takes no arguments — it only needs to signal that something changed, not what.

From this point forward, any file change in the project directory triggers cache invalidation for that repo's domain automatically. A /reindex command does the same explicitly.

5. The bug: invalidate_domain was silently doing nothing

This is the one worth naming carefully, because it is exactly the kind of bug that passes every integration test you write — right up until you write a test that checks the right thing.

invalidate_domain is supposed to find all cache entries for a given domain and delete them. The implementation used RedisVL's FilterQuery to locate entries by their domain tag, then called client.delete() on the resulting keys:

results = await self.index.query(fq)
if not results:
    return
keys = [self.index.key(r["id"]) for r in results]  # ← the bug
await self.client.delete(*keys)

The bug is in that one line. r["id"] in a RedisVL query result is already the full Redis key — prefix included. For this index, a stored key looks like cache:abc123def456. So r["id"] is "cache:abc123def456".

self.index.key(r["id"]) prepends the index prefix to whatever you pass it. That produces "cache:cache:abc123def456".

client.delete("cache:cache:abc123def456") deletes zero keys. Redis returns success — you are allowed to delete a key that does not exist. The function logs "Invalidated N cache entries" and exits cleanly. Nothing is actually deleted.

The fix is one word:

keys = [r["id"] for r in results]  # r["id"] is already the full key

What made this dangerous: the logging told you invalidation succeeded. The code path had no exception to catch. Entries survived silently, and subsequent queries returned stale cached answers with no indication that something was wrong.

The tests caught it. After calling invalidate_domain, the next get() on the same query should return None. It did not. That assertion failure is what led to the fix.

6. Incremental indexing and watcher improvements

Two related improvements went in alongside the cache.

Incremental indexing in ChromaDB

The original implementation re-embedded every source file on every startup. 300 files, restart the REPL, pay 300 embedding API calls — even if nothing had changed since yesterday.

The fix is mtime-based tracking. Each indexed chunk stores the mtime of the file it came from. On startup, the indexer compares current file mtimes against stored ones:

• File is new → embed and index.

• File has the same mtime → skip entirely. No API call.

• File has a newer mtime → delete old chunks for that file, re-embed, re-index.

• File no longer exists → prune its chunks from the collection.

A repo with 300 unchanged files now starts in the time it takes to check 300 mtimes. The first run still pays the full embedding cost. Every run after costs nothing for files that have not changed.

Watcher backend dispatch

The original watcher hardcoded ChromaDB. If you switched to Qdrant in config, file changes updated the ChromaDB index nobody was querying — not the Qdrant index that was.

The fix routes through the factory at call time:

def _run(self, action: str, filepath: str) -> None:
    from clearcode.context.indexers.factory import (
        get_single_file_indexer, get_file_remover
    )
    if action == "delete":
        get_file_remover()(filepath)
    else:
        get_single_file_indexer()(filepath)
    if self.on_change is not None:
        self.on_change()

Switching backends in config is now truly a one-line change — the watcher follows automatically, and the cache invalidation callback fires regardless of which backend is active.

7. handle_query returns a tuple

handle_query used to return a str. It now returns (str, bool) — the answer and whether it came from cache.

async def handle_query(...) -> tuple[str, bool]:
    ...
    if cached_response is not None:
        return cached_response, True
    ...
    return answer, False

This lets main.py print a visual indicator on cache hits without embedding cache-awareness into the orchestrator beyond what it needs to do its job:

> /ask Can you explain what index_codebase does?

Searching for: Can you explain what index_codebase does?...
⚡ cache hit

index_codebase scans all source files in the repo, parses them with
tree-sitter for AST-aware chunking, and upserts the resulting chunks
into the configured vector store backend. On startup and /reindex it
skips files whose mtime hasn't changed, so only new or modified files
are re-embedded.

The ⚡ cache hit line tells you the answer came from Redis. No LLM was invoked. No tools were called.

8. The test suite

The changes in Part 5 are tested end-to-end against a live Redis instance. Nineteen assertions cover the full cache workflow:

1. build_semantic_cache() returns a SemanticCache

2. Threshold loaded from config correctly

3. get() returns None on a cold cache

4. Exact query returns cached answer after put()

5. Semantically similar query hits cache above 0.85 threshold

6. Similar query returns the same stored answer

7. Unrelated query misses (pasta carbonara)

8. Different domain misses on identical query

9. Different model tag misses on identical query

10. invalidate_domain() deletes all domain entries

11. After invalidation, exact query returns None

12. After invalidation, similar query returns None

13. handle_query calls the agent exactly once on miss

14. handle_query returns the agent's answer on miss

15. handle_query reports from_cache=False on miss

16. handle_query returns the cached answer on hit

17. Agent NOT called again on cache hit

18. handle_query reports from_cache=True on hit

19. Watcher callback correctly invalidates via run_coroutine_threadsafe

Tests 10–12 failed on the first run. That is how the double-prefix bug was found.

Known flaws

Cache not invalidated at /plan start. The watcher handles file changes made by the executor's write_file calls. But the cache is not cleared at the start of a /plan run. If a task rewrites a file whose answer is already cached, the stale answer survives until the watcher fires — a 1.5-second window after each file write.

Redis required, no in-process fallback. build_semantic_cache() degrades gracefully when Redis is unreachable — it catches the failure and returns None, falling back to uncached agent calls. But there is no in-process fallback. If you want caching without Redis, there is currently no option.

Cache size is unbounded. TTL controls individual entry expiry, but there is no maximum entry count or LRU eviction. For long-running sessions against a large codebase, Redis memory usage grows without bound until invalidation clears the domain.

Embedding cost on every miss. A cache miss still embeds the query for the similarity search before falling through to the agent. A cold cache costs one extra embedding call per query. At text-embedding-3-small rates this is negligible, but it is nonzero.

invalidate_domain truncates at 10,000 entries. FilterQuery is called with num_results=10_000. For a domain with more than 10,000 cached entries, invalidation silently leaves some stale entries behind.

Where Part 5 leaves things

ClearCode now has a semantic cache that returns answers in milliseconds when you ask a question it has already answered, invalidates automatically when any source file changes, and isolates entries per repo and per model so nothing cross-contaminates.

The incremental indexer means repeated startups cost nothing for files that have not changed. The watcher dispatches correctly to whichever backend is configured.

One real bug was found in testing and fixed before this post was written. Five known flaws are documented above rather than hidden.

The next layer is safety and observability: guardrails on what the agent is allowed to do, structured logging that makes the tool call chain inspectable, and the beginning of the eval harness that will make improvements measurable rather than assumed.

Full source: https://github.com/f2015537/clearcode

Part 1 — Architecture before code: https://blog.divyampatro.dev/clearcode-part-1-reverse-engineering-a-coding-agent-before-writing-a-single-line-of-code Part 2 — Context layer: https://blog.divyampatro.dev/clearcode-part-2-ast-aware-indexing-vector-stores-and-hybrid-retrieval Part 3 — Memory, agent reasoning, skills, and MCP: https://blog.divyampatro.dev/clearcode-part-3-memory-agent-reasoning-skills-and-mcp Part 4 — Tasks layer: https://blog.divyampatro.dev/clearcode-part-4-autonomous-plan-execution-llm-as-judge-and-human-in-the-loop-approval

Part 3 built four capabilities: short-term memory with session persistence, multi-step agent reasoning across tool calls, a three-tier progressive skills system, and MCP integration with GitHub and filesystem servers by default.

The agent could understand a codebase, remember a conversation, apply domain knowledge, and take individual actions via tools. But each action was still in direct response to a user prompt. One question, one answer. One instruction, one file edit.

Part 4 changes the interaction model. The agent can now plan.

The tasks layer

Six new files, one new command, one qualitative shift in what the agent can do.

clearcode/tasks/
├── planner.py      # LLM to structured ExecutionPlan (Pydantic)
├── approval.py     # Human-in-the-loop review: A/M/R loop with Rich table
├── task_store.py   # SQLite WAL store with atomic state transitions
├── executor.py     # Per-task agent with least-privilege tools + LLM-as-judge
├── orchestrator.py # Dependency loop, serial dispatch, retry, re-index after run
└── recovery.py     # Resets IN_PROGRESS to PENDING on restart

The entry point is /plan . Everything else is invisible to the user until execution begins.

Step 1: Structured plan generation

planner.py calls the LLM with response_format=ExecutionPlan, which is OpenAI's structured output mode. The response is a Pydantic model, not free-form text.

ExecutionPlan contains a list of PlannedTask objects. Each task has:

• id - a unique string identifier (e.g. task_001)

• title - a short description of what the task does

• task_type - one of: design, implement, configure, review, test, integrate

• dependencies - a list of task IDs that must be complete before this task can start

• acceptance_criteria - what the LLM judge will verify after execution

The task type is not cosmetic. executor.py uses it to select a least-privilege tool set per task. A design task does not get write access. A test task does not get the ability to create new files. Each task type gets exactly the tools it actually needs.

This is what a real generated plan looks like:

> /plan create a portfolio website for a backend engineer with 8 years of
  experience using HTML, CSS, and JavaScript

Planning with LLM...

Plan: backend_engineer_portfolio
Goal: Professional portfolio site with dark theme, hero, skills, timeline,
      projects, and contact section.
Stack: HTML · CSS · JavaScript | Est: 7.5h

  ID        Type        Title                                      Depends on
  task_001  design      Define site content architecture           -
  task_002  implement   Create semantic HTML page structure        task_001
  task_003  configure   Configure design tokens and dark theme     task_002
  task_004  implement   Style hero, navigation, and header         task_003
  task_005  implement   Style content sections                     task_003
  task_006  implement   Smooth scroll and active nav behavior      task_002, task_004
  task_007  implement   Animations and scroll reveal               task_005, task_006
  task_008  implement   Responsive layout refinements              task_004, task_005
  task_009  review      Accessibility and SEO pass                 task_002, task_006
  task_010  test        Validate HTML and CSS quality              task_008, task_009
  task_011  test        Test responsive behavior                   task_010
  task_012  integrate   Final integration and delivery check       task_011

Twelve tasks. A typed dependency graph. Task types that map to different execution contexts. This is what the LLM returns as a structured Pydantic object before any execution begins.

Step 2: Human-in-the-loop approval

approval.py renders the plan as a Rich table and enters a loop:

[A]pprove / [M]odify task / [R]eject and re-plan:

Full /plan loop: goal input, plan generation, Rich table approval, serial execution, and re-index confirmation

A starts execution. M lets the user edit a specific task's fields before approving. R discards the plan and regenerates from the goal with optional guidance.

Nothing runs until the user approves. This is a design decision, not a missing feature. An autonomous agent that executes before the user has reviewed the plan is not something this architecture is built for.

The approval loop is synchronous by design. It uses rich.Prompt.ask() to block until the user responds, which means it blocks the async event loop during the wait. This is documented in CLAUDE.md as a known flaw. The correct fix is an async input primitive. For now, the blocking is predictable and bounded: it happens once, during approval, before any tool calls run.

Step 3: Persistence before execution

task_store.py writes the approved plan to SQLite before execution begins, not during and not after. WAL mode is enabled for atomic state transitions. Each task has four possible states: PENDING, IN_PROGRESS, COMPLETED, FAILED.

State transitions happen via SQL CASE expressions. The orchestrator never reads a partially updated state. If execution crashes mid-task, the store reflects the last committed state.

Config:

tasks:
  db_path: .clearcode/tasks/tasks.db

db_path is CWD-relative, which means each project directory gets its own isolated task store.

Step 4: Execution with dependency resolution

orchestrator.py loops over the task graph and dispatches ready tasks: those whose dependencies are all in COMPLETED or SKIPPED state. Tasks execute serially. This is a deliberate tradeoff. Serial execution is predictable, debuggable, and avoids tool contention between concurrent agents. Parallel execution would require explicit locking on the filesystem and the index, neither of which is implemented yet.

For the portfolio plan above:

[A]pprove / [M]odify task / [R]eject and re-plan: A

Starting: [task_001] Define site content architecture
Completed: [task_001] Define site content architecture
Starting: [task_002] Create semantic HTML page structure
Completed: [task_002] Create semantic HTML page structure
...
Completed: [task_012] Final integration and delivery check

All 12 tasks completed successfully!

Re-indexing generated files so /ask can query them...
Index updated - you can now use /ask to ask about the generated code.

After the run, the orchestrator calls the indexer with force=True on any files created during execution. This bypasses the skip-if-exists guard and ensures that /ask queries work against the generated code immediately, without restarting.

Step 5: LLM-as-judge per task

executor.py builds a fresh agent for each task with the tool set appropriate for that task's task_type. After the agent completes the task, a second LLM call evaluates the output against the task's acceptance_criteria.

The judge returns a pass or fail verdict. On pass, the orchestrator marks the task COMPLETED and moves to the next ready task. On fail, fail_task increments the retry count and returns the task to PENDING for another attempt.

Tasks retry up to three times before being marked FAILED. A failed task does not stop the run. The orchestrator marks downstream dependents as SKIPPED and continues with tasks whose dependencies are otherwise satisfied.

Step 6: Crash recovery

recovery.py runs on every startup before any command is processed. It finds all tasks in IN_PROGRESS state and resets them to PENDING. This handles the case where the process crashed mid-execution: the next run picks up from the last completed task rather than leaving the store in an inconsistent state.

/task_status shows the current state of all tasks in the active project's store.

The honest flaws

These are all documented in CLAUDE.md.

Duplicate task IDs from the planner. The LLM occasionally generates two tasks with the same id (e.g. two tasks both named task_005). task_store.py does a direct INSERT with no deduplication guard. SQLite raises UNIQUE constraint failed after the user has already approved the plan and execution has started. The entire run is lost. The fix is to validate task IDs before the INSERT, not rely on the database constraint to surface the problem after approval.

Off-by-one in fail_task. fail_task increments retry_count unconditionally before checking whether retry_count >= max_retries. A task that exhausts all three retries ends up with retry_count = 4 in the DB, not 3. The retry log is misleading but the behavior is correct: the task is still marked FAILED after the third attempt.

get_dep_results omits output_files. When a task completes, its results (the text summary) are stored alongside a separate output_files field listing files it created. get_dep_results returns only id, title, and result. It omits output_files. A downstream agent that needs to read files written by its dependency has no way to find them. It sees only the text summary, not the actual generated artifacts. This is the most consequential flaw for any plan where one task writes files that another task must read and extend.

Async event loop blocking during approval. rich.Prompt.ask() is synchronous blocking I/O called inside async def _run_async. The event loop is blocked while waiting for user input at the approval step.

MCP subprocess leak. MultiServerMCPClient is not used as an async context manager. The MCP stdio subprocesses (npx processes for GitHub and filesystem servers) are started at agent creation time but never explicitly terminated. They run for the lifetime of the REPL process and are cleaned up only when the process exits.

What the demo shows

The GIF in the README shows the full /plan loop in real time: goal input, plan generation, Rich table display, approval, serial task execution with progress markers, and the re-index confirmation. The full terminal output is reproduced verbatim in the README.

Where Part 4 leaves things

All seven built layers are now wired together and running: context, agent, memory, MCP, skills, tasks, and the underlying LLM and embedder provider abstraction. The REPL has eight commands (/ask, /plan, /task_status, /show_index, /new_session, /switch, /session, /exit).

Three layers remain: safety, freshness, and eval. The eval layer is the one I am most deliberate about. It is the thing that makes all improvements measurable rather than impressionistic.

Part 5 is not yet decided. As always, drop a preference in the comments.

Full source: https://github.com/f2015537/clearcode

Part 1 - Architecture before code: https://blog.divyampatro.dev/clearcode-part-1-reverse-engineering-a-coding-agent-before-writing-a-single-line-of-code Part 2 - Context layer: https://blog.divyampatro.dev/clearcode-part-2-ast-aware-indexing-vector-stores-and-hybrid-retrieval Part 3 - Memory, MCP, and skills: https://blog.divyampatro.dev/clearcode-part-3-memory-agent-reasoning-skills-and-mcp

Part 2 built the context layer: AST-aware indexing with tree-sitter across 15 languages, three retrieval backends switchable via a single config field, and honest documented limitations around module-level constants. ClearCode could understand a codebase.

Part 3 makes it remember, reason, and act.

Four capabilities were built across approximately 12 commits. This post covers each one with the design decisions, the implementation details, and the things that still do not work correctly. One of the flaws documented below was fixed in the final commit before this post was written. That is not accidental — it is part of how this series works.

1. Short-term memory

What was built

Two modules: memory/session.py handles session identity, memory/short_term.py handles persistence and compression.

Sessions are UUID strings, generated on first run and persisted to .memory/current_session. On startup, ClearCode reads this file and resumes the existing session. Three REPL commands surface session management:

/new_session        start a fresh conversation
/switch         resume a past session by ID
/session            show the current ID

Persistence is handled by LangGraph's SqliteSaver checkpointer, which stores the full conversation state in a SQLite database at .memory/memory.db. Every turn is checkpointed. Closing and reopening ClearCode in the same directory resumes exactly where the conversation ended.

Context compression is handled by SummarizationMiddleware, configured via config.yaml:

memory:
  db_path: .memory/memory.db
  summarize_at_tokens: 4000
  keep_last_messages: 20

Once the conversation crosses 4000 tokens, the middleware compresses everything before the last 20 messages into a summary. The summary replaces the older messages; the last 20 turns stay verbatim. Token cost is bounded across long sessions while preserving fine-grained detail of recent context.

What it looks like in practice

> /ask What is the name of the localStorage key this app uses, and what does
  each task object look like — list its exact fields?

The localStorage key is "orbitTasks".

Each task object has three fields:
  - id        unique identifier — crypto.randomUUID() with Date.now() fallback
  - title     the text of the task
  - completed boolean

> /ask Perfect. Now I want to support multiple named lists (Work, Personal).
  Using the storage key you just identified, propose the minimal change.

Instead of the single key "orbitTasks", use a dynamic key based on the list
name — "orbitTasks-Work", "orbitTasks-Personal", and so on.

Two functions need to change:

    function saveTasks(listName) {
        localStorage.setItem(`orbitTasks-${listName}`, JSON.stringify(tasks));
    }

    function addTask(title, listName) {
        tasks.unshift({
            id: crypto.randomUUID ? crypto.randomUUID() : String(Date.now()),
            title,
            completed: false
        });
        saveTasks(listName);
        renderTasks();
    }

Everything else — toggleTask, deleteTask, renderTasks — remains unchanged.

The second answer picks up "orbitTasks" from the first turn without re-searching. The agent read it from the conversation history that the checkpointer had stored.

On documenting flaws

The original implementation of get_checkpointer() opened a new SQLite connection on every call and never closed it. Connections accumulated across the session. It worked — SQLite tolerates concurrent connections to the same file — but the connection count grew without bound.

I documented this in CLAUDE.md. Writing it down made it impossible to publish with it in that state. The fix — a module-level singleton connection — went in as the final commit before this post was written.

This is the part of building in public that people do not always talk about: the act of documenting a flaw clearly enough that it can be explained to a reader is also the act of understanding it clearly enough to fix it. CLAUDE.md has become a forcing function.

The remaining known flaws

get_session_history() is defined in short_term.py but never called. Dead code. Left in because it documents the interface — the function that would be called if session history needed to be inspected programmatically. It should either be wired up or deleted.

memory.db_path is CWD-relative. Running ClearCode from different directories creates separate .memory/ folders. This is the right default behaviour for a per-project tool, but it means /switch only works if you launch from the same directory where the session was created.

show_index in the Qdrant backends hardcodes limit=1000. For large codebases this silently truncates the displayed index. The correct approach is pagination or a configurable limit.

2. Agent reasoning

Part 2 built a retrieval system that returned relevant chunks. Part 3 wires the agent to use those chunks as the starting point for multi-step reasoning across tool calls.

The agent chains tools: retrieve context, read a specific file, follow a function reference, read that file, return a complete picture. Retrieval is no longer the end of the pipeline — it is the beginning.

A real example, run against a live codebase:

> /ask Trace what happens from the moment a user submits the add-task form to
  when the new task appears on screen. Give me the exact function call chain.

Here's the detailed trace:

1. Event Listener and Form Submission
   The process begins when an event listener captures the form submission event,
   prevented from its default action via event.preventDefault(). Defined in
   todo-app/js/scripts.js, lines 102-115.

2. Task Title Extraction and Verification
   The task title is extracted with taskInput.value.trim(). If the title is
   empty, focus returns to the input field and the function exits.

3. addTask(title) — lines 73-82
   Creates a new task object and unshifts it to the front of the tasks array.
   id is generated with crypto.randomUUID() or String(Date.now()) as a fallback.
   Calls saveTasks() then renderTasks().

4. saveTasks() — lines 14-16
   Serialises the tasks array to localStorage under the key "orbitTasks".

5. renderTasks() — lines 38-71
   Clears the current task list (taskList.innerHTML = ''), re-renders every
   visible task as a 
 with a check button, title span, and delete button,
   then calls updateStats() to refresh the counters.

6. Interface Update
   Input field is cleared and refocused. The empty-state element is toggled
   based on whether any visible tasks remain.

The agent read scripts.js via the filesystem MCP server, found the exact line ranges, and traced the full call chain — not from memory, but from the actual file.

This is the difference between a retrieval system and an agent. Retrieval returns relevant text. An agent uses that text to decide what to read next.

3. Skills

The problem skills solve

A coding agent pointed at a machine learning repository and asked to add a feature should apply different standards than the same agent pointed at a React component library. The ML repo might have conventions around reproducibility, experiment tracking, and data pipeline structure that the React repo does not. The agent has no way to know this from the code alone.

Skills are the mechanism for injecting that domain knowledge without paying for it on every query.

Three-tier progressive disclosure

Skills use a three-tier loading model to keep token costs low while making full expertise available on demand.

Tier 1 — always in the system prompt. A compact index of skill names, descriptions, and when_to_use trigger keywords. The agent sees this on every query, paying only a few tokens per skill regardless of how detailed the skill body is. Twenty skills cost the same as zero in token terms until one is triggered.

Tier 2 — loaded on match. When the user's request matches a skill's trigger keywords, the agent calls load_skill(name) to retrieve the full SKILL.md body — complete instructions, decision rules, code templates, and rationale.

Tier 3 — fetched individually. Skills can ship support files (scripts, templates, reference documentation). After loading Tier 2, the agent sees a listing of available support files and reads only the ones the task actually requires.

Skills live in .clearcode/skills//SKILL.md inside whatever project ClearCode is pointed at. They are project-specific, versioned alongside the code, and visible in the repository.

What it looks like in practice

> /ask I have a 40-page financial report PDF with embedded tables on every
  page. I need to extract all of them into a single pandas DataFrame.

To extract tables from a multi-page PDF into a single DataFrame, use pdfplumber:

    import pdfplumber
    import pandas as pd

    with pdfplumber.open("your_report.pdf") as pdf:
        all_tables = []
        for page in pdf.pages:
            tables = page.extract_tables()
            for table in tables:
                if table:
                    df = pd.DataFrame(table[1:], columns=table[0])
                    all_tables.append(df)

    combined_df = pd.concat(all_tables, ignore_index=True)

pdfplumber preserves table geometry better than pypdf for dense financial
layouts. If a page has no discernible table structure, extract_tables()
returns an empty list — the guard (if table) handles that safely.

The pdf skill was loaded from .clearcode/skills/pdf/SKILL.md. The agent did not guess at the library choice — it followed instructions that encode a decision already made: pdfplumber for dense financial layouts, with the reasoning documented in the skill rather than inferred from the query.

The skill trigger change worth naming

An early version of the skill trigger instruction was suggestive: "if the request matches, consider loading the skill." Under ambiguous queries, the agent sometimes retrieved a result without loading the relevant skill first.

The trigger was rewritten to be directive: "if the request matches these keywords, you must call load_skill before responding." One word in the system prompt, one word in behaviour: the agent now reliably loads domain knowledge before acting on matched requests.

4. MCP

Two servers ship by default

Two MCP servers are configured in clearcode_mcp_servers.json at launch:

• A GitHub server. Public repos work unauthenticated. Add GITHUB_TOKEN to .env for private repos and write access.

• A filesystem server scoped to the current working directory. The agent can read and write files within the project.

Adding a new server — Jira, Linear, Slack, a database — is a single entry in the JSON config. The agent picks it up at next startup with no code changes required.

> /ask Go to the GitHub repo f2015537/clearcode and look at the last 5 commits
  on main. Summarise what has been built layer by layer.

Here is a summary of the last 5 commits on the main branch:

- cf98612 — "Add agent, skills, and memory examples to README"
  Documents the system's live capabilities with real output — code navigation,
  skill-driven answers, and session memory continuity.

- 6dd6b3e — "Update CLAUDE.md and README to reflect current project state"
  Marks agent, memory, MCP, and skills layers complete. Removes the local
  filesystem tools now covered by the filesystem MCP server.

- d9db21f — "Add filesystem MCP server; remove local filesystem tools"
  Shifts filesystem operations to a centralised MCP server scoped to CWD —
  same capabilities, no bespoke tool code to maintain.
...

The agent called GitHub's list_commits tool, retrieved live data from the API, and composed the summary. No local git history involved.

The architectural change that MCP made possible

Early in Part 3, I wrote bespoke filesystem tools: read_file, write_file, append_file, delete_file, list_directory, file_exists. Each was a Python function registered with the agent.

Once the MCP filesystem server was added, those tools became redundant. The filesystem server provides the same capabilities through the same protocol the GitHub server uses. The bespoke code was removed.

This is the point of MCP: not just adding capabilities, but changing the cost structure of adding them. A new capability used to mean writing a Python function, handling errors, managing logging, and registering the tool with the agent. Now it means one entry in a JSON file. The agent does not care whether a capability comes from a local function or an MCP server — it calls tools the same way either way.

Where Part 3 leaves things

The four active capabilities in ClearCode are now: retrieval (dense, sparse, hybrid), agent reasoning (multi-step tool use), skills (three-tier progressive disclosure), and MCP (GitHub and filesystem by default, extensible via config).

The known flaws are documented in CLAUDE.md. The connection leak in get_checkpointer() was fixed before this post was published. The dead code in get_session_history() and the hardcoded limit=1000 in show_index are still there — named, not hidden.

Part 4 is not decided yet. The remaining layers are safety, freshness, observability, and eval. If you have a preference on what comes next, drop it in the comments. I read every one.

Full source: https://github.com/f2015537/clearcode

Part 1 - Architecture before code: https://blog.divyampatro.dev/clearcode-part-1-reverse-engineering-a-coding-agent-before-writing-a-single-line-of-code Part 2 - Context layer: https://blog.divyampatro.dev/clearcode-part-2-ast-aware-indexing-vector-stores-and-hybrid-retrieval

Part 1 covered the architecture before writing any code: the folder structure, the reasoning behind each layer, and the open questions I did not have answers to yet. The context layer was the piece I expected to spend the most time on.

That turned out to be correct. This post covers what I built there and what I learned.

The end state: a working RAG-powered code assistant that runs as a local REPL. Point it at any codebase and ask questions in plain English. It indexes the source files, embeds them into a vector store, and answers using retrieved context.

The chunking decision

The first and most consequential decision in any RAG system for code is how to chunk the source files.

The most common approach in tutorials is RecursiveCharacterTextSplitter: split on a character count, with some overlap. This is the right tool for prose. It is the wrong tool for source code.

Character splits break functions at arbitrary boundaries. They separate docstrings from the function bodies they describe. They mix unrelated code into the same chunk because two functions happened to be close together in the file. The embedding model then has to make sense of a fragment that has no semantic coherence, and retrieval degrades accordingly.

The alternative is structure-aware chunking: use the AST to identify the natural boundaries in the code, and make each named block its own chunk.

I used tree-sitter for this. The code_parser.py module walks the AST of each source file and extracts top-level named blocks - functions, classes, methods - as atomic chunks. For a Python file with 10 functions, you get 10 chunks, each containing the signature, the docstring, and the full body of one function. For a JavaScript file with classes and methods, each method is one chunk.

Two decisions within the chunking strategy are worth explaining:

No nested functions indexed separately. The _walk function stops recursing the moment it hits a named block node. A nested function is included in its parent's chunk, not indexed as a separate unit. Indexing nested functions separately would duplicate context - the outer function already contains the inner one - and produce misleading chunk boundaries where the inner function appears without its surrounding context.

Fallback for non-code files. Text files, markdown, config files, and anything without meaningful AST structure fall back to a sliding window chunker. The two strategies are composable: the AST chunker handles source files, the sliding window handles everything else.

The byte offset bug

This is the detail that cost me the most time.

tree-sitter returns byte offsets when you ask it where a node starts and ends in the source file. Most Python source files are ASCII, so byte offsets and character offsets are identical, and you never notice the difference. The moment a source file contains a multi-byte character - a Unicode string literal, a non-ASCII variable name, a comment with an emoji - byte and character offsets diverge.

If you slice source on character indices using a byte offset, you get the wrong content. The slice is off by however many extra bytes appeared before that point in the file. There is no error. The chunk is silently wrong.

The fix is simple once you understand it: encode the source to bytes first, slice on byte indices, then decode the result. All source slicing in code_parser.py now happens this way.

source_bytes = source.encode("utf-8")
chunk_content = source_bytes[start_byte:end_byte].decode("utf-8")

This is the kind of correctness detail that only surfaces when you run against real-world codebases rather than toy examples. The test suite now includes a file with multi-byte characters to catch regressions.

Three retrieval backends

The retrieval layer sits behind a factory interface. Switching backends is a single field in config.yaml:

vector_store:
  retrieval_mode: hybrid  # dense | sparse | hybrid

Three backends are implemented:

Dense (ChromaDB or Qdrant) - embeds each chunk with text-embedding-3-small and retrieves by cosine similarity at query time. Best for conceptual and design questions where the question and the answer use different vocabulary.

Sparse (BM25 via fastembed) - indexes each chunk as a bag of terms and retrieves by keyword overlap. No embedding model required. Best for exact identifier lookups, error string searches, constant name lookups.

Hybrid (Qdrant) - stores both a dense embedding and a sparse BM25 vector per chunk, and fuses the two scores at query time. Best when a question mixes exact identifiers with conceptual reasoning.

What the difference looks like in practice

These are real outputs from all three retrieval modes, run against the ClearCode codebase itself.

Question 1: Conceptual design question

> /ask what is the philosophy behind not indexing nested functions?

The design philosophy is outlined across code_parser.py and CLAUDE.md. The _walk
function (lines 113–135) stops recursing the moment it hits a named block node,
ensuring that each top-level function or class is indexed as a single coherent
unit. Indexing nested functions separately would duplicate context — the outer
function already contains the inner one — and produce misleading chunk boundaries.

This also simplifies the stable chunk ID scheme (source::name::start_line) used
for idempotent upserts in ChromaDB, where collisions on nested definitions with
the same name would silently overwrite chunks.

Dense retrieval returned the design rationale from code_parser.py and CLAUDE.md with full context - the _walk function, the reasoning about semantic coherence, and the note about stable chunk IDs.

Sparse retrieval on the same question retrieved auth-related chunks with no connection to the question. "Philosophy", "nested", and "functions" do not co-occur as exact terms in the relevant source locations - the design rationale is written in prose that uses different vocabulary from the question.

Question 2: Exact identifier lookup

> /ask what value does CHUNK_OVERLAP have and where is it defined?

CHUNK_OVERLAP is used in the _sliding_window function in
clearcode/context/indexers/code_parser.py at line 154, where it controls the
step size between successive window positions as CHUNK_SIZE - CHUNK_OVERLAP.

Sparse retrieval found the exact usage site in code_parser.py at line 154, where CHUNK_OVERLAP appears as a literal identifier.

Dense retrieval on the same question found "10-line overlap" in CLAUDE.md where the concept is described in prose - right answer, wrong location. No line reference, no code context. The embedding captured the concept but not the identifier.

Question 3: Mixed identifier and conceptual

> /ask how does _extract_name fall back when no identifier node is found,
  and why does that cause a problem with BLOCK_NODE_TYPES like arrow_function?

_extract_name (code_parser.py, lines 138–143) iterates over the children of a
block node looking for a child with type "identifier", "name", or
"property_identifier". If none is found, it returns node.type as a fallback —
so for an arrow_function node with no direct identifier child, the chunk name
becomes "arrow_function".

This is a problem because the identifier for an arrow function (const x = () => {})
lives on the parent variable_declarator, not on the arrow_function node itself.
The fallback produces a generic, non-unique name that collides across all
anonymous arrow functions in the same file, and makes retrieval results
meaningless for JS codebases heavy in functional patterns.

Hybrid retrieval retrieved the full picture: the fallback implementation from code_parser.py lines 138-143, and the parent/child AST relationship explanation that makes the arrow_function case a problem.

Dense alone retrieved the conceptual relationship but not the exact implementation. Sparse alone pinned the identifiers but missed the reasoning. Hybrid composed both.

The honest limitation

Module-level constants defined between functions do not fall inside any named AST node. They are not indexed by the AST chunker.

Ask for the value of CHUNK_OVERLAP defined at module level (lines 44-47 in code_parser.py) and neither dense nor sparse retrieval will find it. Dense found the value in documentation prose. Sparse found the usage site. Neither found the definition.

This is not a bug I missed. It is a tradeoff with a known fix: index module-level statements (constants, imports, top-level assignments) as an additional chunk type per file. That is the next improvement to the context layer.

It is documented explicitly rather than papered over because a system that knows what it does not know is more useful than one that guesses.

Multi-project use

ChromaDB stores its index in a .chromadb/ folder inside whichever directory you run clearcode from. Each project automatically gets its own isolated index.

Qdrant uses a single named collection by default. Running clearcode in a new project reuses the existing collection rather than re-indexing. For multi-project use, ChromaDB is the recommended starting point.

What comes next

Part 3 is taking shape. The current plan is to focus on memory and tools - giving the agent the ability to remember context across sessions and take real actions on the codebase rather than just answering questions about it.

But I am genuinely curious what you would prioritize. If you were building this, what would you tackle next? Drop a comment - I read every one, and the best suggestions go directly into the roadmap.

Follow along at https://blog.divyampatro.dev/series/clearcode

Full source: https://github.com/f2015537/clearcode Part 1: https://blog.divyampatro.dev/clearcode-part-1-reverse-engineering-a-coding-agent-before-writing-a-single-line-of-code

I use Claude Code every day. For the longest time it felt like a black box.

I type a prompt. Code appears. Files change. Tests run. Pull requests get written. I have no real idea what is happening between my input and those outputs.

That is not a complaint. It is a problem I want to solve, for myself, by doing the only thing that has ever actually worked for me when I want to understand something: building it from scratch.

ClearCode is my attempt to reverse engineer a production-grade autonomous coding agent. Not to beat Anthropic. Not to ship a competing product. To understand how these systems actually work under the hood, and to document every decision and dead end publicly so anyone else curious about the same question has a ground-up reference to learn from.

This is part 1: the architecture before the code.

What a coding agent actually needs to do

Before designing anything, it helps to think clearly about what the problem actually is. A coding agent is not just an LLM with a text editor. It needs to:

• Understand the codebase it is working in, not just the file you are currently looking at

• Know which parts of that codebase are relevant to the current task

• Keep that understanding current as files change

• Take actions: read files, write files, run commands, search, navigate

• Reason about multi-step problems where the right next action depends on what previous actions returned

• Know when it is done, and when it is stuck

• Not do something irreversible without checking first

That list is not exhaustive. But it is enough to start making architectural decisions.

The folder structure

The first design decision was how to divide the problem into layers. Here is where I landed:

clearcode/
│
├── context/                     # Context layer
│   ├── indexers/                # Build indexes over the codebase
│   ├── retrievers/              # Query those indexes
│   └── memory/                  # Short-term and long-term memory
│
├── agent/                       # Agent reasoning layer
│
├── llm/                         # LLM provider abstraction
│
├── tools/                       # Individual tool functions
│
├── mcp/                         # MCP server integrations
│
├── skills/                      # Higher-level composed capabilities
│
├── safety/                      # Safety layer
│
├── freshness/                   # Freshness layer
│
├── observability/               # Observability layer
│
└── eval/                        # Evaluation layer
    ├── datasets/                # Shared golden dataset
    ├── retrieval/               # Recall@k, MRR, NDCG, Hit Rate
    ├── context/                 # Context precision, context recall
    ├── generation/              # Faithfulness, answer relevancy (RAGAS)
    └── agent/                   # Task success rate, step accuracy

Let me explain the reasoning behind each layer.

The context layer

This is where I expect to spend the most time and learn the most. A coding agent's understanding of the codebase it is working in is the ceiling for everything else. If the agent retrieves the wrong files, or retrieves the right files but in the wrong form, no amount of reasoning quality will save the output.

The context layer has three parts:

Indexers build representations of the codebase that can be searched. This is where the chunking strategy decisions from my earlier RAG work become directly relevant. For code, AST-based chunking (preserving functions and classes as atomic units) is almost certainly better than character-based chunking. I expect to spend significant time here.

Retrievers query those indexes. From my RAG case study, I know that hybrid retrieval (semantic plus BM25) handles the full range of query types better than either alone. An exact function name lookup and a conceptual query about permission logic need different retrieval strategies.

Memory is the layer I am most excited and most uncertain about. Short-term memory needs to track what has happened in the current session. Long-term memory needs to persist things the agent should remember across sessions. From my LLM memory patterns work, I know the scoping distinction that matters: STM is per-thread, LTM is per-user.

The agent layer

The agent is the reasoning loop. It decides what to do next, calls a tool, observes the result, and decides what to do after that.

I do not have a firm opinion on the agent architecture yet. The main question is how much structure to impose on the loop: a free-form ReAct-style loop where the LLM decides everything, a more structured plan-then-execute pattern, or something in between. I expect this to be one of the most consequential decisions in the whole project.

The LLM provider abstraction

This layer exists so that the rest of the system does not know or care which model it is using. From my earlier work with LiteLLM, I know that decoupling the model from the agent logic means you can benchmark different models against the same tasks and switch based on cost, latency, or capability without touching any other layer.

Tools

Tools are the atomic capabilities the agent can call: read a file, write a file, run a terminal command, search the codebase, look something up. Each tool is a function with a well-defined input and output.

The principle here is the same one I applied in the MCP server project: write the tool logic once, expose it cleanly, and let the agent layer decide when and how to use it.

MCP server integrations

MCP is the protocol that makes tools portable across agents and frameworks. Rather than binding tools directly to a single agent, wrapping them in an MCP server means they can be used by any MCP-compatible client.

I will wire ClearCode's tools into an MCP server so they are reusable outside the agent loop itself.

Skills

Skills are composed capabilities built on top of individual tools. "Refactor this function" is a skill that might call read-file, analyse-code, write-file, and run-tests in sequence. "Add a feature" is a skill that might involve planning, searching, writing, and testing.

The distinction between a tool and a skill is the level of composition. Tools are atomic. Skills are workflows.

Safety

A coding agent that can write and execute code needs guardrails. The safety layer is where I will enforce things like: no running destructive commands without confirmation, no writing outside the project directory, no making network calls without explicit permission.

I do not have a concrete plan for this layer yet. It is one of the genuinely hard problems in autonomous agent design.

Freshness

The codebase changes as the agent works. A file the agent indexed at the start of a session may be different by the time the agent tries to use that index later in the same session. The freshness layer is responsible for detecting staleness and triggering re-indexing.

This is a problem I have not seen addressed clearly in most agent tutorials. I suspect it is more important in practice than the literature suggests.

Evaluation

This is the layer I am most deliberate about upfront, because it is the one that gets skipped most often in agent projects. The eval layer has four sub-layers, each measuring a different dimension:

Retrieval: Recall@k, MRR, NDCG, Hit Rate. Does the context layer return the right files?

Context: Context precision and context recall. Is the retrieved context relevant, and is it complete?

Generation: Faithfulness and answer relevancy via RAGAS. Does the agent's output actually reflect what it retrieved, and does it answer the question?

Agent: Task success rate and step accuracy. Does the agent complete the task correctly, and does it take reasonable steps to get there?

Without this layer, I cannot tell whether the changes I make to the context pipeline or the agent loop are actually improving anything.

What I do not know yet

Quite a lot. The folder structure above represents the shape of the problem as I understand it today. Some of those layers will turn out to be more complex than I expect. Some will be simpler. Some of the things I think I need will turn out to be unnecessary. Some things I have not thought of yet will turn out to be critical.

That is the point of building this in public. The unknown unknowns are more interesting than the known unknowns, and the only way to find them is to start.

What comes next

Part 2 will cover the context layer: how to build an index over a codebase, what chunking strategy to use for code specifically, and how to query it. That is where the building actually starts.

Source code (work in progress): https://github.com/f2015537/clearcode

If you have ever wondered how tools like Claude Code or Cursor work under the hood, follow along. I am going to find out.

Most explanations of agent memory start at the wrong altitude. They jump straight to "use a vector database" without explaining what problem that solves, or why a vector database is sometimes the wrong tool entirely.

This post walks through memory the way I built it: as a progression of seven small, runnable scripts, starting from a completely stateless LLM call and ending at a two-layer system with separate short-term and long-term memory. Each step introduces exactly one new concept. By the end, the line between short-term and long-term memory stops being fuzzy and becomes a concrete architectural decision: what is scoped to a thread, and what is scoped to a user.

All code referenced here is in the open-source repo at the end of this post. Every script is self-contained and runnable on its own.

1. Stateless: the baseline problem

The simplest possible setup is a single call to an LLM with no history attached. Tell it your name, ask it your name in the next message, and it has no idea. Every call starts from zero.

This is not a bug. It is the literal absence of memory, and it is the baseline every other pattern in this guide is solving for.

2. Short-term memory in RAM

The most direct fix: keep the full conversation in a Python list, and send that list with every call to the model.

history.append({"role": "user", "content": user_input})
response = llm.invoke(history)
history.append({"role": "assistant", "content": response.content})

Now the model can recall earlier turns, because it is literally being shown them every time. The catch: this history lives in process memory. Restart the script and it is gone. This pattern is memory in the loosest sense, useful for a single uninterrupted session, useless across sessions.

3. Persistence and threads

The next step is durability. Write the conversation history to SQLite after every turn, and reload it on startup.

This alone solves the restart problem. The second piece is the thread_id: rather than one global history, each thread_id is its own isolated conversation. Multiple conversations can live in the same store without bleeding into each other, and you can switch between them on demand.

At this point memory has gone from "exists only while the process is running" to "exists independently of the process, scoped to a conversation."

4. Sliding window

A real conversation grows without bound, and sending the full history on every call gets expensive fast, both in tokens and in latency. The sliding window pattern fixes the cost problem directly: a @before_model middleware hook trims the stored history down to the last N messages right before every call to the model.

@before_model
def trim_to_window(messages):
    return messages[-WINDOW:]

Token cost is now bounded, regardless of how long the conversation runs. The tradeoff is blunt: anything said more than N turns ago is permanently inaccessible to the model. If the user mentioned an important constraint 10 turns ago and the window is 4, that constraint is gone.

This is the right tool when recent context is what matters and the conversation is expected to be short or self-contained.

5. Summarization

Summarization is the more graceful version of the same idea. Instead of discarding old messages outright, a @before_model middleware hook compresses them into a short summary once the stored message count crosses a threshold, while keeping the most recent few messages verbatim alongside that summary.

@before_model
def summarize_if_needed(messages):
    if len(messages) > SUMMARIZE_AFTER:
        old, recent = messages[:-KEEP_RECENT], messages[-KEEP_RECENT:]
        summary = llm.invoke([SUMMARY_PROMPT, *old])
        return [summary_message(summary), *recent]
    return messages

This retains the gist of everything that happened earlier in the conversation, at the cost of losing fine-grained detail. A sliding window would have kept the literal text of the last N turns and nothing before that; summarization keeps a compressed signal of everything before that, plus the literal text of the most recent few turns.

Neither pattern is strictly better. Sliding window is cheaper and simpler. Summarization preserves more of the conversation's shape, at the cost of an extra LLM call to generate the summary.

6. Long-term memory: the user profile

Every pattern so far is still short-term memory. It lives within one conversation thread and, even with persistence, none of it is shared if the user starts a different thread.

Long-term memory is a different scope entirely: durable facts about a user that should be available no matter which conversation they are currently having. The pattern here uses a second LLM call after every turn to extract durable facts (name, job, stated preferences) and write them to a key-value store (LangGraph's SqliteStore):

facts = extract_facts_llm.invoke([extraction_prompt, *recent_turns])
store.put(namespace=("user", user_id), key="profile", value=facts)

On the next session, even a completely separate thread, those facts are loaded from the store and injected into the system prompt. This is the first pattern in the series where memory survives across sessions that have no shared conversation history at all.

7. Combining short-term and long-term memory

The full architecture uses both layers together, each with a different scope:

• Short-term memory, via SqliteSaver: conversation history, scoped to a thread_id. Restored automatically on reconnect to the same thread.

• Long-term memory, via SqliteStore: durable behavioral rules, scoped to a user_id. The agent saves rules autonomously via a save_rule tool, and those rules apply to every future session for that user, regardless of thread.

The clarifying detail: open a new thread for the same user, and the conversation history resets, but the rules that user taught the agent in a previous thread are already active. Switch to a different user entirely, and that user inherits none of the first user's rules. Each user's long-term memory is fully isolated.

This is the insight that took the fuzziness out of "short-term vs long-term memory" for me. They are not the same kind of memory at different durations. They are scoped along different dimensions: STM resets per thread, LTM persists per user across every thread that user ever opens.

Choosing a pattern

Further reading

This series draws directly on LangChain and LangGraph's memory primitives. For the underlying API reference:

• LangChain short-term memory

• Before-model middleware

• Middleware overview

• Message summarization

• LangChain long-term memory

• LangGraph stores

Conclusion

Memory in LLM applications is not one feature, it is a set of independent tradeoffs: durability versus simplicity, recall versus token cost, thread-scoped versus user-scoped. Building each pattern in isolation, in order, makes those tradeoffs visible rather than hidden behind a single "add memory" abstraction.

Full source, all 7 scripts, self-contained and runnable: https://github.com/f2015537/llm-memory-patterns

PageIndex, an open-source project by VectifyAI, is currently the #1 trending repository on GitHub with over 30,000 stars. Its central claim is direct and worth taking seriously: similarity is not relevance, and most RAG systems are optimizing for the wrong thing.

This post is my breakdown of how PageIndex works, why the architecture makes sense for a specific class of documents, and where I think the tradeoffs are understated. All credit for the underlying system goes to VectifyAI. Links to the original repository are at the end.

The problem PageIndex is responding to

Traditional RAG chunks a document into fixed-size windows, embeds each chunk into a vector space, and retrieves whichever chunks are closest to the query's embedding at inference time. This works reasonably well for short, homogenous documents.

It breaks down for long, structured professional documents: financial reports, regulatory filings, legal contracts, technical manuals. A chunk can be lexically and even semantically similar to a query while still being the wrong section to answer it, because professional documents encode relevance through structure (which section, which subsection, which exception clause) that flat vector similarity cannot see.

This is the same failure mode I found in my own RAG case study on code search: a docstring containing the actual answer got split away from its function body by character-based chunking, and the embedding retriever never found it. PageIndex generalizes that observation into a different default: do not chunk at all.

How PageIndex works

The system has two stages.

Stage 1: Build a hierarchical tree index.

Instead of slicing a document into arbitrary chunks, PageIndex parses it into a tree structure resembling a table of contents. Each node has a title, a node ID, a start and end page range, and an LLM-generated summary of that section's content. Nested sections become nested tree nodes.

This tree is generated once per document, using an LLM (GPT-4o by default in the open-source version) to identify section boundaries and write summaries.

Stage 2: Reasoning-based retrieval through tree search.

At query time, instead of embedding the query and running a similarity search, an LLM reasons over the tree. It reads node summaries, decides which branches are worth exploring further, and navigates down toward the most relevant leaf nodes. This is explicitly compared to AlphaGo in the project's README: tree search guided by a learned evaluator, applied to documents instead of a game board.

The result is retrieval that is traceable. You can see exactly which path through the document tree the LLM took to arrive at an answer, and which page ranges it drew from, rather than an opaque list of cosine similarity scores.

The evidence

VectifyAI cites a system called Mafin 2.5, built on PageIndex, that scored 98.7% on FinanceBench, a benchmark for financial document question answering. They claim this outperforms vector-based RAG approaches on the same benchmark.

This is a strong, specific, falsifiable claim, which is more than most RAG tooling marketing offers. It is worth treating as a meaningful signal rather than a definitive conclusion. Benchmark performance on FinanceBench specifically does not automatically generalize to every document type, especially ones with less clean hierarchical structure than financial filings.

Where the architecture makes sense

PageIndex's design assumptions hold up well for documents that already have strong hierarchical structure: SEC filings, regulatory text, academic textbooks, legal contracts, technical manuals. These documents are written with a table of contents in mind. A tree index captures structure that genuinely exists in the source material rather than imposing artificial structure through chunk boundaries.

For this category of document, removing chunking entirely is not a workaround, it is the more accurate representation of how the information is actually organized.

Where I would want more detail before adopting it in production

Latency and cost at query time. Vector similarity search is a single embedding call plus a fast nearest-neighbor lookup. Tree search requires the LLM to reason over multiple nodes, potentially across multiple reasoning steps per query. This trades a cheap, fast retrieval step for a more expensive, slower one. Whether that tradeoff is worth it depends entirely on your latency budget and query volume.

Documents without clean hierarchical structure. Financial filings and legal documents are well-suited to this approach because their structure is the point. Documents that are unstructured prose, internal Slack threads exported to PDF, meeting transcripts, casual documentation, do not have a meaningful tree to build, and PageIndex's self-hosted version's reliance on standard PDF parsing rather than the enhanced OCR pipeline in the paid tier may struggle to extract a useful structure from these.

Scale beyond a single document. The open-source repo and its core tree search operate on a single document. VectifyAI's newer "PageIndex File System" extension addresses multi-document reasoning, but that capability sits in the commercial layer, not the open-source core covered in this repository.

The broader pattern worth noticing

This project is evidence of a shift in how the RAG community is thinking about chunking. My own case study on code RAG found that chunking strategy mattered more than retrieval method when the chunking destroyed semantically important context. PageIndex is a more radical version of the same insight: rather than finding the right chunk size, remove the chunking step and index the structure that already exists in the document.

Whether vectorless reasoning-based retrieval becomes a mainstream pattern or remains well-suited to a specific category of structured professional documents is an open question. The architecture is a serious, well-evidenced answer to a real limitation in standard RAG, and it is worth understanding even if you do not adopt it directly.

Credit and links

All credit for PageIndex goes to VectifyAI and the project's contributors.

Original repository: https://github.com/VectifyAI/PageIndex Project homepage: https://vectify.ai/pageindex

If this approach is useful to you, consider starring the original repository.

Most RAG discussions treat chunking and retrieval as separate concerns. Pick a chunk size, pick a retriever, tune k, done. What is less discussed is how much these decisions interact - specifically, how a bad chunking strategy can make retrieval irrelevant before it even runs.

This case study runs the same three questions through four different RAG configurations on the same Python codebase. Semantic RAG with character chunking, semantic RAG with AST-based chunking, lexical RAG with BM25, and hybrid RAG with Reciprocal Rank Fusion. Same LLM. Same codebase. Only the chunking strategy or retrieval method changes.

The finding: chunking strategy mattered more than retrieval strategy. AST-based chunking won or tied on every query. The retrieval method only became decisive after bad chunking had already destroyed the relevant context.

The setup

The target codebase is a small Python task-management application with auth, permissions, billing, notifications, audit logging, and a recurring-task scheduler. Thirteen files. Realistic enough to produce interesting retrieval failures.

Every agent is built the same way: a LangChain agent with a single search_codebase tool, instructed to always search before answering and to cite specific files and functions.

Three questions were chosen to stress different aspects of retrieval:

Q1 tests whether a permission rule documented only in a docstring survives chunking. Q2 tests the lexical gap: the question uses different vocabulary from the code. Q3 tests exact identifier lookup, the easiest case for any retriever.

The four strategies

Script 1: Semantic RAG, character-based chunking RecursiveCharacterTextSplitter at 256 chars / 32 overlap. OpenAI embeddings. Chroma vector store. 173 chunks from 13 files. This is the baseline most RAG tutorials implement.

Script 3: Semantic RAG, AST-based chunking Python's ast module parses each file into a syntax tree. Each top-level class or function becomes one chunk - signature, docstring, and body together. 28 chunks from the same 13 files. Far fewer, far larger, semantically coherent units.

Script 5: Lexical RAG, BM25 Same character-based chunks as Script 1 (173 chunks), retrieved with rank_bm25 instead of embeddings. No vector store. No embedding API calls.

Script 6: Hybrid RAG, Reciprocal Rank Fusion LangChain EnsembleRetriever fusing semantic and BM25 retrievers at equal weights. Same character-based chunks. RRF re-ranks by combining the ranked lists from both retrievers.

Results

Four RAG strategies, same codebase, same questions - results from the case study

Q1: The docstring question

The answer ("admin only") lives entirely in the docstring of TaskService.assign_task:

def assign_task(self, token: str, task_id: int, assignee_id: int) -> Task:
    """
    Reassign a task to another user.
    Requires task:assign permission (admin only).
    """

With 256-char chunking, this docstring and the surrounding permission logic are split across multiple character windows, mixed with fragments from main.py seed data and database.py helpers. The embedding retriever returned chunks that mention assignment but never the docstring itself. The agent hedged: "may be admin-only, but not definitively stated."

BM25 and hybrid fared worse. Their top chunks were dominated by unrelated auth.py text, and both returned "I could not find that in the codebase."

With AST-based chunking, the entire TaskService class including that docstring is one chunk. The retriever returns it directly. The agent quotes the docstring verbatim and answers with full confidence.

This is a chunking effect, not a retrieval effect. The retriever in Script 1 was not wrong - it returned the most semantically similar character windows. The relevant information had simply been destroyed by the chunking step before retrieval ran.

Q2: The lexical gap question

"If a teammate's work gets handed off to someone else, how do they find out?"

The code says "assignee" and "on_task_assigned". The question says "teammate" and "handed off". These share almost no tokens.

BM25 matched on "task" and "find" and returned irrelevant auth.py and integrations chunks. It has no way to bridge the gap between "handed off" and "on_task_assigned" without semantic understanding. Result: failed.

Both embedding retrievers found the correct chunk because "hand off work to a teammate" and "assign a task to a user" are close in embedding space even without shared vocabulary. The semantic representation captures the conceptual similarity that lexical matching cannot.

Hybrid RRF also succeeded. Its semantic half compensated for its BM25 half, pulling the correct chunk into the top-k results.

Q3: The exact identifier question

"What exception does check_task_limit raise, and when?"

Every strategy got this right. When the query contains the literal function name from the code, BM25's exact term matching is as effective as embeddings and far cheaper - no embedding API calls required.

This is BM25's strongest use case: exact identifier lookups, error string searches, config key lookups. For these queries, adding a semantic retriever adds cost without adding quality.

Metrics: the precision/recall tradeoff

Script 7 runs a golden set of 6 queries with known-relevant documents against an InMemoryVectorStore at k values of 1, 3, and 5:

Recall climbs toward 1.0 as k grows. Precision falls. This is the expected tradeoff: retrieving more chunks eventually finds every relevant document, but an increasing share of retrieved chunks are irrelevant noise the LLM has to filter out.

The failure at k=3 flags one query whose answer requires synthesising information from two documents. Only one of the two was retrieved. This is a common failure mode in RAG: questions that need multi-document synthesis are harder than questions with a single-document answer.

The decision framework

What comes next

The best configuration this case study points to - AST-based chunking combined with hybrid retrieval - was not wired up in this repo. It is the natural next step, and the most likely setup to handle the full range of query types the three test questions represent.

tree-sitter is worth exploring as an alternative to Python's ast module for multi-language codebases. It supports the same unit-based chunking principle with broader language coverage.

A larger golden set would also strengthen the metrics analysis. Six queries is enough to illustrate the precision/recall tradeoff but not enough to draw statistical conclusions about relative strategy performance.

Conclusion

Chunking strategy and retrieval strategy are not independent decisions. Chunking runs first and sets the ceiling for what retrieval can possibly find. A retriever cannot return information that chunking has destroyed.

For code Q&A, AST-based chunking raises that ceiling dramatically. The retrieval method matters most in the middle - once chunking has preserved the relevant context, and once the query's vocabulary diverges from the code's.

The practical order of operations: get chunking right first. Then tune retrieval.

Full source: https://github.com/f2015537/agentic-rag-case-study

RAG (Retrieval-Augmented Generation) is one of those patterns that looks simple on a diagram and gets complicated fast in practice. The chunking strategy, embedding model choice, vector store selection, and the split between ingestion and retrieval all have real consequences for answer quality, cost, and maintainability.

This post walks through every design decision I made building a RAG chatbot that answers natural language questions about Washington state hiking trails. The domain is specific. The decisions generalise.

Architecture overview

The system has two distinct stages that run independently:

DATA PIPELINE (data_ingestion.ipynb)

Web Sources          LangChain              ChromaDB
───────────      ──────────────────      ──────────────
NPS.gov      →   WebBaseLoader       →   text-embedding
Wikipedia        BeautifulSoup           -3-small
Recreation.gov   RecursiveCharacter      ./chroma_db/
                 TextSplitter            washington_hikes

AGENT RUNTIME (hiking_agent/)

User Query
    │
    ▼
Google ADK Agent  →  retrieve_info()  →  Chroma similarity search (k=5)
(Gemini 2.5 Flash)   FunctionTool
    │
    ▼
Grounded Response

The ingestion pipeline runs once and produces a persisted Chroma collection. The agent is stateless and reconnects to that collection on every query.

Decision 1: RAG instead of fine-tuning

Trail conditions, permit availability, and access windows change seasonally. A fine-tuned model would need a full retraining cycle every time the data changes.

RAG sidesteps this entirely. When the data needs refreshing, re-run the ingestion notebook. The agent does not need to be redeployed. The model does not need to be retrained. The knowledge base updates independently of the running system.

This is the core argument for RAG over fine-tuning whenever the underlying data has a meaningful refresh rate: the separation between knowledge and inference is a feature, not a limitation.

Decision 2: Chunk size - 1000 chars / 150 char overlap

Chunk size is one of the most consequential RAG decisions and one of the least discussed.

Too small: individual chunks lose context. A chunk that says "permits are required" without the surrounding sentence explaining which trail and which season is useless at retrieval time.

Too large: chunks exceed the token budget for retrieval. With k=5 results passed to Gemini, the combined retrieved context needs to fit within the model's context window while leaving room for the system prompt and response.

1000 chars hits the right balance for NPS and Wikipedia content, which tends to be dense and information-rich. The 150-char overlap is not arbitrary either. It prevents information from being lost at chunk boundaries, which matters most when a key fact (a distance, an elevation, a permit requirement) happens to fall at the end of a chunk.

Decision 3: text-embedding-3-small over text-embedding-3-large

OpenAI's large embedding model improves retrieval recall on English tasks. It also costs approximately 5x more per token than the small model.

On a 640-chunk corpus, the recall improvement is marginal. The retrieval interface is identical between the two models, so swapping later requires changing a single constructor argument. Starting with the cheaper model and benchmarking before upgrading is the correct order of operations.

Decision 4: Chroma local persistent store over a hosted vector DB

Pinecone, Weaviate, and Qdrant are all excellent options for production RAG systems. For a single-developer showcase project, they introduce infrastructure cost and setup friction with no meaningful benefit.

Chroma's local persistent mode stores the vector collection on disk and reconnects to it on every session. The retrieval API (as_retriever()) is identical to hosted alternatives. Swapping to a hosted backend if this were to scale requires changing a single constructor call.

Decision 5: Decoupled ingestion and agent

This is the architectural decision with the most operational impact.

The ingestion pipeline is scraping-dependent. It hits 29 URLs, filters and cleans the content, chunks and embeds it, and writes to disk. It is a one-shot operation that can be triggered manually or scheduled via cron or a workflow orchestrator.

The agent has no scraping dependencies at all. It connects to the persisted Chroma store, runs similarity search, and passes the retrieved chunks to Gemini. It can be deployed to any environment that has access to the chroma_db directory without any of the ingestion tooling.

This separation means:

• Refresh the knowledge base without touching the agent

• Deploy the agent without any scraping setup

• Run the ingestion pipeline on a schedule without coordinating with the live agent

Data sources and the honest limitation

The ingestion pipeline scrapes 29 URLs across three authoritative sources: NPS.gov for park-level information and seasonal guidance, Wikipedia for rich articles on individual trails and wilderness areas, and Recreation.gov for permit and quota information.

The honest limitation: Washington Trails Association (WTA) is the most comprehensive per-trail database in the state, with difficulty ratings, distances, and elevation profiles for over 10,000 hikes. It is protected by Cloudflare's JS challenge and cannot be reliably scraped with a standard HTTP client.

This is the single highest-impact data improvement available: a Playwright-based scraper that executes the JS challenge and extracts WTA's per-trail data would transform the chatbot's ability to answer specific trail queries (distance, difficulty, elevation for individual hikes) rather than region-level queries.

I documented this limitation explicitly in the README rather than papering over it. A system that knows what it does not know is more useful than one that guesses.

What I would change at larger scale

Hybrid search. Dense vector search alone misses exact-match queries. Combining it with BM25 sparse retrieval improves recall on specific trail names like "Rattlesnake Ledge" or "The Enchantments" where keyword matching is more reliable than semantic similarity.

Metadata filtering. Storing region, difficulty, and distance as Chroma metadata fields would enable structured pre-filtering before semantic search. "Easy hikes in the North Cascades" becomes a metadata filter + semantic search rather than relying entirely on the embedding to do both.

Evaluation harness. A golden dataset of Q&A pairs with known correct answers would let me measure retrieval precision and answer quality across data refreshes. Without it, I am eyeballing the demo outputs.

Scheduled re-ingestion. Trail conditions and permit availability are seasonal. A scheduled pipeline that re-ingests the data sources on a weekly or monthly cadence would keep the knowledge base current without manual intervention.

Conclusion

The RAG pattern is simple. Getting it right requires deliberate decisions on chunking, embeddings, retrieval depth, and the boundary between ingestion and inference. Every one of those decisions has a tradeoff, and the right choice depends on the data characteristics, the query patterns, and the operational constraints of the system.

The honest limitation section in the README is not an apology. It is a specification of what would make the system better, written for the next person who works on it.

Full source: https://github.com/f2015537/RAG-Chatbot

Most AI image generation workflows are open-loop: you write a prompt, you get an image, you decide if it's good. If it's not, you tweak the prompt and try again. The feedback loop runs through the human.

This project closes that loop. It's a multi-agent pipeline built on Google's Agent Development Kit (ADK) that generates portrait images and self-critiques them — iterating autonomously until every visual attribute passes a strict per-attribute evaluation, or the iteration limit is reached.

The necklace failed all five times. That failure is the most interesting result.

Architecture overview

The pipeline has four components arranged as a SequentialAgent wrapping a LoopAgent:

User message
    │
    ▼
IntakeAgent (LlmAgent)
    │  Parses description → state["person_description"]
    ▼
PortraitRefinementLoop (LoopAgent, max 5 iterations)
    │
    ├── PortraitWriterAgent (LlmAgent)
    │       └── generate_image tool
    │             • gemini-3.1-flash-image-preview
    │             • saves PNG to disk + ADK artifact
    │
    └── PortraitCriticAgent (LlmAgent)
            └── critique_image tool
                  • gemini-3.5-flash (vision)
                  • per-attribute evaluation
            └── exit_loop tool
                  • sets actions.escalate = True

Each agent has a single, clearly scoped responsibility. The writer generates. The critic evaluates. The loop orchestrates. No agent does more than one thing.

Decision 1: Per-attribute evaluation instead of holistic scoring

The critic doesn't ask "does this image match the description?" It operates in three steps:

1. Parse every distinct visual attribute from the description into an explicit sub-checklist — hair colour, iris detail, clothing texture, lighting direction, necklace position, composition.

2. Verify each attribute individually. A vague resemblance does not pass. The match must be exact and unambiguous.

3. Apply general photographic standards regardless of description — face fully visible and sharply rendered, professional lighting, portrait-standard composition.

A single failing attribute triggers another iteration with feedback naming exactly what is wrong and why.

This is more expensive than a holistic score. It's also dramatically more precise. The example run demonstrates why.

The necklace problem

The test description included: "gold chain necklace visible above the collar."

The necklace failed all five iterations. Every generated version rendered it draped on the sweater body below the fold — not above the collar as specified.

This is a physically precise spatial constraint. The necklace is present in every image. It's gold. It's near the collar. A holistic "does this match the description?" check would likely pass it.

But above the collar is an exact positional requirement. The image model consistently defaulted to the more statistically common visual — necklace on chest — regardless of what the prompt said. The per-attribute critic caught this every time. A pass/fail prompt would not have.

Iteration 1 — Fail: Gold chain necklace positioned below the turtleneck collar rather than above it

Iteration 2 — Fail: Necklace still below collar; iris shows golden-brown inner ring instead of green

Iteration 3 — Fail: All other attributes pass; necklace remains draped onto the sweater body below the fold

Iteration 4 — Fail: Necklace below collar; shadow cast on wrong cheek; iris lacks defined green inner ring

Iteration 5 — Fail: Necklace below collar; eyes appear fully green rather than hazel with a green inner ring

The iris detail showed a similar pattern. The description specified "hazel eyes with a distinct green inner ring." The model repeatedly collapsed this to a uniform colour — fully green or fully brown — unable to render the two-tone structure at portrait scale.

These aren't prompt engineering failures. They're the edges of what current image generation models can reliably produce, surfaced by evaluation that's precise enough to find them.

Decision 2: Tools for image bytes, strings for inter-agent communication

ADK session state is JSON-serialisable. Image bytes are not. This constraint shapes the tool design.

The generate_image tool handles the google.genai API call, saves the PNG to disk, and saves an ADK artifact for inline UI rendering. It returns a file path string to the agent — not image bytes.

The critique_image tool reads the image from disk, sends it to Gemini's vision model alongside the structured evaluation prompt, and returns a text critique string.

All inter-agent communication flows as plain strings: a file path and a critique. The tools own the binary I/O; the agents never touch it. This keeps the session state clean and the agent logic simple.

Decision 3: Async tools

adk web runs on a FastAPI/uvicorn event loop. Image generation via the Gemini API can take several seconds per call.

Making generate_image and critique_image async — using client.aio.models.generate_content — prevents these calls from blocking the server. tool_context.save_artifact() is also an async ADK method, which requires the tool function itself to be async.

This is a small implementation detail that becomes a significant reliability issue at scale. A synchronous image tool in a web-served agent will stall the event loop under any real load.

Decision 4: IntakeAgent as a state bridge

When running via adk web, the user's input arrives as a chat message in the conversation history — not pre-loaded into session state. Downstream agents that expect state["person_description"] would fail to find it.

The IntakeAgent solves this by reading the conversation and writing a clean, normalised value into session state. Downstream agents consume it via {person_description} template substitution in their system prompts.

This is a pattern worth noting for any ADK pipeline built for adk web: the first agent in the sequence often needs to be a state initialisation agent, bridging between the chat interface and the structured state that the rest of the pipeline expects.

Decision 5: Early exit with escalate

The LoopAgent runs up to 5 iterations unconditionally unless told to stop. The critic calls exit_loop, which sets tool_context.actions.escalate = True. ADK treats this as a termination signal and breaks the loop.

The practical effect: if the image passes on iteration 2, you pay for 2 Gemini API calls, not 5. Cost and latency scale with the quality of the generation, not the iteration ceiling. For a pipeline that makes multiple API calls per iteration, this matters.

What I'd change at larger scale

Parallelise critique and generation where possible. The current loop is strictly sequential. On a pipeline with more agents, some evaluation steps could run in parallel to reduce wall-clock time per iteration.

Add a confidence threshold to the critic. Currently it's binary — pass or fail. A confidence score per attribute would allow the writer to focus on low-confidence attributes rather than regenerating from scratch on each iteration.

Persist iteration history for analysis. The current run saves each portrait to disk but doesn't log the full critique history in a queryable format. A structured log of attribute-level pass/fail per iteration would let you analyse which attributes current models struggle with most — useful data for prompt engineering research.

HTTP SSE transport for remote deployment. The current setup runs locally. Exposing the pipeline as a network-accessible service requires switching the ADK transport and adding authentication.

Conclusion

Closing the generation-evaluation loop with a per-attribute critic surfaces failure modes that holistic evaluation misses. The necklace-above-the-collar failure isn't a prompt engineering problem — it's the image model's spatial reasoning hitting its limit, made visible by an evaluator precise enough to find it.

The architecture pattern — sequential agents, loop with early exit, tools owning binary I/O — is reusable for any iterative generation task where quality criteria can be made explicit.

Full source + example run images: https://github.com/f2015537/iterative-portrait-agent

The AI agent ecosystem is fragmented. Every framework — LangChain, Google ADK, CrewAI, AutoGen — has its own tool format, its own abstractions, and its own way of connecting agents to capabilities. If you build tools for one framework, you typically can't use them in another without rewriting them.

The Model Context Protocol (MCP) exists to fix this. MCP is an open standard that defines how agents and tools communicate — independent of the framework on either side. This post walks through a project that demonstrates exactly that: a custom MCP server that wraps Google ADK tools and exposes them to any MCP-compatible client.

The problem with tightly coupled tools

When you define a tool directly inside an ADK agent, it's coupled to that agent. If you want to use the same tool in a different agent, a different framework, or a different process, you have to rewrite it.

This is the same problem that HTTP solved for web services — before a common protocol, every client and server spoke their own language. MCP is that common protocol for AI tools.

Architecture overview

The project implements a full ADK ↔ MCP integration loop:

User
 └─► ADK LlmAgent (Claude Sonnet 4.6 via LiteLLM)
      └─► McpToolset (MCP client)
           └─► my_adk_mcp_server.py (custom MCP server, subprocess)
                ├─► ADK FunctionTool (create_file)
                └─► ADK FunctionTool (create_file_with_content)

Every layer in this chain communicates over a defined protocol. The agent doesn't know what the tools do — it knows what tools are available via MCP discovery. The MCP server doesn't know what model is driving the agent — it just handles tool calls.

Decision 1: Wrapping ADK FunctionTools in an MCP server

The MCP server (my_adk_mcp_server.py) is the architectural centrepiece. It takes ADK FunctionTool instances and exposes them as MCP tools — making them available to any MCP-compatible client, not just ADK agents.

This is the key pattern: write tools once in ADK, expose them everywhere via MCP.

The two tools exposed in this project are intentionally simple — create_file and create_file_with_content. The simplicity is deliberate: the point is the pattern, not the tools. Swap in any ADK FunctionTool and the architecture is identical.

Decision 2: stdio transport

MCP supports multiple transport mechanisms. This project uses stdio — the agent spawns the MCP server as a subprocess and communicates over stdin/stdout.

stdio is the right choice for local development and single-machine deployments:

• No network ports to configure

• No service discovery

• No authentication setup

• The server process lifecycle is managed by the agent

The tradeoff is that stdio doesn't work for distributed deployments where the agent and server run on different machines. For production systems with remote tool servers, HTTP SSE or WebSocket transport would be appropriate. For this use case, stdio is the simplest correct option.

Decision 3: LiteLLM for model flexibility

Google ADK natively targets Gemini models. This project routes through LiteLLM, which means the agent can be pointed at any LLM — Claude Sonnet 4.6, GPT-4o, Mistral, or any other LiteLLM-supported model — without changing the agent logic.

The model becomes a configuration detail:

# In agent.py
model="claude-sonnet-4-6"  # swap this for any LiteLLM-supported model

This is a small change with significant architectural implications. You can now benchmark different models against the same agent and tool setup, or switch models based on cost and latency requirements, without touching the agent or tool code.

Decision 4: McpToolset as the MCP client

Inside the ADK agent, McpToolset acts as the MCP client. It handles the connection to the MCP server, discovers available tools, and translates MCP tool calls into ADK-compatible invocations.

This means the agent definition is clean:

# agent.py — simplified
agent = LlmAgent(
    model="claude-sonnet-4-6",
    tools=[McpToolset(server_params=StdioServerParameters(
        command="python",
        args=["my_adk_mcp_server.py"]
    ))]
)

The agent doesn't enumerate its tools directly — it discovers them at runtime via MCP. Add a new tool to the MCP server and the agent picks it up automatically.

Running the agent

# CLI
adk run pilot_agent

# Web UI
adk web
# Open http://localhost:8000, select pilot_agent

Example interactions:

> Create a file called notes.txt
> Create a file called report.md with content "Hello, World!"

The agent resolves the intent, selects the appropriate MCP tool, and the MCP server executes it.

What I'd extend at larger scale

Add more ADK FunctionTools to the MCP server. The current tools are deliberately minimal. The pattern scales linearly — each new ADK tool added to the server becomes available to every MCP client automatically.

Switch to HTTP SSE transport for distributed deployments. stdio works for local development but isn't suitable for scenarios where the agent and tool server run in different environments. HTTP SSE transport makes the MCP server network-accessible.

Add tool authentication. The current setup assumes a trusted local environment. Production MCP servers exposed over a network need authentication — OAuth or API key validation at the MCP layer.

Multi-agent coordination. The MCP server can serve multiple agents simultaneously. This opens up multi-agent architectures where specialised agents share a common tool layer without duplicating tool implementations.

Conclusion

ADK and MCP are complementary, not competing. Wrapping ADK tools in an MCP server gives you the best of both: ADK's agent framework and tool primitives, with MCP's protocol-level interoperability.

The pattern is simple, the separation is clean, and the architecture scales naturally as you add more tools and more agents.

Full source: https://github.com/f2015537/building-agents-with-google-adk

Infrastructure-as-code has a reputation for complexity. But when it's done well — with a typed language, a composable abstraction layer, and clear stack boundaries — it becomes the most reliable way to build and reproduce cloud infrastructure at any scale.

This post walks through the architecture of Space Finder: a full-stack AWS application for listing and discovering venues. Every piece of infrastructure is defined in TypeScript using the AWS CDK, deployed across six stacks. I'll cover the key design decisions, the tradeoffs I made, and what I'd change at larger scale.

Architecture overview

The application has two main surfaces: a React 19 frontend served via CloudFront, and a serverless API backed by API Gateway, Lambda, and DynamoDB.

Browser
  │
  ├── CloudFront → S3 (React SPA)
  │
  ├── API Gateway /spaceFinder ←── Cognito JWT authorizer
  │     └── Lambda (Node 20)
  │           └── DynamoDB
  │
  ├── S3 (photos) ←── direct upload via Cognito Identity Pool credentials
  │
  └── Cognito User Pool ←── SES for verification emails

Six CDK stacks each own a single concern: auth, database, API, storage, frontend hosting, and monitoring. This makes individual stacks deployable and testable in isolation.

Decision 1: Direct browser-to-S3 uploads

This is the decision that shaped the most of the architecture.

Lambda has a 6 MB payload limit on API Gateway-proxied requests. Routing photo uploads through the backend would mean either hitting that limit on moderately-sized images, chunking uploads in the client, or running a separate upload service. None of these are good options for a lean stack.

The alternative: after a user signs in, Amplify's fetchAuthSession() returns temporary AWS credentials from the Cognito Identity Pool. These credentials carry an IAM role that grants scoped S3 write access. The browser uploads directly to S3 using the AWS SDK v3 — the backend is never in the data path.

This eliminates unnecessary data transfer costs, removes Lambda from the failure surface for uploads, and sidesteps the payload limit entirely. The tradeoff is that the frontend carries more responsibility: it must handle S3 upload state, retries, and progress tracking.

Decision 2: Per-user data isolation at the DynamoDB layer

A common pattern for multi-tenant data isolation is to enforce access control in application logic — check the requesting user's ID against the resource owner before returning data. This works, but it's fragile. Logic drifts. Edge cases accumulate.

In Space Finder, isolation is enforced at the query layer. API Gateway validates the Cognito JWT before the request reaches Lambda. The Lambda then reads the sub claim from the authorizer context — a stable, unique identifier per user — and stamps it onto every DynamoDB write. Reads filter by it.

There's no application logic that can accidentally return another user's spaces, because the query itself is scoped. This is a simpler invariant to reason about and audit.

Decision 3: Single Lambda for all CRUD

The Lambda handler routes on httpMethod internally — GET, POST, PUT, and DELETE are all handled in one function.

The conventional microservices pattern would give each operation its own Lambda. That's the right call at scale: independent deployment, independent scaling, independent cold start profiles. But for a solo project with modest traffic, one handler means one deployment, one log stream to tail, and one cold start to reason about.

This is a deliberate tradeoff, not an oversight. The codebase is structured to make the migration straightforward: each route's logic lives in its own module, so splitting into separate Lambdas later is a refactor, not a rewrite.

Decision 4: Monitoring wired into CDK

Observability is not an afterthought here. A CloudWatch alarm monitors the API error rate. When it fires, it publishes to an SNS topic, which triggers a dedicated Lambda that POSTs to a Slack webhook.

The entire chain is defined in CDK. There are no manual console configurations to drift from the code. A new deployment recreates the monitoring stack from scratch if needed.

SES is used for transactional email during signup. By default, SES runs in sandbox mode — it only sends to verified addresses. Requesting production access via the AWS console lifts this restriction, but the sandbox is sufficient for development and demo purposes.

Deployment

The full infrastructure deploys with three commands:

npm install
cdk bootstrap   # first time only
cdk deploy --all

CDK outputs the API URL, Cognito pool IDs, and CloudFront domain after each deployment. These populate the .env files for both the backend test scripts and the Vite frontend build.

The UIDeploymentStack handles the frontend: it builds the React app, uploads the assets to S3, and automatically invalidates the CloudFront cache. No manual cache invalidation step.

What I'd change at larger scale

Separate Lambdas per route. The single-Lambda approach is pragmatic but doesn't scale operationally. As traffic grows, independent scaling and deployment become important.

DynamoDB single-table design. The current schema partitions spaces per user with a straightforward key structure. A single-table design with composite keys and GSIs would support more access patterns without additional tables.

SES production access from day one. The sandbox restriction catches developers off-guard. Requesting production access early avoids surprises when real users try to sign up.

End-to-end tests against the deployed API. The current test directory contains manual HTTP scripts. Automated integration tests running against the live stack would close the confidence gap between CDK deploy and production readiness.

Conclusion

AWS CDK with TypeScript makes cloud infrastructure feel like application code — typed, composable, and version-controlled. The six-stack structure keeps concerns separate without adding operational overhead. The direct S3 upload pattern and DynamoDB-layer isolation are patterns worth carrying into any multi-tenant AWS application.

The full source is on GitHub: github.com/f2015537/space-finder

Live demo: https://d24kqex7dx8ru7.cloudfront.net