ClearCode Part 5: Semantic Caching, Incremental Indexing, and the Hardest Part of Caching

Where Part 4 left us

Part 4 built the tasks layer: structured plan generation with Pydantic, human-in-the-loop approval before anything runs, serial execution with dependency resolution, an LLM-as-judge verifying each task's output, and crash recovery that resets incomplete tasks on restart. ClearCode could take a goal, break it into steps, and implement it autonomously.

Part 5 is about making it fast. And about the harder problem that fast creates.

The problem with asking the same question twice

Every /ask query goes through the same pipeline: embed the question, retrieve relevant chunks, invoke the agent with that context, stream tool calls until an answer emerges, return the result. For a complex question, that might be four or five tool calls and two LLM completions. For a simple one, it is still at least one embedding call and one completion.

If you ask the same question twice — or a slightly different version of the same question — you pay the full price both times.

This compounds quickly in practice. You ask the same types of questions about a codebase repeatedly. "Where is X defined?", "What does Y do?", "How is Z called?" drift in phrasing but converge on the same answer. The agent does the same work, produces the same output, and charges you again.

The fix is a cache. But an exact-match cache fails the moment the phrasing changes. "What does index_codebase do?" and "Can you explain what index_codebase does?" should hit the same entry. They will not under an exact-match scheme.

The right cache for a language model application is a semantic cache: one that measures similarity between queries in embedding space and returns the stored answer if the new question is close enough to one already answered.

1. Semantic cache: why cosine similarity, not exact match

The cache stores (question_embedding, answer) pairs. On a new query, it embeds the question, finds the nearest stored embedding by cosine similarity, and returns the associated answer if the similarity exceeds a configurable threshold. Below the threshold, it falls through to the agent.

async def get(self, query: str, domain: str, model: str) -> str | None:
    hit = await self._top_match(query, domain=domain, model=model)
    if hit is None:
        return None
    similarity = 1 - float(hit["vector_distance"])
    if similarity >= self.threshold:
        return hit["response"]
    return None

The threshold is the key tuning parameter. Too high and you only hit on near-exact phrasing — the cache becomes useless. Too low and you start returning cached answers for questions that are only superficially similar, which is worse than no cache at all. 0.85 cosine similarity is the default, chosen empirically. It catches rephrasings and synonym variations while rejecting questions that are genuinely different.

The embeddings come from the same model used for retrieval — text-embedding-3-small by default. This is intentional. The cache and the retrieval layer operate in the same vector space. Switching embedding models requires rebuilding both, but they are always consistent with each other.

2. Redis + RedisVL

The backing store is Redis with a RedisVL HNSW vector index. Redis was the natural choice: fast, widely deployed, and RedisVL gives it native approximate nearest-neighbour search without requiring a separate vector database.

The index schema is defined once at startup:

def _build_index_schema(dims: int) -> dict:
    return {
        "index": {"name": "semantic_cache", "prefix": "cache:"},
        "fields": [
            {
                "name": "query_vector",
                "type": "vector",
                "attrs": {
                    "dims": dims,
                    "algorithm": "HNSW",
                    "distance_metric": "cosine",
                    "datatype": "float32",
                },
            },
            {"name": "response",   "type": "text"},
            {"name": "query_text", "type": "text"},
            {"name": "domain",     "type": "tag"},
            {"name": "model",      "type": "tag"},
            {"name": "created_at", "type": "numeric"},
        ],
    }

overwrite=False on index.create() means the index is reused across restarts rather than wiped. Cache entries survive a REPL restart — you do not pay embedding costs again on the first query of a new session if you already asked the same question before.

One correctness detail worth naming: embed_query on LangChain's embedder is a synchronous function. Calling it directly from an async method blocks the event loop. The fix is asyncio.to_thread:

async def _embed(self, text: str) -> list[float]:
    return await asyncio.to_thread(self.embedder.embed_query, text)

This runs the sync call in a threadpool executor and awaits the result without blocking. Missing this produces a TypeError and a confusing error message that does not point at the actual cause.

3. Domain and model isolation

A single Redis instance can serve multiple repos. Without isolation, cached answers from one codebase would contaminate queries about another — both happening to use the same question phrasing.

Each repo gets a domain: the first 16 characters of the SHA-256 hash of its absolute path.

def get_repo_domain(repo_path: str) -> str:
    return hashlib.sha256(repo_path.encode()).hexdigest()[:16]

Every cache entry is tagged with both domain and model. The VectorQuery filters on both before scoring similarity:

q.set_filter((Tag("domain") == domain) & (Tag("model") == model))

Two repos on the same Redis instance never share entries. And switching from gpt-4o to claude-opus-4 does not cause the cache to serve answers generated by a different model. Different models have different capabilities and different failure modes. An answer correct for one may not be for another.

4. The watcher → cache bridge

A cached answer that is no longer correct is worse than no cache at all.

Ask "What does process_payment do?", get an accurate summary, cache it. Then a /plan task rewrites process_payment completely. The next time you ask, the cache returns the old answer without checking the codebase. You are now confidently wrong.

The solution is to invalidate the cache whenever the codebase changes. ClearCode already has a filesystem watcher that fires on create, modify, rename, or delete. The cache invalidation just needs to be wired to that callback.

The wiring has one complication: the watchdog observer runs in a background daemon thread. The asyncio event loop — which owns the Redis client — runs in the main thread. You cannot call an async function from a background thread directly.

The bridge is asyncio.run_coroutine_threadsafe:

loop = asyncio.get_running_loop()

def invalidate_cache_on_change():
    if semantic_cache is not None:
        asyncio.run_coroutine_threadsafe(
            semantic_cache.invalidate_domain(cache_domain), loop
        )

This submits the coroutine to the event loop from the watcher thread and returns a Future. The event loop picks it up when it is next free. The callback takes no arguments — it only needs to signal that something changed, not what.

From this point forward, any file change in the project directory triggers cache invalidation for that repo's domain automatically. A /reindex command does the same explicitly.

5. The bug: invalidate_domain was silently doing nothing

This is the one worth naming carefully, because it is exactly the kind of bug that passes every integration test you write — right up until you write a test that checks the right thing.

invalidate_domain is supposed to find all cache entries for a given domain and delete them. The implementation used RedisVL's FilterQuery to locate entries by their domain tag, then called client.delete() on the resulting keys:

results = await self.index.query(fq)
if not results:
    return
keys = [self.index.key(r["id"]) for r in results]  # ← the bug
await self.client.delete(*keys)

The bug is in that one line. r["id"] in a RedisVL query result is already the full Redis key — prefix included. For this index, a stored key looks like cache:abc123def456. So r["id"] is "cache:abc123def456".

self.index.key(r["id"]) prepends the index prefix to whatever you pass it. That produces "cache:cache:abc123def456".

client.delete("cache:cache:abc123def456") deletes zero keys. Redis returns success — you are allowed to delete a key that does not exist. The function logs "Invalidated N cache entries" and exits cleanly. Nothing is actually deleted.

The fix is one word:

keys = [r["id"] for r in results]  # r["id"] is already the full key

What made this dangerous: the logging told you invalidation succeeded. The code path had no exception to catch. Entries survived silently, and subsequent queries returned stale cached answers with no indication that something was wrong.

The tests caught it. After calling invalidate_domain, the next get() on the same query should return None. It did not. That assertion failure is what led to the fix.

6. Incremental indexing and watcher improvements

Two related improvements went in alongside the cache.

Incremental indexing in ChromaDB

The original implementation re-embedded every source file on every startup. 300 files, restart the REPL, pay 300 embedding API calls — even if nothing had changed since yesterday.

The fix is mtime-based tracking. Each indexed chunk stores the mtime of the file it came from. On startup, the indexer compares current file mtimes against stored ones:

• File is new → embed and index.

• File has the same mtime → skip entirely. No API call.

• File has a newer mtime → delete old chunks for that file, re-embed, re-index.

• File no longer exists → prune its chunks from the collection.

A repo with 300 unchanged files now starts in the time it takes to check 300 mtimes. The first run still pays the full embedding cost. Every run after costs nothing for files that have not changed.

Watcher backend dispatch

The original watcher hardcoded ChromaDB. If you switched to Qdrant in config, file changes updated the ChromaDB index nobody was querying — not the Qdrant index that was.

The fix routes through the factory at call time:

def _run(self, action: str, filepath: str) -> None:
    from clearcode.context.indexers.factory import (
        get_single_file_indexer, get_file_remover
    )
    if action == "delete":
        get_file_remover()(filepath)
    else:
        get_single_file_indexer()(filepath)
    if self.on_change is not None:
        self.on_change()

Switching backends in config is now truly a one-line change — the watcher follows automatically, and the cache invalidation callback fires regardless of which backend is active.

7. handle_query returns a tuple

handle_query used to return a str. It now returns (str, bool) — the answer and whether it came from cache.

async def handle_query(...) -> tuple[str, bool]:
    ...
    if cached_response is not None:
        return cached_response, True
    ...
    return answer, False

This lets main.py print a visual indicator on cache hits without embedding cache-awareness into the orchestrator beyond what it needs to do its job:

> /ask Can you explain what index_codebase does?

Searching for: Can you explain what index_codebase does?...
⚡ cache hit

index_codebase scans all source files in the repo, parses them with
tree-sitter for AST-aware chunking, and upserts the resulting chunks
into the configured vector store backend. On startup and /reindex it
skips files whose mtime hasn't changed, so only new or modified files
are re-embedded.

The ⚡ cache hit line tells you the answer came from Redis. No LLM was invoked. No tools were called.

8. The test suite

The changes in Part 5 are tested end-to-end against a live Redis instance. Nineteen assertions cover the full cache workflow:

1. build_semantic_cache() returns a SemanticCache

2. Threshold loaded from config correctly

3. get() returns None on a cold cache

4. Exact query returns cached answer after put()

5. Semantically similar query hits cache above 0.85 threshold

6. Similar query returns the same stored answer

7. Unrelated query misses (pasta carbonara)

8. Different domain misses on identical query

9. Different model tag misses on identical query

10. invalidate_domain() deletes all domain entries

11. After invalidation, exact query returns None

12. After invalidation, similar query returns None

13. handle_query calls the agent exactly once on miss

14. handle_query returns the agent's answer on miss

15. handle_query reports from_cache=False on miss

16. handle_query returns the cached answer on hit

17. Agent NOT called again on cache hit

18. handle_query reports from_cache=True on hit

19. Watcher callback correctly invalidates via run_coroutine_threadsafe

Tests 10–12 failed on the first run. That is how the double-prefix bug was found.

Known flaws

Cache not invalidated at /plan start. The watcher handles file changes made by the executor's write_file calls. But the cache is not cleared at the start of a /plan run. If a task rewrites a file whose answer is already cached, the stale answer survives until the watcher fires — a 1.5-second window after each file write.

Redis required, no in-process fallback. build_semantic_cache() degrades gracefully when Redis is unreachable — it catches the failure and returns None, falling back to uncached agent calls. But there is no in-process fallback. If you want caching without Redis, there is currently no option.

Cache size is unbounded. TTL controls individual entry expiry, but there is no maximum entry count or LRU eviction. For long-running sessions against a large codebase, Redis memory usage grows without bound until invalidation clears the domain.

Embedding cost on every miss. A cache miss still embeds the query for the similarity search before falling through to the agent. A cold cache costs one extra embedding call per query. At text-embedding-3-small rates this is negligible, but it is nonzero.

invalidate_domain truncates at 10,000 entries. FilterQuery is called with num_results=10_000. For a domain with more than 10,000 cached entries, invalidation silently leaves some stale entries behind.

Where Part 5 leaves things

ClearCode now has a semantic cache that returns answers in milliseconds when you ask a question it has already answered, invalidates automatically when any source file changes, and isolates entries per repo and per model so nothing cross-contaminates.

The incremental indexer means repeated startups cost nothing for files that have not changed. The watcher dispatches correctly to whichever backend is configured.

One real bug was found in testing and fixed before this post was written. Five known flaws are documented above rather than hidden.

The next layer is safety and observability: guardrails on what the agent is allowed to do, structured logging that makes the tool call chain inspectable, and the beginning of the eval harness that will make improvements measurable rather than assumed.

Full source: https://github.com/f2015537/clearcode

Part 1 — Architecture before code: https://blog.divyampatro.dev/clearcode-part-1-reverse-engineering-a-coding-agent-before-writing-a-single-line-of-code Part 2 — Context layer: https://blog.divyampatro.dev/clearcode-part-2-ast-aware-indexing-vector-stores-and-hybrid-retrieval Part 3 — Memory, agent reasoning, skills, and MCP: https://blog.divyampatro.dev/clearcode-part-3-memory-agent-reasoning-skills-and-mcp Part 4 — Tasks layer: https://blog.divyampatro.dev/clearcode-part-4-autonomous-plan-execution-llm-as-judge-and-human-in-the-loop-approval