# ClearCode Part 5: Semantic Caching, Incremental Indexing, and the Hardest Part of Caching ## Where Part 4 left us Part 4 built the tasks layer: structured plan generation with Pydantic, human-in-the-loop approval before anything runs, serial execution with dependency resolution, an LLM-as-judge verifying each task's output, and crash recovery that resets incomplete tasks on restart. ClearCode could take a goal, break it into steps, and implement it autonomously. Part 5 is about making it fast. And about the harder problem that fast creates. * * * ## The problem with asking the same question twice Every `/ask` query goes through the same pipeline: embed the question, retrieve relevant chunks, invoke the agent with that context, stream tool calls until an answer emerges, return the result. For a complex question, that might be four or five tool calls and two LLM completions. For a simple one, it is still at least one embedding call and one completion. If you ask the same question twice — or a slightly different version of the same question — you pay the full price both times. This compounds quickly in practice. You ask the same types of questions about a codebase repeatedly. "Where is X defined?", "What does Y do?", "How is Z called?" drift in phrasing but converge on the same answer. The agent does the same work, produces the same output, and charges you again. The fix is a cache. But an exact-match cache fails the moment the phrasing changes. "What does `index_codebase` do?" and "Can you explain what `index_codebase` does?" should hit the same entry. They will not under an exact-match scheme. The right cache for a language model application is a **semantic cache**: one that measures similarity between queries in embedding space and returns the stored answer if the new question is close enough to one already answered. * * * ## 1\. Semantic cache: why cosine similarity, not exact match The cache stores `(question_embedding, answer)` pairs. On a new query, it embeds the question, finds the nearest stored embedding by cosine similarity, and returns the associated answer if the similarity exceeds a configurable threshold. Below the threshold, it falls through to the agent. ```python async def get(self, query: str, domain: str, model: str) -> str | None: hit = await self._top_match(query, domain=domain, model=model) if hit is None: return None similarity = 1 - float(hit["vector_distance"]) if similarity >= self.threshold: return hit["response"] return None ``` The threshold is the key tuning parameter. Too high and you only hit on near-exact phrasing — the cache becomes useless. Too low and you start returning cached answers for questions that are only superficially similar, which is worse than no cache at all. **0.85 cosine similarity** is the default, chosen empirically. It catches rephrasings and synonym variations while rejecting questions that are genuinely different. The embeddings come from the same model used for retrieval — `text-embedding-3-small` by default. This is intentional. The cache and the retrieval layer operate in the same vector space. Switching embedding models requires rebuilding both, but they are always consistent with each other. * * * ## 2\. Redis + RedisVL The backing store is Redis with a RedisVL HNSW vector index. Redis was the natural choice: fast, widely deployed, and RedisVL gives it native approximate nearest-neighbour search without requiring a separate vector database. The index schema is defined once at startup: ```python def _build_index_schema(dims: int) -> dict: return { "index": {"name": "semantic_cache", "prefix": "cache:"}, "fields": [ { "name": "query_vector", "type": "vector", "attrs": { "dims": dims, "algorithm": "HNSW", "distance_metric": "cosine", "datatype": "float32", }, }, {"name": "response", "type": "text"}, {"name": "query_text", "type": "text"}, {"name": "domain", "type": "tag"}, {"name": "model", "type": "tag"}, {"name": "created_at", "type": "numeric"}, ], } ``` `overwrite=False` on `index.create()` means the index is reused across restarts rather than wiped. Cache entries survive a REPL restart — you do not pay embedding costs again on the first query of a new session if you already asked the same question before. One correctness detail worth naming: `embed_query` on LangChain's embedder is a synchronous function. Calling it directly from an `async` method blocks the event loop. The fix is `asyncio.to_thread`: ```python async def _embed(self, text: str) -> list[float]: return await asyncio.to_thread(self.embedder.embed_query, text) ``` This runs the sync call in a threadpool executor and awaits the result without blocking. Missing this produces a `TypeError` and a confusing error message that does not point at the actual cause. * * * ## 3\. Domain and model isolation A single Redis instance can serve multiple repos. Without isolation, cached answers from one codebase would contaminate queries about another — both happening to use the same question phrasing. Each repo gets a **domain**: the first 16 characters of the SHA-256 hash of its absolute path. ```python def get_repo_domain(repo_path: str) -> str: return hashlib.sha256(repo_path.encode()).hexdigest()[:16] ``` Every cache entry is tagged with both `domain` and `model`. The `VectorQuery` filters on both before scoring similarity: ```python q.set_filter((Tag("domain") == domain) & (Tag("model") == model)) ``` Two repos on the same Redis instance never share entries. And switching from `gpt-4o` to `claude-opus-4` does not cause the cache to serve answers generated by a different model. Different models have different capabilities and different failure modes. An answer correct for one may not be for another. * * * ## 4\. The watcher → cache bridge A cached answer that is no longer correct is worse than no cache at all. Ask "What does `process_payment` do?", get an accurate summary, cache it. Then a `/plan` task rewrites `process_payment` completely. The next time you ask, the cache returns the old answer without checking the codebase. You are now confidently wrong. The solution is to invalidate the cache whenever the codebase changes. ClearCode already has a filesystem watcher that fires on create, modify, rename, or delete. The cache invalidation just needs to be wired to that callback. The wiring has one complication: the watchdog observer runs in a **background daemon thread**. The asyncio event loop — which owns the Redis client — runs in the **main thread**. You cannot call an `async` function from a background thread directly. The bridge is `asyncio.run_coroutine_threadsafe`: ```python loop = asyncio.get_running_loop() def invalidate_cache_on_change(): if semantic_cache is not None: asyncio.run_coroutine_threadsafe( semantic_cache.invalidate_domain(cache_domain), loop ) ``` This submits the coroutine to the event loop from the watcher thread and returns a `Future`. The event loop picks it up when it is next free. The callback takes no arguments — it only needs to signal that *something* changed, not what. From this point forward, any file change in the project directory triggers cache invalidation for that repo's domain automatically. A `/reindex` command does the same explicitly. * * * ## 5\. The bug: `invalidate_domain` was silently doing nothing This is the one worth naming carefully, because it is exactly the kind of bug that passes every integration test you write — right up until you write a test that checks the right thing. `invalidate_domain` is supposed to find all cache entries for a given domain and delete them. The implementation used RedisVL's `FilterQuery` to locate entries by their `domain` tag, then called `client.delete()` on the resulting keys: ```python results = await self.index.query(fq) if not results: return keys = [self.index.key(r["id"]) for r in results] # ← the bug await self.client.delete(*keys) ``` The bug is in that one line. `r["id"]` in a RedisVL query result is already the full Redis key — prefix included. For this index, a stored key looks like `cache:abc123def456`. So `r["id"]` is `"cache:abc123def456"`. `self.index.key(r["id"])` prepends the index prefix to whatever you pass it. That produces `"cache:cache:abc123def456"`. `client.delete("cache:cache:abc123def456")` deletes zero keys. Redis returns success — you are allowed to delete a key that does not exist. The function logs "Invalidated N cache entries" and exits cleanly. Nothing is actually deleted. The fix is one word: ```python keys = [r["id"] for r in results] # r["id"] is already the full key ``` What made this dangerous: the logging told you invalidation succeeded. The code path had no exception to catch. Entries survived silently, and subsequent queries returned stale cached answers with no indication that something was wrong. The tests caught it. After calling `invalidate_domain`, the next `get()` on the same query should return `None`. It did not. That assertion failure is what led to the fix. * * * ## 6\. Incremental indexing and watcher improvements Two related improvements went in alongside the cache. ### Incremental indexing in ChromaDB The original implementation re-embedded every source file on every startup. 300 files, restart the REPL, pay 300 embedding API calls — even if nothing had changed since yesterday. The fix is mtime-based tracking. Each indexed chunk stores the `mtime` of the file it came from. On startup, the indexer compares current file mtimes against stored ones: * File is new → embed and index. * File has the same mtime → skip entirely. No API call. * File has a newer mtime → delete old chunks for that file, re-embed, re-index. * File no longer exists → prune its chunks from the collection. A repo with 300 unchanged files now starts in the time it takes to check 300 mtimes. The first run still pays the full embedding cost. Every run after costs nothing for files that have not changed. ### Watcher backend dispatch The original watcher hardcoded ChromaDB. If you switched to Qdrant in config, file changes updated the ChromaDB index nobody was querying — not the Qdrant index that was. The fix routes through the factory at call time: ```python def _run(self, action: str, filepath: str) -> None: from clearcode.context.indexers.factory import ( get_single_file_indexer, get_file_remover ) if action == "delete": get_file_remover()(filepath) else: get_single_file_indexer()(filepath) if self.on_change is not None: self.on_change() ``` Switching backends in config is now truly a one-line change — the watcher follows automatically, and the cache invalidation callback fires regardless of which backend is active. * * * ## 7\. `handle_query` returns a tuple `handle_query` used to return a `str`. It now returns `(str, bool)` — the answer and whether it came from cache. ```python async def handle_query(...) -> tuple[str, bool]: ... if cached_response is not None: return cached_response, True ... return answer, False ``` This lets `main.py` print a visual indicator on cache hits without embedding cache-awareness into the orchestrator beyond what it needs to do its job: ```plaintext > /ask Can you explain what index_codebase does? Searching for: Can you explain what index_codebase does?... ⚡ cache hit index_codebase scans all source files in the repo, parses them with tree-sitter for AST-aware chunking, and upserts the resulting chunks into the configured vector store backend. On startup and /reindex it skips files whose mtime hasn't changed, so only new or modified files are re-embedded. ``` The `⚡ cache hit` line tells you the answer came from Redis. No LLM was invoked. No tools were called. * * * ## 8\. The test suite The changes in Part 5 are tested end-to-end against a live Redis instance. Nineteen assertions cover the full cache workflow: 1. `build_semantic_cache()` returns a `SemanticCache` 2. Threshold loaded from config correctly 3. `get()` returns `None` on a cold cache 4. Exact query returns cached answer after `put()` 5. Semantically similar query hits cache above 0.85 threshold 6. Similar query returns the same stored answer 7. Unrelated query misses (pasta carbonara) 8. Different `domain` misses on identical query 9. Different `model` tag misses on identical query 10. `invalidate_domain()` deletes all domain entries 11. After invalidation, exact query returns `None` 12. After invalidation, similar query returns `None` 13. `handle_query` calls the agent exactly once on miss 14. `handle_query` returns the agent's answer on miss 15. `handle_query` reports `from_cache=False` on miss 16. `handle_query` returns the cached answer on hit 17. Agent NOT called again on cache hit 18. `handle_query` reports `from_cache=True` on hit 19. Watcher callback correctly invalidates via `run_coroutine_threadsafe` Tests 10–12 failed on the first run. That is how the double-prefix bug was found. * * * ## Known flaws **Cache not invalidated at** `/plan` **start.** The watcher handles file changes made by the executor's `write_file` calls. But the cache is not cleared at the start of a `/plan` run. If a task rewrites a file whose answer is already cached, the stale answer survives until the watcher fires — a 1.5-second window after each file write. **Redis required, no in-process fallback.** `build_semantic_cache()` degrades gracefully when Redis is unreachable — it catches the failure and returns `None`, falling back to uncached agent calls. But there is no in-process fallback. If you want caching without Redis, there is currently no option. **Cache size is unbounded.** TTL controls individual entry expiry, but there is no maximum entry count or LRU eviction. For long-running sessions against a large codebase, Redis memory usage grows without bound until invalidation clears the domain. **Embedding cost on every miss.** A cache miss still embeds the query for the similarity search before falling through to the agent. A cold cache costs one extra embedding call per query. At `text-embedding-3-small` rates this is negligible, but it is nonzero. `invalidate_domain` **truncates at 10,000 entries.** `FilterQuery` is called with `num_results=10_000`. For a domain with more than 10,000 cached entries, invalidation silently leaves some stale entries behind. * * * ## Where Part 5 leaves things ClearCode now has a semantic cache that returns answers in milliseconds when you ask a question it has already answered, invalidates automatically when any source file changes, and isolates entries per repo and per model so nothing cross-contaminates. The incremental indexer means repeated startups cost nothing for files that have not changed. The watcher dispatches correctly to whichever backend is configured. One real bug was found in testing and fixed before this post was written. Five known flaws are documented above rather than hidden. The next layer is safety and observability: guardrails on what the agent is allowed to do, structured logging that makes the tool call chain inspectable, and the beginning of the eval harness that will make improvements measurable rather than assumed. Full source: https://github.com/f2015537/clearcode Part 1 — Architecture before code: https://blog.divyampatro.dev/clearcode-part-1-reverse-engineering-a-coding-agent-before-writing-a-single-line-of-code Part 2 — Context layer: https://blog.divyampatro.dev/clearcode-part-2-ast-aware-indexing-vector-stores-and-hybrid-retrieval Part 3 — Memory, agent reasoning, skills, and MCP: https://blog.divyampatro.dev/clearcode-part-3-memory-agent-reasoning-skills-and-mcp Part 4 — Tasks layer: https://blog.divyampatro.dev/clearcode-part-4-autonomous-plan-execution-llm-as-judge-and-human-in-the-loop-approval