Drawer 06 · Maturity assessment

Condition report

CodeScope is a working prototype with a well-chosen architecture and no verification layer. The retrieval spine — AST chunking, dual indexes, rank fusion, cross-encoder reranking — is real, complete and defensible. Around it sit four unreachable modules, one placeholder endpoint, a summary store nothing writes to, a shutdown hook that raises, and zero automated tests.

06.1Assessment in one paragraph

The design decisions are sound and mostly current: Reciprocal Rank Fusion with k = 60, a cross-encoder over the fused head, syntax-aware chunking, contextual enrichment before embedding, and a provider abstraction that lets the embedding backend move between a local GPU and a hosted API. What is missing is everything that turns a prototype into a service: no tests, no CI, no migrations, no job persistence, no index compaction, no authentication at the service boundary, and a README that documents a module layout the project abandoned. The gap between “the algorithm is right” and “the system is dependable” is where all the remaining risk sits.

06.2Subsystem matrix

Maturity matrix of sixteen subsystems against five criteria A grid with sixteen labelled rows and five columns. The columns ask whether the subsystem is implemented, whether it is wired into the running application, whether any automated test exercises it, whether it has an explicit failure path, and whether its behaviour matches the README and product requirements document. Filled marks indicate yes, half marks indicate partial, and outlined marks indicate no. The test column is empty for every row because the repository contains no test files. Rows for the GitHub webhook, the background index worker, the server memory system and the legacy vector store are unfilled in the wiring column. condition matrix — 16 subsystems × 5 criteria subsystem implemented wired into the app covered by a test explicit failure path matches the written docs AST chunking · ast_chunker.py Local embeddings · embeddings.py Cloud embeddings · cloud_embeddings.py Vector store · vector_store_faiss.py Keyword index · zoekt.py Rank fusion · hybrid_search.py Reranking · reranker.py Query routing · query_router.py Contextual enrichment · contextual_retrieval.py Repository lifecycle · repo_manager.py Chat / RAG · routes/chat.py Code navigation · code_navigation.py Living summary · living_summary.py Server memory · services/memory.py GitHub webhook sync · github_sync.py Background indexing · index_worker.py legend yes partial — conditional, degraded, or reachable only by explicit call no The third column is empty for every row: the repository contains no test files at all — no pytest module, no conftest.py, no JavaScript spec, no CI workflow — although pytest and pytest-asyncio are declared under the dev extra in pyproject.toml:78.
Figure 06.1 — Condition matrix Column two is the operative one: four subsystems — the webhook sync, the background index worker, the server-side memory system and the pre-FAISS vector store — span six files and 2 286 lines that cannot be reached from a running instance.

06.3Not implemented, or implemented and disconnected

Item 06-1routes/index.py:39

POST /index is a placeholder

The endpoint walks the tree, counts files with one of eight extensions and returns a random 8-character id. It touches neither the chunker nor the vector store. The comment is explicit: “Placeholder implementation — counts files. Will be replaced with actual indexing in Task 14.3.” Real indexing happens only through the repository lifecycle.

Item 06-2services/living_summary.py

A store with no writer

LivingSummaryService is constructed at startup, its table is created, and chat.py reads from it for architecture questions. Nothing in the tree calls save_summary() or update_summary(). The summaries table is always empty, so get_summary_for_context() returns "" and the architecture branch adds nothing to the prompt.

Item 06-3routes/github.py

The webhook is never mounted

routes/__init__.py imports github, but main.py:356-364 does not include its router. Signature verification, push parsing and GitHubSync — 736 lines across two files — are unreachable. Consequently there is no automatic re-index on push, which is what index_worker.py was written to serve.

Item 06-4services/index_worker.py:315

The worker would not run if it were wired

_incremental_index() calls self.indexer._extract_chunks(file_path, relative_file) and reads chunk.line. CodeIndexer has no _extract_chunks method and CodeChunk has no line attribute — the chunker exposes chunk_file() and line_start. The incremental path has never executed.

Item 06-5routes/memory.py:44

Memory falls back to a stub

Seven endpoints try from autopilot.memory import AMEM. autopilot is not a dependency of this distribution, so StubAMEM answers — dictionaries in process memory, lost on restart. The 608-line ServerAMEM in services/memory.py is never referenced by the routes.

Item 06-6vector_store_faiss.py:54

No scaling path

HNSW_THRESHOLD is declared and never read. There is no quantisation, no IVF, no sharding. Search is exact inner product over every vector; latency grows linearly with corpus size. The project records this itself, in claude.md, and defers it in TASKS.md.

06.4Defects visible on reading

Register K — issues located by reading, with the line that causes them
No. Observation Site Effect
K-1 Chat retrieval never passes project_path, and the keyword backend requires it. chat.py:218 · zoekt.py:150 Chat is semantic-only; a KEYWORD_ONLY route returns zero chunks.
K-2 vector_store.close() is called on shutdown but the class defines no such method. main.py:262 AttributeError, swallowed and logged; the final index save does not happen.
K-3 CodeIndexer is constructed without data_path. main.py:175 · indexer.py:106 The hash manifest lands in ./data relative to the process CWD, outside every mounted volume.
K-4 Hash manifest keys are repo-relative paths, shared across all repositories. indexer.py:171 Two repositories with the same internal path collide; the second is skipped as unchanged.
K-5 Zoekt shard directories are named from project_path.name alone. zoekt.py:100, 174 Same-named clones overwrite one another's keyword index.
K-6 Path confinement uses str.startswith on resolved paths. repos.py:420 · files.py:187 A sibling directory whose name extends the repo path passes the check.
K-7 /navigate/index-file and /navigate/symbol-at-position accept absolute paths unconstrained. navigate.py:194, 173 Any file the process can read may be parsed or sampled through the API.
K-8 The service has no authentication of its own; CORS defaults to ["*"] with credentials allowed. config.py:19-20 · main.py:279 Everything depends on the Next.js layer being the only route to port 8080.
K-9 The symbol index is a per-process dictionary, populated only by explicit POST /navigate/index-file. code_navigation.py:126 Go-to-definition returns [] until a client indexes files one at a time; nothing does.
K-10 find_references is a regular-expression scan of every file under the repository root. code_navigation.py:319 Matches inside comments and strings; cost is a full tree read per call.
K-11 Reranking never receives the detected device. reranker.py:92 On a GPU host the cross-encoder runs on CPU, which is the slowest stage of the query.
K-12 Synonym substitution replaces case-sensitively and matches substrings. reranker.py:566 Capitalised queries expand to nothing; get rewrites the inside of widget.
K-13 Blocking work runs directly inside an async task. repo_manager.py:293 Indexing a repository stalls the event loop for its whole duration.
K-14 datetime.utcnow() is used in 14 places. repo_manager.py, index_worker.py Deprecated since Python 3.12, which is the image's interpreter; naive timestamps are stored.
K-15 Six bare except: clauses. files.py:109, 205 · routes/memory.py:306, 386 · services/memory.py:446, 586 Catches KeyboardInterrupt and SystemExit; violates the project's own ruff E selection.
K-16 The Claude model tables in the backend and the Next.js handler disagree. chat.py:37 · web/src/app/api/chat/route.ts The same UI selection resolves to different model ids on the two paths.
K-17 useKeyboardShortcuts is exported and never imported. web/src/hooks/use-keyboard-shortcuts.ts Four of the five documented shortcuts do nothing; only the duplicated ⌘K handler in the shell works.
K-18 Anthropic streaming is emitted as one JSON line built by string interpolation. chat.py:647 Not streaming, and any quote or newline in the answer produces invalid JSON.

06.5Verification and tooling

Automated tests
None. No test_*.py, no conftest.py, no *.test.ts, no *.spec.ts, no tests/ directory anywhere in the repository — while pytest and pytest-asyncio are declared under the dev extra and the README documents pytest as the way to run them.
Continuous integration
No workflow definitions of any kind. Deployment is a shell script.
Type checking
[tool.mypy] strict = true is configured (pyproject.toml:101). The tree would not pass it: services are full of Optional[Any] parameters, untyped inner functions and implicit returns.
Linting
Ruff is configured with line-length = 88 and select = ["E","F","I","N","W","B","Q"] — under the legacy [tool.ruff] table rather than [tool.ruff.lint]. Twenty-eight of the forty-one Python files contain at least one line over the configured limit; there are 120 such lines in total, plus the six bare except: clauses that E722 would reject. Neither tool is run anywhere in the repository.
Documentation drift
README.md:86-98 documents an architecture with routes/health.py, routes/tasks.py and services/executor.py; none exist. It lists five environment variables and no search, indexing or chat endpoints. PRD.md proposes chat_service.py, symbols.py and symbol_index.py; the functionality was built, under other names.
Docstrings
The one consistently strong artefact. Nearly every class and method carries an accurate Args/Returns docstring, and several record their own limitations candidly — including the FAISS deletion comment and the index.py placeholder note. Reading this codebase is unusually easy for its stage.

06.6What the project already knows

claude.md in the repository root is a self-critique that grades the system 6.3 out of 10 against contemporary code-search products. It reaches three of the same conclusions independently: that IndexFlatIP does not scale and the declared HNSW threshold is never applied; that raw code is embedded with no natural-language rewrite; and that background incremental indexing is absent. TASKS.md then records five completed phases — the limit increase to 50, Cohere reranking, the query router, the living-summary store, and the confidence and sources fields — and moves HNSW scaling, a code-specific embedding model and background indexing to a backlog.

Where the record and the code diverge

Phase 3 in TASKS.md is marked complete, including “generate repo overview during indexing” and “add incremental update on re-index”. What exists is the storage and retrieval half: LivingSummaryService persists, formats and budget-trims summaries correctly, and chat.py injects them. The generation half — anything that would call save_summary() during indexing — is not in the tree. Phase 2's fast keyword path is likewise marked complete but is defeated by K-1.

06.7What is genuinely solid

  • The fusion layer. RRF with per-list weights, normalisation against the batch maximum and a clean key on file:line is textbook and correctly implemented.
  • Graceful degradation. Almost every optional dependency — FAISS, tree-sitter grammars, sentence-transformers, Cohere, Zoekt, Ollama — is import-guarded or probe-guarded with a working fallback. The service starts and answers on a laptop with none of them.
  • The model fallback chain. EmbeddingService._load_model() tries progressively smaller models and, critically, reports the width that actually loaded so the FAISS index is created to match, rather than trusting a constant.
  • Provider abstraction. UnifiedEmbeddingService gives the indexer one synchronous interface over a local model and two hosted APIs, including the awkward sync-over-async bridge, and correctly rebuilds the index when the width changes.
  • Chunk identity. Deriving the chunk id from path:line:type makes re-indexing idempotent for unchanged code without a separate identity table.
  • Transparency in the response. Returning the routing decision, its confidence, its prose reasoning, the chunk count and the per-file sources is more than most retrieval services expose, and makes the system debuggable from the client.

06.8The shortest path to dependable

  1. Pass project_path from chat.py, or drop the keyword leg from that call site honestly — K-1 silently halves retrieval for the flagship feature.
  2. Add close() to FAISSVectorStore, or call save() on shutdown — K-2 loses everything written since the last explicit save.
  3. Namespace the hash manifest and the Zoekt shard by repo_id — K-4 and K-5 both produce wrong results rather than errors.
  4. Move CodeIndexer.index() onto an executor — K-13 is the difference between a service that stays responsive during indexing and one that does not.
  5. Write the first test: chunk a fixture file, embed it with a stub, fuse two synthetic lists, and assert the RRF ordering. The retrieval spine is pure enough to test in isolation, which is exactly why the absence of tests is conspicuous.
  6. Either mount the webhook router and fix index_worker.py, or delete both — 1 141 lines of unreachable code is a maintenance liability that reads like a feature.