Drawer 04 · Storage & schema

Where the collection is shelved

There is no external datastore. Vectors sit in a FAISS flat index beside a pickled metadata map; repository records, two embedding caches and the repository summaries each get their own SQLite file; keyword shards belong to Zoekt; and the incremental-indexing manifest is a JSON file written relative to the process working directory.

04.1Shelf map

Map of every persistent store and its schema Two configured roots, the data path and the index path, expand into their contents. Under the data path are a FAISS index directory holding faiss dot index and metadata dot pickle, a cache directory holding three SQLite files, and a repos directory holding cloned working trees plus repos dot db. Under the index path is a zoekt directory of shards named after the last path segment of each clone. A seventh store, file hashes dot json, hangs outside both roots because it is written relative to the process working directory. Beneath the map, four schema cards give the exact columns of the repos table, the embeddings cache table, the summaries table, and the structure of the pickled FAISS sidecar. persistent state — defaults from config.py:36-38 COMPUTE_DATA_PATH default /app/data COMPUTE_INDEX_PATH default /app/indexes process working directory — not configurable, not mounted as a volume in either compose profile faiss_index/ faiss.index — IndexFlatIP metadata.pkl — pickle sidecar cache/ embeddings.db — local service unified_embeddings.db — router summaries.db — living summary repos/ <user_id>/<owner>/<name>/ clones repos.db — record table zoekt/ one directory per clone, named project_path.name only written by the zoekt-index binary ./data/file_hashes.json path → sha256[:16], one manifest for the whole service; indexer.py:106 default, never overridden volume bindings, docker-compose.yml — gpu profile ${CODESEARCH_DATA_PATH:-./data} → /app/data ${CODESEARCH_INDEXES_PATH:-./indexes} → /app/indexes ${CODESEARCH_REPOS_PATH:-./repos} → /app/repos ${CODESEARCH_CACHE_PATH:-./cache} → /app/cache (HF_HOME) /app/repos is bound but unused: RepoManager clones into /app/data/repos. repos.db · table repos — repo_manager.py:76 id TEXT PRIMARY KEY uuid4()[:8] name TEXT NOT NULL url TEXT NOT NULL branch TEXT DEFAULT 'main' status TEXT DEFAULT 'pending' local_path TEXT files_indexed INTEGER DEFAULT 0 last_indexed TEXT last_commit TEXT error_message TEXT created_at TEXT NOT NULL metadata TEXT DEFAULT '{}' user_id TEXT DEFAULT 'global' metadata.pkl · pickled dict — vector_store_faiss.py:181 { "id_to_idx": Dict[str, int] chunk id → row "idx_to_doc": Dict[int, VectorDocument] "next_idx": int monotonic counter } VectorDocument = id · file · line · content · repo_id · context · embedding · score The full chunk text is stored here, not only a pointer — the sidecar is the corpus, and the pickle is loaded on every start with no schema check. embeddings.db · table embeddings — embedding_cache.py:82 key TEXT PRIMARY KEY sha256(text) embedding BLOB NOT NULL float32 tobytes() created_at REAL NOT NULL hit_count INTEGER DEFAULT 0 INDEX idx_created_at (created_at) · TTL 7 days · LRU cap 10 000 No dimension or model column: vectors of different widths share a table. summaries.db · table summaries — living_summary.py:57 repo_id TEXT PRIMARY KEY overview TEXT NOT NULL module_summaries TEXT DEFAULT '{}' JSON object generated_at TEXT NOT NULL file_count INTEGER · chunk_count INTEGER Written by save_summary() / update_summary() — which nothing calls. index widths must agree — the FAISS index is created with the dimension the embedding model actually reported at load time (main.py:111). Changing provider or model without deleting faiss_index/ leaves a width mismatch that surfaces as a FAISS assertion.
Figure 04.1 — Shelf map and schemas Six stores, four of them SQLite, one a FAISS binary with a pickled sidecar, one a directory of Zoekt shards. Column lists are transcribed from the CREATE TABLE statements.

04.2The vector store in detail

compute_service/services/vector_store_faiss.py : 114-133_init_index
def _init_index(self) -> None:
    # Start with flat index (exact search)
    # Will upgrade to HNSW when threshold is reached
    self._index = faiss.IndexFlatIP(self.dimensions)   # inner product = cosine on unit vectors

    if self.use_gpu:
        try:
            res = faiss.StandardGpuResources()
            self._index = faiss.index_cpu_to_gpu(res, 0, self._index)
        except Exception as e:
            logger.warning(f"GPU not available for FAISS: {e}")
The upgrade never happens. HNSW_THRESHOLD is declared at line 54 and referenced nowhere else in the tree; no code path constructs IndexHNSWFlat or any quantised variant. Search is exact inner product over every stored vector, always. The project's own notes in claude.md flag this as a known gap and place it in the backlog in TASKS.md with the annotation “only needed at 50k+ vectors”.
Register G — FAISS store behaviours worth knowing before operating it
OperationWhat actually happensSite
add_batch() Vectors are L2-normalised in place and appended; metadata rows written to two dicts. Returns the count, never fails loudly if FAISS is absent — it returns 0. :248
add() On an existing id it appends a second vector and repoints the metadata; the superseded vector stays in the index and can still be returned, then dropped because its row is no longer in _idx_to_doc. :216
search() Over-fetches — k = limit × 5 when filtering by repository, limit × 2 otherwise — then filters in Python and stops at limit. A repository with few chunks in a large index can be starved. :326
delete_by_repo() Metadata-only. The vectors remain; only the mapping is removed. Reported document counts drop, index.ntotal does not. :399
reinitialize() Unlinks faiss.index and metadata.pkl and starts empty. Triggered automatically when an embedding provider with a different width is configured. :135
close() Not defined. main.py:262 calls vector_store.close() on shutdown inside a try, so the resulting AttributeError is caught and logged as a warning. The final save on shutdown does not occur. main.py:260

04.3Concurrency and durability

  • The index is per process. FAISSVectorStore holds the index in memory and writes whole files. Running ./scripts/start.sh prod starts four uvicorn workers, each with its own copy loaded at boot and each writing the same two paths. There is no lock and no shared-memory path.
  • SQLite access is synchronous and connection-per-call. RepoManager opens sqlite3.connect(), runs one statement and closes, inside async def methods (repo_manager.py:189, 218, 336, 359, 423). aiosqlite is a declared dependency but is never imported.
  • The embedding cache commits per row. EmbeddingCache.set() issues an INSERT OR REPLACE and a commit() for each entry (embedding_cache.py:212). This only bites on the query path, because document embedding passes use_cache=False.
  • The sidecar is a pickle. Loading metadata.pkl reconstructs VectorDocument instances, so the file is bound to the class's import path and to the Python that wrote it. A failed load falls through to _init_index() (vector_store_faiss.py:108-110) — a corrupt sidecar silently empties the catalogue rather than refusing to start.
  • Zoekt shards are named by directory basename. index_name = project_path.name at both zoekt.py:100 and zoekt.py:174. Two clones called api share one shard directory, and clear_index() called with no argument removes the entire zoekt/ tree.

04.4Configuration surface

Settings come from a single pydantic-settings class with the prefix COMPUTE_ and an optional .env file. A second, unrelated set of variables is read directly from os.environ deeper in the tree; those do not take the prefix.

Register H — configuration, by mechanism
VariableDefaultRead byMechanism
COMPUTE_HOST0.0.0.0config.py:13Settings
COMPUTE_PORT8080config.py:14Settings
COMPUTE_WORKERS1config.py:15Settings
COMPUTE_CORS_ORIGINS["*"]config.py:19Settings
COMPUTE_DATA_PATH/app/dataconfig.py:37Settings
COMPUTE_INDEX_PATH/app/indexesconfig.py:38Settings
COMPUTE_MAX_CONCURRENT_TASKS10config.py:25Settings — logged at startup, never enforced
COMPUTE_EMBEDDING_MODELBAAI/bge-base-en-v1.5config.py:29Settings — shadowed below
EMBEDDING_MODELBAAI/bge-m3main.py:96os.environ, checked first
EMBEDDING_PROVIDERlocalmain.py:122os.environ
EMBEDDING_API_KEY—main.py:123os.environ
RERANKER_MODELBAAI/bge-reranker-v2-m3reranker.py:74os.environ, read at class definition
COHERE_API_KEY—reranker.py:479os.environ, decides the reranker
OLLAMA_URLhttp://localhost:11434chat.py:28os.environ
OLLAMA_MODELqwen2.5-coder:14bchat.py:29os.environ
CONTEXT_MODELllama3.2:3bcontextual_retrieval.py:48os.environ, read at class definition
ANTHROPIC_API_KEY—chat.py:34os.environ, header takes precedence
GITHUB_TOKEN—repo_manager.py:265os.environ, header takes precedence
Two model settings, one winner

Settings.embedding_model defaults to BAAI/bge-base-en-v1.5, but main.py:96 reads the bare EMBEDDING_MODEL environment variable first and only falls back to the setting, whose own default is a third value. Setting COMPUTE_EMBEDDING_MODEL works only when EMBEDDING_MODEL is unset. Class-level reads in reranker.py and contextual_retrieval.py happen at import, so those two cannot be changed by a .env file loaded later.

04.5Declared but unused dependencies

Six runtime dependencies in pyproject.toml have no import anywhere in the package. Three of them are the official SDKs for services the code does call — over raw httpx instead.

anthropic ≥ 0.21.0
Claude is called by hand-rolled POST to /v1/messages with an anthropic-version header (chat.py:376).
voyageai ≥ 0.3.0
Voyage is called via httpx.AsyncClient(base_url="https://api.voyageai.com/v1") (cloud_embeddings.py:130).
openai ≥ 1.0.0
OpenAI embeddings likewise, through a raw client at cloud_embeddings.py:275.
aiosqlite ≥ 0.19.0
All four SQLite users are synchronous sqlite3.
anyio ≥ 4.0.0
No direct import; present transitively through Starlette regardless.
python-multipart ≥ 0.0.6
No endpoint accepts form data or file uploads.

cohere is the exception: it is imported and used for real (reranker.py:37), guarded so that its absence downgrades to the local cross-encoder rather than breaking the import.