Drawer 01 · Ingest pipeline

Accessioning a repository

One POST /repos starts a background task that clones a repository, walks it, cuts it into syntax-aware chunks, optionally writes a one-paragraph description of each chunk, embeds them in batches of 32, and appends the vectors to a FAISS index. The request returns before any of that begins.

01.1The accession slip

compute_service/routes/repos.py : 79-149POST /repos
@router.post("", response_model=AddRepoResponse)
async def add_repo(
    request: AddRepoRequest,
    req: Request,
    x_github_token: Optional[str] = Header(None, alias="X-GitHub-Token"),
    x_embedding_provider: Optional[str] = Header(None, alias="X-Embedding-Provider"),
    x_embedding_key: Optional[str] = Header(None, alias="X-Embedding-Key"),
) -> AddRepoResponse:
    # ... optional cloud-embedding reconfiguration, then:
    record = await repo_manager.add_repo(
        url=request.url, branch=request.branch,
        name=request.name, github_token=x_github_token,
    )
Header side effects. Supplying X-Embedding-Provider and X-Embedding-Key on this request does more than authenticate: if the new provider's vector width differs from the current one, repos.py:126 calls vector_store.reinitialize(new_dims), which deletes faiss.index and metadata.pkl and starts an empty index. Every previously accessioned repository becomes unsearchable at that moment.

01.2The pipeline

The accession pipeline from HTTP request to stored vector Seven numbered stages running down the page. Stage one, the HTTP request, inserts a database row and spawns a background asyncio task, returning immediately. Stage two performs a shallow git clone and records the head commit. Stage three shells out to zoekt-index. Stage four collects candidate files through three filters: skip directories, extension allow-list, and a one megabyte size cap. Stage five loops per file computing a truncated SHA-256, cutting chunks with tree-sitter, and optionally calling a local Ollama model once per chunk. Stage six batches thirty-two chunks at a time, builds an embedding text from chunk type, name, signature and contextualised body, and appends the normalised vectors to FAISS. Stage seven saves the index, writes the file-hash manifest, and flips the repository status to ready. Side notes mark where work accumulates in memory and where the persistent embedding cache is bypassed. accession pipeline — repo_manager.py + indexer.py + ast_chunker.py 1 · POST /repos repos.py:79 returns before any work uuid4()[:8] → repo_id status = "cloning" INSERT INTO repos asyncio.create_task( _clone_and_index ) repo_manager.py:182 task handle kept in a dict on the RepoManager; lost on restart — no job persistence 2 · git clone --branch B --depth 1 repo_manager.py:273 git rev-parse HEAD → last_commit status = "indexing" Destination path is repos_path / user_id / owner / repo_name. A token, from the header or GITHUB_TOKEN, is spliced into the clone URL. Non-zero exit → status = "error", stderr stored in error_message. 3 · zoekt-index subprocess, 300 s cap zoekt.py:107 Runs first, before any Python-side work, and only if both zoekt-index and zoekt are on PATH. The index directory is named after the last path segment alone (zoekt.py:100), so two repositories whose directories share a name write into the same Zoekt index. 4 · _collect_files rglob("*") over the tree indexer.py:294 returns List[Path] filter 1 — SKIP_DIRS .git · node_modules · dist .venv · target · vendor 20 names, matched on parts filter 2 — extension 28-entry allow-list incl. .md .txt .json .yaml .sql indexer.py:49 filter 3 — size MAX_FILE_SIZE = 1 048 576 bytes larger files dropped 5 · per-file loop — indexer.py:167-211 sha256(bytes)[:16] compare to manifest unchanged → files_skipped read_text(errors=ignore) whole file into memory used as context document chunker.chunk_file() tree-sitter walk, or 80-line windows if no parser contextual_retrieval one Ollama /api/generate call per chunk, sequential All chunks for every file accumulate in three parallel Python lists — all_chunks, chunk_repo_ids, chunk_contexts — before a single embedding is computed. Peak memory is the whole corpus of chunk text, not one file's worth. 6 · batch loop — indexer.py:238-292, EMBEDDING_BATCH_SIZE = 32 build embedding text f"{chunk_type} {name}:\n" f"{signature}\n\n{context}" falls back to bare content embed_documents(texts) local model or cloud provider use_cache = False on this path L2-normalised on return vector_store.add_batch(docs) (id, file, line, content, vec, repo) faiss.normalize_L2 then index.add metadata kept in two dicts The periodic autosave fires only when the running vector count lands exactly on a multiple of 10 000 at a batch boundary. 7 · commit vector_store.save() _save_hashes() status = "ready" faiss.index metadata.pkl (pickle) written whole, every save no incremental format file_hashes.json path → sha256[:16] keys are repo-relative, so paths collide across repos repos.db files_indexed, last_indexed set by _update_repo() repo_manager.py:299
Figure 01.1 — Accession pipeline Stages 4 through 7 are one synchronous function, CodeIndexer.index(), called from an async task without an executor (repo_manager.py:293). While a large repository is being chunked and embedded, the event loop is blocked and the service will not answer other requests.

01.3What a chunk is

ASTChunker declares node types per language and walks the whole tree, recording every node whose type matches a function, class or import entry for that language. Because the walk is recursive and unconditional, a method inside a class produces a chunk and its enclosing class produces a chunk containing it — the same lines are embedded twice at two granularities. Ranges are de-duplicated by (start_line, end_line) only (ast_chunker.py:364), which does not catch nesting.

Register C — chunk geometry, ast_chunker.py:119-256
FieldSourceNotes
idf"{path}:{line}:{chunk_type}"Also the FAISS metadata key; stable across re-index of unchanged code.
filepath relative to repo rootCarried into search results verbatim.
line_start / line_endtree-sitter start_point / end_point, 1-basedOnly line_start reaches the vector store.
contentjoined source linesRaw code. No natural-language rewrite is performed.
chunk_typefunction · class · import · module · blockmodule = residue not covered by a node; block = line-window fallback.
namefirst identifier-like childPositional, not field-based; returns None for many node shapes.
signaturefirst ≤10 lines until parens balancePrepended to the embedding text when both name and signature exist.
MAX_CHUNK_LINES = 150
Nodes longer than this are split into overlapping windows by _split_large_chunk(); parts are renamed name (part n) and lose their signature.
MIN_CHUNK_LINES = 5
Uncovered regions shorter than five lines are discarded rather than emitted as module chunks.
OVERLAP_LINES = 10
Applied when splitting oversized nodes and in the no-parser fallback, where the window is 80 lines with 10 lines of overlap.
Language coverage
Fourteen languages have node maps. Five ship as hard dependencies — Python, JavaScript, TypeScript, Go, Rust. The other nine (Java, C, C++, C#, Ruby, PHP, Swift, Kotlin, Scala) sit behind the languages extra in pyproject.toml:62. Each grammar import is individually guarded, so a missing grammar silently downgrades that language to line-window chunking.
Files indexed but never parsed
The indexer's extension allow-list includes .md, .txt, .rst, .json, .yaml, .toml, .sql and shell scripts. None of these appear in ASTChunker.LANGUAGE_MAP, so documentation and configuration are always chunked as 80-line blocks.

01.4Contextual enrichment

ContextualRetrievalService implements the published contextual retrieval pattern: before embedding, ask a small model to situate the chunk inside its file, then prefix that sentence to the text being embedded. The prompt is built at contextual_retrieval.py:222 and the result is formatted as [From {file}: {context}]\n\n{content}.

compute_service/services/contextual_retrieval.py : 43-60class constants
OLLAMA_URL = os.environ.get("OLLAMA_URL", "http://localhost:11434")
CONTEXT_MODEL = os.environ.get("CONTEXT_MODEL", "llama3.2:3b")
MAX_FILE_CONTEXT = 6000      # chars of the file sent as the document
MAX_CONTEXT_LENGTH = 200     # num_predict for the generated context
TIMEOUT = 30.0
AVAILABILITY_CACHE_TTL = 60.0
Cost shape. Enrichment is on by default (use_contextual_retrieval=True, indexer.py:87) and runs one HTTP request per chunk, in a plain for loop (contextual_retrieval.py:196). There is no batching, no concurrency and no cache of generated contexts. If Ollama is unreachable the availability probe fails, the result is cached for sixty seconds, and every chunk is embedded as raw code — silently, with no change in the indexing result object.

01.5Re-accession and incremental work

  1. Change detection is per file, not per chunk. A file whose SHA-256 prefix matches the manifest is skipped entirely (indexer.py:173). A file that changed is fully re-chunked and fully re-embedded.
  2. Superseded vectors are not removed. Re-indexing a changed file appends new vectors; the old ones stay in the FAISS index. Only the metadata mapping is overwritten, and only for identical chunk ids.
  3. Deletion is metadata-only. delete_by_repo() and delete_by_file() pop entries out of _id_to_idx and _idx_to_doc (vector_store_faiss.py:362-416). The comment is candid: “FAISS doesn't support true deletion.” The index file therefore grows monotonically until someone calls clear().
  4. The manifest is global. CodeIndexer receives no data_path from main.py:175, so it uses its default of Path("./data") (indexer.py:106) — one file_hashes.json for the whole service, keyed by repo-relative path. Two repositories that both contain src/main.py share a key.
  5. There is no watcher. Re-indexing happens when someone calls POST /repos/{id}/reindex. The push-webhook path that would automate it exists in routes/github.py and services/github_sync.py but its router is not included in main.py.

01.6Accession in the log

Terminal 2 · GPU profile bring-upmessages as emitted by main.py and services/
$ docker compose --profile gpu up --build
autopilot-compute-gpu  | INFO:compute_service.main:Starting Compute Service...
autopilot-compute-gpu  | INFO:compute_service.main:GPU available: True (device: cuda)
autopilot-compute-gpu  | INFO:...main:Initializing embedding service with model: BAAI/bge-m3
autopilot-compute-gpu  | INFO:...embeddings:EmbeddingService initialized with device: cuda
autopilot-compute-gpu  | INFO:...embeddings:Loading embedding model: BAAI/bge-m3
autopilot-compute-gpu  | INFO:...embeddings:Model loaded successfully: BAAI/bge-m3, Dimensions: 1024, Device: cuda
autopilot-compute-gpu  | INFO:...main:Embedding model loaded with 1024 dimensions
autopilot-compute-gpu  | INFO:...main:Unified embedding service initialized (supports local + cloud)
autopilot-compute-gpu  | INFO:...main:Initializing FAISS vector store with 1024 dimensions...
autopilot-compute-gpu  | INFO:...vector_store_faiss:Loaded FAISS index with 0 vectors
autopilot-compute-gpu  | INFO:...main:Initializing Zoekt search service...
autopilot-compute-gpu  | INFO:...main:✓ Zoekt available for fast keyword search
autopilot-compute-gpu  | INFO:...ast_chunker:Available tree-sitter languages: ['python', 'javascript', 'typescript', 'go', 'rust']
autopilot-compute-gpu  | INFO:...ast_chunker:Initialized tree-sitter parsers: ['python', 'javascript', 'typescript', 'go', 'rust']
autopilot-compute-gpu  | INFO:...main:AST chunker initialized for semantic code extraction
autopilot-compute-gpu  | INFO:...reranker:Using CrossEncoderReranker (local model)
autopilot-compute-gpu  | INFO:...main:Reranker initialized: CrossEncoderReranker
autopilot-compute-gpu  | INFO:...main:Initializing repo manager at /app/data/repos...
autopilot-compute-gpu  | INFO:...main:All services initialized successfully
autopilot-compute-gpu  | INFO:...living_summary:Living summary database initialized at /app/data/cache/summaries.db
autopilot-compute-gpu  | INFO:...main:Living summary service initialized
autopilot-compute-gpu  | INFO:...main:Server running on 0.0.0.0:8080
autopilot-compute-gpu  | INFO:...main:Max concurrent tasks: 10
Terminal 3 · accessioning one repositorycounts are illustrative; message text is not
$ curl -sS -X POST http://localhost:8080/repos \
    -H 'Content-Type: application/json' \
    -H "X-GitHub-Token: $GITHUB_TOKEN" \
    -d '{"url":"https://github.com/OWNER/REPO","branch":"main"}'
{"id":"4f2a91c7","name":"REPO","status":"cloning","message":"Repository REPO added, cloning started"}

# service log, background task
INFO:...repos:Adding repository: https://github.com/OWNER/REPO
INFO:...indexer:Indexing project: /app/data/repos/global/OWNER/REPO (repo_id: 4f2a91c7)
INFO:...zoekt:Successfully indexed /app/data/repos/global/OWNER/REPO to /app/indexes/zoekt/REPO
INFO:...indexer:Zoekt indexing complete
INFO:...indexer:Found 412 files to index
INFO:...indexer:Generating embeddings for 3184 chunks...
INFO:...indexer:Processed 500/3184 chunks
INFO:...indexer:Processed 1000/3184 chunks
INFO:...indexer:Indexing complete: 412 files, 3184 chunks, skipped 0, errors: 0
INFO:...repo_manager:Repository 4f2a91c7 ready, indexed 412 files

$ curl -sS http://localhost:8080/repos | python3 -m json.tool
{
    "repos": [
        {
            "id": "4f2a91c7",
            "name": "REPO",
            "branch": "main",
            "status": "ready",
            "files_indexed": 412,
            "last_indexed": "2025-11-18T09:41:07.512884",
            "last_commit": "9c1e0b4a…",
            "error_message": null
        }
    ],
    "total": 1
}
Read alongside

The status values cloning → indexing → ready | error are the only progress signal available. There is no chunk-level progress endpoint, and the files_indexed figure is written once, at the end. A repository that fails mid-embed remains indexing until the process is restarted. See Drawer 06.