Notes on AI, retrieval,
and shipping good software.

The first time you point a code-analysis tool at a real repository, you discover something humbling: developers leak secrets constantly. Not maliciously — accidentally. An AWS key pasted into a test file because the CI environment wasn't set up yet. A bcrypt hash committed during a debug session. An entire customer email list dropped into a fixtures directory for a one-off migration. The repo isn't actively breached, but it's a smorgasbord for anyone who gains access.

For an embedding-and-RAG product, this matters more than for a static analyzer. The static analyzer reads the file and reports findings — the file stays on your disk. The RAG product reads the file, sends 1,000-character chunks to an embedding model, and stores the resulting vectors in a managed database. Every secret in your code becomes a vector that's been transmitted to a third party. Even if the embedding API doesn't train on inputs (Google's developer-tier Gemini doesn't), the secret has now crossed a boundary it shouldn't have.

Repotrix scrubs PII and credentials BEFORE chunking. The scrub happens in-memory inside the ARQ worker process, between the git clone and the chunker. Nothing leaves the worker un-redacted.

What we look for, in priority order:

1. **AWS credentials** — Access keys (`AKIA[0-9A-Z]{16}`), secret keys (40-char base64 in proximity to the word 'aws_secret'), STS session tokens (much longer). Pattern matching is cheap; we run it across every file.

2. **Generic API keys** — `sk_live_...` (Stripe), `xoxb-...` (Slack), `ghp_...` (GitHub), `AIza...` (Google), `eyJ...` (anything JWT-shaped). The list grows whenever we see a new pattern in production telemetry.

3. **Email addresses** — Standard RFC pattern. Customer emails in fixtures are the noisiest false-positive source for this; we tune the regex to also catch obvious obfuscations like `user [at] domain.com`.

4. **JWTs and bearer tokens** — A three-segment base64 string separated by dots and surrounded by either whitespace or quotes. The base64 alphabet check catches incomplete JWT-looking strings that aren't actually JWTs.

5. **Phone numbers and SSN-shaped strings** — North American formats by default; you can disable these for non-US repos to cut false positives.

When a match is found, we don't just delete it. Deletion changes line numbers, which silently breaks every later citation our chunker produces. Instead, we replace each match in-memory with a placeholder of the same length: `[REDACTED:AWS_KEY]` (padded with `_` to match the original). The file's structure stays exactly the same; the secret is gone.

We also keep a SHA-256 hash of every redacted value, never the value itself. The hash is stored in the compliance audit table alongside the file path and line number. If an auditor later asks 'did this specific exposed key ever leave our boundary?' we can prove it didn't by showing the hash entry — without us having to retain the secret to prove it.

False positives are real and have to be managed. The most common one we see is regex matches inside test fixtures — a base64 string in a unit test that LOOKS like an API key but is in fact a deliberately-fake placeholder. We mitigate this with a contextual filter: if the match sits inside a file whose path contains `test/`, `tests/`, `__fixtures__/`, or `mock`, AND the match is surrounded by string literal delimiters, AND no real key prefix matches (like `sk_live`), we lower the confidence score. High-confidence matches still get redacted; low-confidence matches get flagged in the compliance report so a human can review.

The biggest false NEGATIVE we've seen: bespoke internal API keys that don't match any public format. A company invents a key like `RPTX-2026-PROD-xxx` and we don't know about that pattern. Catching these requires either (a) the customer telling us about their key format, or (b) entropy analysis — flagging any high-entropy string above a certain length, accepting more false positives in exchange for fewer false negatives. We do (b) as a fallback for any string above 32 characters with high Shannon entropy, with a lower confidence score than the explicit pattern matches.

The net effect: by the time the ARQ worker hands chunks to the embedding service, the chunks contain zero detectable credentials. The embedded vectors in Pinecone don't carry your secrets. The cached LLM responses don't carry your secrets. The PII you exposed in your codebase doesn't escape your repository's boundary — even though it's right there in the code.

This is the unsexy part of an AI-on-code product, and we spend a lot of engineering time on it. The reason is simple: a single leak undoes years of trust, and we'd rather refuse to embed something ambiguous than send a key to a third party 'just in case it isn't real.' Better a slightly noisier compliance report than a customer's secret in a vector store we don't own.

Mermaid is a markup-to-SVG library that turns short text descriptions into flowcharts, sequence diagrams, and dependency graphs. LLMs are good at producing the markup. They're bad at producing markup that matches reality.

The failure mode is consistent: the model invents nodes (a service that doesn't exist), drops edges (skipping a dependency that does exist), or labels a relationship wrongly (calls a one-shot HTTP call a 'webhook' or vice versa). For a tool whose value proposition is 'this is your actual architecture, drawn for you', that's a brand-killer.

The fix is two-stage: a strict prompt that forces the model to ground every node and edge in the retrieved chunks, and a post-processing sanitizer that catches the most common syntactic mistakes Mermaid can't render. Together they take us from 'roughly correct most of the time' to 'rendered diagram every time, factually correct in the 95%+ case'.

The prompt: we feed the model the import graph + a list of route definitions + the top-K retrieved chunks for the repo. We tell it to draw ONLY relationships visible in those inputs. Every Mermaid edge must correspond to either an import, a function call, or an HTTP route present in the context. If the model invents an edge, the user can click through to the 'source' file and immediately see we made it up — so the model is told to omit, not guess.

The sanitizer: Mermaid syntax is finicky. Edges with arrows that have spaces in the wrong place fail to parse. Pipe characters inside labels need escaping. Node IDs that start with a digit are silently dropped. The model gets these wrong about 5% of the time. So we run the output through a regex pass that auto-repairs the known broken patterns before sending the markup to Mermaid for rendering. If sanitization fails (genuine syntax we can't fix), we render an error card with a link to re-run.

The result is a diagram with two properties that matter: factual (the relationships match what's in the code) and rendered (Mermaid actually produces an SVG). Skipping either step gives you what the LLM wants to show you, not what your codebase actually is.

A webhook URL is a public secret. The moment you give it to GitHub (or Slack, or Stripe, or any platform), anyone who learns the URL can POST to it. Without verification, every webhook endpoint is an unauthenticated mutation endpoint into your production code.

GitHub solves this with HMAC-SHA256 signatures. When you register a webhook, GitHub asks you to set a 'Secret' — a random string. On every delivery, GitHub computes HMAC-SHA256 of the request body using that secret as the key, and puts the result in an `X-Hub-Signature-256` header. Your server recomputes the HMAC using the same secret and constant-time-compares the two values. If they match, the request genuinely came from GitHub. If they don't, you reject it.

Three footguns that catch teams new to this:

1. **Verify against the RAW request body, not the parsed JSON.** Any whitespace difference between what GitHub signed and what your framework re-serialized will invalidate the signature. Read `request.body` before any JSON middleware touches it.

2. **Use `hmac.compare_digest`, not `==`.** Comparing two HMAC values with `==` short-circuits at the first differing byte, which leaks timing information about how many leading bytes the attacker got right. Constant-time compare always takes the same wall-clock time regardless of where the mismatch is.

3. **Reject deliveries with NO signature header.** It's surprisingly easy to write `if signature and not hmac_matches: reject` — but the `and` short-circuits when signature is missing, so unsigned requests sail through. Make the absence of a signature an explicit reject.

Replay protection comes next. GitHub includes an `X-GitHub-Delivery` header — a UUID for the specific delivery. If you persist seen UUIDs in Redis with a 24h TTL, you can detect and reject replays even if an attacker captures a signed request. We do this in Repotrix; it's about 12 lines of code and the storage cost is a few hundred KB even for a busy repo.

The last layer is an IP allowlist. GitHub publishes the IP ranges that send webhooks at https://api.github.com/meta. Restrict your webhook ingress to those CIDRs at the reverse-proxy layer (Cloudflare, ALB, Nginx). This won't catch a sophisticated attacker who's already on GitHub's network, but it shuts down 99% of the noise from the open internet.

Three controls — HMAC verification, replay detection, IP allowlist — and your webhook goes from 'cheapest backdoor' to a defensible authenticated channel. Skip any one of them at your peril.

Embedding choice gets the headlines. Chunking choice moves the metrics. In every RAG-quality experiment we've run on real code, the gap between a naive chunker and a careful one is wider than the gap between two reasonable embedding models.

The naive baseline is fixed-size character chunks: take the file, split it every 1000 characters, embed each piece. This is what every tutorial defaults to and it's wrong for code. Functions get split across chunk boundaries; an import statement at the top of a file lands in a different chunk from the code that uses it; class definitions get severed mid-method. The model receives chunks that don't carry their own context.

Strategy one — language-aware splitting. For each file, identify the language from the extension and use a parser-aware splitter. Python: split between top-level functions and class bodies. Go: between functions and type declarations. TypeScript/JavaScript: between exports, classes, and top-level functions. LangChain ships `RecursiveCharacterTextSplitter` with language-specific separator lists for this; we use it with a chunk size of 1000 chars and 200-char overlap. This single change is worth roughly 10 retrieval-recall points on our internal benchmark over the fixed-size baseline.

Strategy two — keep imports with the code. For most languages, the import block at the top of a file is critical context for understanding any chunk in that file. We prepend the file's import section (up to a 200-char cap) to every chunk we extract from that file. The chunk becomes self-contained — the model sees both 'this function uses jwt.encode' AND 'jwt is imported as PyJWT' in the same context. Recall on questions about library usage jumps another 8 points with this trick.

Strategy three — semantic anchors. For deeply-nested code (a method 5 levels deep inside nested classes inside a module), pure language-aware splitting loses the breadcrumb. We add a one-line anchor at the top of every chunk: 'In file X.py, inside class Foo, method bar:'. The model gets the full hierarchical path for free, without us having to embed the entire ancestor chain. This is worth another 4-6 points for code that lives in deep hierarchies (Java, C#, large Python codebases).

Measured against a fixed-size baseline, the three improvements compound: ~22 points of retrieval recall on a 200-question evaluation set we built by hand. The embedding model didn't change. The vector database didn't change. The LLM didn't change. We just cut the code up smarter before sending it to be embedded.

The takeaway is dull but useful: before you go shopping for a fancier embedding model or a larger vector database, look at what you're feeding into the embedder. The biggest win in RAG is almost always upstream of the model.

Large language models have huge context windows now — Gemini 3.5 Flash takes a million tokens in a single prompt. So why not just paste the whole repo into the chat and ask?

The answer is that attention is not free. Even with a million-token window, model performance degrades dramatically when the relevant information is buried in noise. Anthropic's needle-in-a-haystack benchmarks and Google's own RULER scores both show the same shape: as context length grows, recall for any specific fact buried in the middle drops sharply. A function definition on line 42,000 might as well not exist if the model has no signal pointing to it.

There's also the small matter of money. Even at Flash pricing, blasting a million tokens at the model for every user question costs around 30¢ per call. Across a team of 20 engineers asking 50 questions a day, that's a $9,000/month bill — and most of the input is being burned on tokens the model didn't need to look at.

Retrieval-augmented generation (RAG) flips the script. Instead of letting the model fish through everything, you fish first — find the specific chunks of code most relevant to the question — and then hand only those chunks to the model. The result: shorter prompts, sharper answers, lower cost, and crucially, citations the user can verify.

The pipeline in Repotrix looks like this. At ingestion time, the repo is cloned, split into roughly 1,000-character chunks with language-aware boundaries (we never split mid-function for Python or mid-block for Go), and each chunk is sent to Gemini's gemini-embedding-001 model. The resulting 3,072-dimensional vectors are upserted into a Pinecone namespace scoped to that single repository. We persist file path, line range, language, and commit SHA as metadata so we can reconstruct citations later.

At query time, the user's question is embedded by the same model, into the same 3,072-dimensional space. Pinecone runs an approximate-nearest-neighbour search — cosine similarity, under 50ms for repos of a few hundred thousand chunks — and returns the top-K matches. K is usually 8 to 16; bigger isn't always better, because at some point you reintroduce the haystack problem you just solved.

Those chunks become the model's context. The system prompt explicitly forbids using outside knowledge: every claim has to be grounded in one of the supplied chunks, and the model is instructed to cite the file and line range alongside its answer. When the question genuinely isn't answerable from the retrieved context, the model is told to say so rather than make something up. That last instruction is the difference between a chatbot that hallucinates and an assistant your team actually trusts.

One thing that took us a while to learn: RAG quality is rarely about the model — it's about the retrieval. If the top-K chunks are wrong, the world's best LLM will produce a confident wrong answer. So a lot of the engineering effort in a RAG system goes into chunking strategy, embedding model selection, hybrid retrieval (vector + keyword), reranking, and metadata filters. Each of these can move recall by 10 to 20 points.

The model never sees the rest of the codebase — and that's a feature, not a limitation. Less context means less noise, lower cost, faster responses, and an answer the user can audit by clicking through to the cited lines. RAG isn't a workaround for small context windows. It's the right architecture even when the window is infinite.