6 projects in development.

How much has a model actually read about this?

DataSubjects is a study I'm running on estimating the per-subject composition of an LLM's training data — how often a given subject was sampled — from the outside, with calibrated uncertainty rather than vibes.

The primary estimator needs no access to the model at all. A BPE tokenizer's ordered merge list is a fossil record of the corpus it was built on: each merge constrains the mixture of data the tokenizer saw, and solving those constraints recovers absolute byte-share proportions from nothing but the shipped tokenizer.json — zero inference calls, zero cost. A second, behavioural estimator calibrates signals that scale log-linearly with exposure (cloze accuracy, Min-K% logprobs, perturbation deltas) against models whose training data is public and countable.

The two estimators measure different things — what went into the pool versus what the model was actually trained on — so their disagreement is the interesting signal. It reverse-engineers a data-curation changelog: which subjects got upsampled, and which got filtered out.

Written as a design document of record first — estimands, access regimes, and an experiment matrix where every experiment has a kill criterion agreed in advance. Two have passed so far; the next one runs on Pythia checkpoints.