A 125M-parameter language model, trained from scratch on US case law, SEC filings, and educational web text — pure next-token prediction, no instruction-tuning.
| parameter | value | what it means |
|---|---|---|
| total parameters | 125.8M | size of the model — small enough to run inference on CPU with no GPU |
| vocabulary size | 16,384 | number of unique tokens the custom byte-level BPE tokenizer can produce |
| hidden size | 768 | width of each token's internal representation as it flows through the network |
| layers | 12 | number of stacked transformer blocks the input passes through |
| attention heads | 12 | parallel attention patterns computed per layer (head dim 64, standard multi-head attention) |
| feed-forward size | 3,072 | width of the expansion layer inside each block (SwiGLU activation) |
| context length | 1,024 | maximum number of tokens the model can attend over at once |
| tied embeddings | yes | input and output embedding matrices share weights, reducing parameter count |
| tokens per parameter | ~16.0 | 2.02B training tokens ÷ 125.8M params — a Chinchilla-style ratio; ~20 is considered compute-optimal |
Next-token prediction only, one epoch, no instruction data — this teaches the model to speak the legal/financial register fluently, but it has no notion of following an instruction or answering a question (that comes from the fine-tuned version).
| learning rate | 6e-4 → 6e-5 | step size for weight updates; cosine decay from a high starting rate |
| warmup | 200M tokens | gradually ramps up the learning rate at the start to avoid early instability |
| global batch size | 524,288 tok | total tokens averaged per optimizer step, via gradient accumulation |
| weight decay | 0.1 | regularization that discourages overly large weights |
| gradient clipping | 1.0 | caps the gradient norm to prevent destabilizing update spikes |