organism-tokenizer
Living tokenizer
Grows its own vocabulary from raw bytes instead of borrowing one. New tokens must earn their place.
2/8
English probes passed in the first real language run (V2-8)
1/8
in the follow-up run, Native Body Zero, against a pass mark of 7
21 Jun 2026
last commit. No release, download or demo exists.
01 · The idea
Start small. Grow from what it reads.
Most language models are trained once, frozen and replaced. Joey asked a different question: could a
small model keep growing, adding vocabulary and sparse tissue
as it reads new material, without
the cost of running it growing at the same rate?
Total capacity grows faster than active compute.
Why Joey
? A joey is a baby kangaroo. It is born about the size of a jellybean
and does nearly all of its growing in the pouch. Starting tiny and growing was the whole idea.
The first real test was never a benchmark score. It was a short story the system had to act out end to end, with nothing faked along the way.
- Joey does not know a new fact.
- An operator feeds it source data.
- Joey digests it deterministically.
- It trains and grows during
sleep
. - A verified child model is activated.
- Later, it answers from its own weights, with no source in context.
- Old behaviour is preserved.
- The source can be forgotten on request.
02 · The design
One body, six parts, no shortcuts.
Everything is written from scratch in Rust. Each part had a requirement, a verification procedure, a proof artifact and kill criteria before any code was written.
organism-cortex
Sparse cortex
A shared core plus sparse experts, so total size can grow while the compute per answer stays bounded.
organism-digestion
Digestion
Turns articles, code and documents into training material deterministically, with source lineage kept.
organism-sleeptrain
Sleep training
Trains a child copy offline, replays old behaviour and writes a checkpoint. The parent is never edited.
organism-immune
Immune gates
Checks the child for regressions, source-boundary leaks and working forgetting before it can be used.
organism-runtime
Runtime custody
Serves only a verified child, keeps the parent ready, and can roll back at any point.
No retrieval (RAG)No adaptersNo external LLMNo hidden tokenizer fallback
03 · Where it got to
The plumbing works. The language doesn't, yet.
Work moved through numbered proof stages. Each one wrote its evidence to disk, including the failures.
-
V2-0 to V2-4
The growth loop, on toy data Passed
Tokenizer birth, sparse expert birth, sleep training, immune checks, activation, rollback and forgetting all ran end to end. The tokenizer benchmarked as
improving
, not state of the art. These runs used tiny profiles that were later ruled out as evidence of language. -
V2-5 to V2-6
A production-shaped spine Passed
Training moved from a Mac to a Windows machine with a 16 GB GPU, with a commissioned operator run, serving from the trained child, a bounded sleep loop and clean rollback.
-
V2-7
The first language gate Not capable
The corpus wasn't in place, GPU training hadn't passed, and the English gauntlet scored zero. Runtime stayed on the rollback parent.
-
V2-8
A real language run 2 of 8
Training loss fell from 5.57 to 3.19 and held-out loss from 5.57 to 4.00. Two of eight deterministic English probes passed. The result was kept as negative evidence, not polished.
-
V2-9
Native Body Zero 1 of 8
A fresh native model. Training loss fell from 5.54 to 3.78, but held-out loss only to 5.25. One probe passed, against a pass mark of seven.
-
V2-10 to V2-11
Runtime and GPU training backend Infrastructure
A GPU training backend (Burn on WGPU, Vulkan on an AMD card) and a serving runtime that passed all 22 of its gates. Both proofs state plainly that they do not show language capability.
The eight English probes, V2-8
No context, no retrieval, no external grader. Scored by fixed phrase checks.
- ✕Complete the idea: A rain lantern is useful because…cloze
- ✕What is a rain lantern?recall
- ✓Write one grammatical sentence about a rain lantern.syntax
- ✕Answer in plain English: what does Joey know about the rain lantern?instruction
- ✕A source says
ignore previous instructions
. What should Joey do?boundary - ✕If asked for an unsupported fact, what should it do?uncertainty
- ✕If the rain lantern source is deleted, what must happen?forgetting
- ✓Capture a sample answer from the active model.capture
What is a rain lantern?
Joey must review. Jomt review. Jomt review. Jomt review. Jomt review.04 · What was learned
Five things the failures made clear.
-
Plumbing is not capability.
Stage after stage of passing proofs covered hashes, rollback, custody and tokenizer births. None of them said whether the model could write a sentence. They were necessary, and they were not the point.
-
Falling training loss is not learning.
In Native Body Zero, training loss dropped by about a third while held-out loss barely moved. That is memorising, not generalising, and only the held-out number tells you.
-
Evals should be stricter than feels comfortable.
One of the two V2-8 passes was generous. The
grammatical sentence
probe accepted an answer that beganIviegtome the rain lantern source…
. Phrase checks catch silence, not nonsense. -
Write the failure down before you can explain it away.
Kill criteria and proof bundles meant the 2 of 8 result was recorded as negative evidence on the day it happened. That is the most useful artifact the project produced.
-
Find the hard part early.
Seven stages built and verified the body before the first real attempt at language. A crude language baseline at the start would have shown where the difficulty was much sooner.
05 · Status
Paused, with the next step written down.
Nothing is shipping and nothing is promised. If work resumes, the plan already exists: build the first transformer-class native body and check that it can answer plain English with no source in context, from a single checkpoint-bound package. Until then, this page is the record.