ttok 1.0
- ID
- 33510
- Status
- summarized
- Published
- 09 Oct 2026, 8:34 AM
- Fetched
- 09 Oct 2026, 12:53 PM
- Provider
- Simon Willison
- Category
- developer-ai
- Original URL
- https://simonwillison.net/2026/Oct/9/ttok/
- Source URL
- https://simonwillison.net/atom/everything/
Summary
- Score
- 5.5
- Created
- 09 Oct 2026, 12:53 PM
- Tags
- Audience
- developersai_ml_learnersai_agent_users
What happened
Simon Willison shipped ttok 1.0, a CLI that counts and truncates text by tokens. The trigger was upgrading to ttok 0.4 and noticing it still defaulted to the GPT-4 tokenizer, so 1.0 switches the default to the GPT-5/GPT-6 family. OpenAI has not confirmed GPT-6 shares the GPT-5 tokenizer, but the post cites a commit by William Liu documenting an experiment where seven GPT models (5.5, 5.6 Sol/Terra/Luna, 6 Astra/Sol/Luna) all reported 44,794 tokens and matched each other on all 31 fixtures.
Why it matters
If any script, cost estimate, or context-budget check of yours calls ttok, upgrading to 1.0 silently changes the numbers because the default tokenizer moved off GPT-4. The 44,794-token, 31-fixture result is evidence that GPT-5 and GPT-6 count identically, not vendor confirmation, so treat cross-model token parity as an assumption to re-test rather than a fact to build billing on.
Discussion angle
Do you pin a tokenizer version in your cost or context-window logic, or take whatever the tool defaults to? Walk through what a silent default change like this does to an agent's truncation and per-request cost estimates.