Rest of World · Ananya Bhattacharya ·

Study: non-English languages can consume far more AI tokens, with the most expensive language using 1.5x to 5x more than the cheapest, causing a "tokenizer tax"

Some languages require far more tokens for AI models to process, raising costs and reducing context for the people who use them.

Lead Source

Rest of World — Study: non-English languages can consume far more AI tokens, with the most expensive language using 1.5x to 5x more than the cheapest, causing a "tokenizer tax" ↗

How this story grew

Coverage · 0 Discussion · 0

Oct 27Oct 28

Discussion

23 points, 1 comments

Related stories

CNBC: Memo: Amazon says measuring AI adoption solely by token usage or code generation is "deeply flawed," as it discourages staff from tokenmaxxing

VentureBeat: Z.ai prices GLM-5.3 API access at $1.40/1M input tokens and $4.40/1M output tokens, unchanged from GLM-5.2; Kimi K3 costs $3/1M input and $15/1M output tokens

OpenAI: OpenAI releases GPT-6.1 Terra, touting "fast and capable computer use at scale" and clearer, more natural writing, priced at $1/1M input and $5/1M output tokens

Apollo Global Management: Outside of tech there has been no financial payoff to US companies for using AI, now viewed as a margin-hurting cost center, as VC-backed firms burn more tokens

The Deep View: GPT-6 Sol costs $2/1M input and $10/1M output tokens, and GPT-6 Luna costs $0.10/1M input and $0.50/1M output tokens, both about 50% cheaper than GPT-5.6

Discuss with AI

ChatGPT ↗ Claude ↗

← Back to TechSnif