The same request sent to artificial intelligence can cost considerably more depending on the language used. A study by the Coalition for Language Equity in AI Research, published in September 2026 and reported by Rest of World, found differences in token consumption ranging from 1.5 to 5 times across the twenty most widely used languages on the Internet. This technical difference creates economic inequality that particularly affects languages and writing systems underrepresented in digital data.

Imagine two companies offering exactly the same service: an assistant that answers customer questions. One operates mainly in English, the other in an African or Asian language. With comparable features, the second company may have to purchase more units of linguistic computation to convey the same content. Neither its customers nor its developers necessarily asked for a more complex service. Their language is simply less well supported by the mechanism that prepares text for AI.

This difference has a name: the tokenization tax, or tokenizer tax. And it raises a seldom-asked question: why should the architecture of a tool described as universal give certain languages an economic advantage?

Why does the same sentence use more tokens in some languages?

A language model does not receive words exactly as we read them. Before processing, a program called a tokenizer splits text into smaller units: tokens. These units may be whole words, parts of words, or fragments of characters.

A common method called byte-pair encoding, or BPE, learns to progressively merge frequent sequences. When a combination appears often in the texts used to build the vocabulary, it is more likely to become a single token. Less frequent combinations remain split into pieces.

The problem arises when the data used to build this vocabulary heavily favors English. Common English structures then benefit from efficient segmentation. Other languages, especially those using underrepresented scripts, may need many more fragments to express a comparable idea.

An English sentence and its translation are not equal to the tokenizer

Rest of World demonstrates this using an OpenAI tokenizer. The English sentence “People should benefit from AI, regardless of the language they speak.” is segmented into relatively recognizable units. Its translation into Malayalam, a language written in an Indian script, is split much more finely, sometimes even within characters represented by several bytes.

This does not reflect a difference in expressive richness, nor does it prove that a language is naturally more complicated. It is a consequence of how digital representations were constructed.

The phenomenon is not limited to languages with few speakers. According to the summary published by explainx.ai, Burmese, Tibetan, Tifinagh, Khmer, Lao and Ol Chiki scripts are among those most heavily penalized in the reported comparisons. By contrast, Asturian, despite its limited digital presence, can benefit from the characteristics of its Latin script.

In other words, the size of a language community alone cannot predict its bill. The written form of the language and the tokenizer’s design choices also matter.

How much does this language inequality really cost?

The September 2026 study reported by Rest of World found a 1.5- to 5-fold difference in token counts between the most and least favored of the twenty most widely used languages on the Internet. Expanding the comparison to two hundred languages, researchers reported a gap of up to approximately 17 times for one model.

That last result requires caution, however: available accounts do not provide enough detail about the model and procedure used, and the precise wording of the gap differs between reports. It would therefore be misleading to claim that every request in Yoruba, Swahili or Amharic systematically costs 17 times as much as in English.

To understand the economic impact, consider a purely illustrative calculation. If a request takes 100 tokens in one language and 300 in another, the second consumes three times as many input units. At the same price per token, its input cost is therefore tripled. This is not a measurement of Swahili, Hindi or any other specific language: it merely illustrates how billing works.

Advertising

Subscription-based chat interfaces do not necessarily pass this difference directly on to individual users’ bills. For a company using an API billed by token volume, however, the effect can appear in operating costs.

This imbalance is not entirely new. In May 2023, Ahia and colleagues analyzed the cost and utility of OpenAI’s API across 22 typologically diverse languages. Their work already showed that many speakers faced higher costs while receiving worse performance, often in regions where AI services were less affordable.

This is where the real injustice lies: paying more does not guarantee getting more. In some linguistic contexts, the reverse is true.

Why does this tax also reduce memory and response quality?

Tokens are used not only for billing but also to measure the context window: the amount of information a model can consider at a given moment.

If a language requires more tokens to convey the same content, the available window fills up faster. That leaves less room for documents, detailed instructions or previous exchanges.

In an educational assistant, for example, retaining a student’s past mistakes helps adapt later explanations. Pratyush Choudhury, co-founder of Activate AI, specifically highlights the importance of this memory in how a digital tutor works. If conversation history consumes context more quickly in a given language, educational continuity may suffer.

The penalty also affects speed. Because models generate responses token by token, an answer requiring more tokens may take longer to produce. This added latency is particularly problematic where computing resources and connectivity are limited.

Two phenomena must nevertheless be distinguished. Poor segmentation mechanically reduces the context space available for equivalent content. Poor language quality can also result from limited training data or imperfect model adaptation. Tokenization alone therefore does not explain every bad answer.

What do African and Asian languages reveal about AI’s limitations?

Swahili, Yoruba, Amharic and Hindi illustrate the diversity of languages for which generalizing from English results would be risky. Their scripts, linguistic structures and digital representation are not identical. No universal multiplier can calculate their relative cost without knowing the tokenizer being used.

A benchmark published on GitHub by bedr-ux compares Amharic and English using 1,012 parallel sentences from FLORES-200 and nine different tokenizers. It examines tokens per word, characters per token and tokens relative to UTF-8 bytes. This approach demonstrates the value of testing tools on comparable content rather than reasoning solely from speaker counts.

In South Africa, the founder of ManyHands, a company developing among other things an assistant accessible on WhatsApp, points to the difficulty of making an agent converse effectively in the country’s eleven official languages. The issue goes beyond merely translating an interface: it concerns the ability of language infrastructure to serve different populations.

In my view, this is where the issue becomes political in the economic sense. An underrepresented language is not merely less visible in AI-generated content. It may also have less conversational space and impose higher costs on services seeking to support it.

We often talk about democratizing AI through accessible interfaces. But a translated interface is not enough if the invisible cost of running it continues to disadvantage the same populations.

How can you use fewer tokens without giving up your language?

It would be absurd to routinely ask users to abandon their own language to work around a technical flaw. Several measures can nevertheless improve the situation today.

  • Write precise, concise requests: remove repetition, avoid needlessly copying context and ask for a response format suited to the task. These changes can reduce the amount processed, although their exact effectiveness depends on the tokenizer.
  • Compare tokenizers before choosing a model: a multilingual company should test the same content in the languages its customers actually use and measure both input and output tokens.
  • Limit unnecessary conversation history: retaining essential information in summaries can free up context space.
  • Evaluate open models: open-weight models, or open-weight systems, allow greater customization and control. They can provide a cost-effective alternative, provided the necessary skills and infrastructure are available.
  • Test response quality in each language: an efficient tokenizer does not automatically guarantee a reliable model. Understanding, accuracy and usefulness must also be checked.

Perhaps the most important advice for a company is never to size its multilingual budget using only tests conducted in English. Measuring actual costs in the target languages helps avoid unpleasant surprises once the service is used at scale.

Can multilingual and locally deployed models fix the problem?

Tokenizers better adapted to multiple scripts are a logical avenue: a vocabulary built with more balanced language representation may reduce excessive fragmentation. But this effect must be verified experimentally. Merely describing a model as multilingual does not prove that all languages receive fair segmentation.

Producing better-quality language data offers another route. In India, the RESPIN initiative, led by the Indian Institute of Science with funding from the Gates Foundation, collected texts and read speech from native speakers covering 38 dialects. Such work can help better represent neglected language use.

Open models, potentially deployed locally, also provide opportunities for adaptation. Daniel Björkegren of Columbia University notes that cost is a major obstacle for businesses and governments in low-income countries. The ability to customize a model can therefore be important, even when it requires technical expertise.

It would nevertheless be an overstatement to present local deployment as an automatic solution: a model running on your own machines retains its tokenizer’s limitations and requires resources to operate.

A lasting solution cannot consist solely of optimizing users’ prompts. Model developers must also take underrepresented languages seriously when creating datasets, vocabularies and evaluations. A truly global AI should not ask disadvantaged populations to pay for the flaws in its linguistic architecture.

Frequently asked questions

Why is AI more expensive in some languages?

Models use tokens to process and bill for text. A tokenizer can break a sentence into far more fragments in a language poorly represented in its vocabulary. For APIs billed per token, this difference directly increases the cost of equivalent content.

Do Swahili, Amharic and Hindi always cost more than English?

There is no universal multiplier for these languages. Token counts depend on the model, tokenizer, script and text being analyzed. A reliable comparison must use equivalent sentences and the tools actually deployed.

Why does poor tokenization reduce an AI assistant’s memory?

The context window has a limited capacity measured in tokens. When the same document requires more tokens, it occupies more of that capacity. The assistant can then retain less history or handle fewer documents simultaneously.

How can a company reduce multilingual AI costs?

It can compare several models on real content, measure input and output tokens separately, shorten redundant instructions and avoid sending unnecessary history. It can also explore open models and customizable solutions. The right choice must account for cost, language quality and technical constraints.

Sources