Generate fonts where every LLM token is the same width

Generate fonts where every LLM token is the same width ampdot.mesh.host

94 points•z-mach9•5 days ago•23 comments•

#every #token #llm #fonts #generate #width #every llm #generate fonts

23 comments

I typed some German and it was breaking up words so much more than English. Not really surprising given tokenizers are optimized for most commonly used text.

Here's the token efficiency of a corpus translated into various languages and tokenized with the latest OpenAI one:

  Language              Relative tokens
  --------------------------------------
  English                    1.00x
  Portuguese                 1.23x
  Chinese (Simplified)       1.25x
  German                     1.31x
  Spanish                    1.32x
  French                     1.37x
  Arabic                     1.38x
  Chinese (Traditional)      1.42x
  Korean                     1.47x
  Swahili                    1.49x
  Hindi                      1.57x
  Japanese                   1.66x
  Burmese                    3.16x
  Amharic                    5.78x
  Santali                   13.70x

Source: "Tokenizer Fairness in 2026", a reproduction/extension of Petrov, La Malfa, Torr & Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS 2023), using FLORES-200.

https://github.com/partyfly/tokenizer-fairness-2026

dTal4 days ago

This page reliably hangs Firefox 155 at 100% CPU for me.

bbor4 days ago

Yeah this site has a severe issue of some kind that I don't think I've ever seen before. NGL, I'm pretty impressed! It takes so much compute that scrolling lags, which feels very, very strange

devindotcom4 days ago

smooth as butter on 152 in windows 10, as long as we're all reporting

ampdot4 days ago

I did most of my testing in Firefox 156 and it's fine for me

totetsu4 days ago

This made me realize that my usual practice of typing at 80% accuracy, full of mistakes, when sending input to an llm is probably increasing my input token count.

synthos3 days ago

Maybe spell checking for prompt text box would save a few bucks

williamdclt3 days ago

I'm always wondering if typos and grammar mistakes impact significantly the quality of the response. After all LLMs are next-token predictors, and I suspect that in the training set (internet), bad writing is correlated to low-quality content?

creativeSlumber3 days ago

I was wondering about this one too.To make things worse I also use speech to text and that introduces its own inaccurate transcriptions and typos. Are there any reliable research around this ?

LoganDark4 days ago

The kerning is absolutely awful in Safari. Looks fine in Chrome though...

Applejinx4 days ago

I typed the first stanza of 'Jabberwocky'.

Worked about as well as expected :D

Read the full thread on Hacker News →

23 comments

Related stories