You don't have a buffer overflow problem if you read it in a memory-safe way i.e. read it in chunks and realloc when you reach the size of your allocation.

What you will have is a potential denial-of-service attack - although this one isn't particularly great because there's zero amplification (they might as well just send garbage into your firewall)

In regular unicode, a grapheme can be made up of an arbitrary number of codepoints (and thus an arbitrary number of bytes), which does cause issues at times.

OTH UTF-8 is just one variable-length stream encoding among many others (RLE, LBE128, etc...).

Limit the codepoint to the number of atoms in the universe (less than 32 bytes).

I don’t think there’s harm is speccing out the arbitrary encoding and then having a different spec that references that spec but puts hard limits on it. Many rfcs are like that.

> […] why limit yourself to 24-bit numbers?

For compatibility with UTF-16:

    o  Restricted the range of characters to 0000-10FFFF (the UTF-16
       accessible range).

* https://datatracker.ietf.org/doc/html/rfc3629#section-12

* https://en.wikipedia.org/wiki/UTF-16

The original spec had 31 bits (the UTF-32/UCS-4 range):

* https://datatracker.ietf.org/doc/html/rfc2279

* https://en.wikipedia.org/wiki/UTF-32

> ...24-bit numbers?

Technically current UTF-8 only goes up to 21 bits (that's the current UNICODE range), for the encoding itself that is an arbitrary limit though, with the 'single lead byte' method of traditional UTF-8 it could go up to 36 bits "payload".

I think that wouldn't change much. They would just make use of more grapheme clusters.

For emojies they already make heavy use of the Zero-Width-Joiner. So a woman firefighter is the woman emoji + ZWJ + fire engine. Sure the UTF-8000 approach is much better encoding size wise.

I predict eventual convergence between UTF-whatever and most popular tokenizer for whatever LLM escapes to become world-ruling AGI.

I mean, if someone's seriously going to try encoding birdsong and dog barks, at this point they're basically reinventing tokens for multi-modal language models.

We still have about 85% of codepoint space unused. Hopefully, by the time it becomes a problem, UTF-16 will be long dead

Important to recognize that characters have individuality, that's why there can only be a limited number of them. Unicode is enumerating a finite set of things, not encoding an infinite set. Aenything without this property - any generic form of encoding - is not characters, it's something else like images. If it's not in any alphabet it shouldn't be in unicode, you should use an escape tag for image data instead. (Emojis probably shouldn't, but they do behave like an alphabet)

There cannot be 4 trillion characters because humans would need to know all of them and humans cannot know that many things.

Yeah that kind of line is what I see all the time in my chats. Even worse worse is when they put it in code comments.

It's not even true as being able to tell the character length from the first byte is not a property that extending UTF-8 past 36 bit payloads preserves.