Tokenization: A Survey for Modern NLP | Hacker News - But Better

Tokenization: A Survey for Modern NLP alphaxiv.org

While modern language models take raw text as their input and produce raw text as output, they do not operate over text directly. Hidden in the very first step of language model pipelines is...

4 points•mcyc•about 10 hours ago•0 comments•

#modern #tokenization #nlp #survey #text #modern nlp #raw #raw text

0 comments

No comments yet.

Related stories

Show HN: Text alerts when Codex resets are announcedresetalerts.com
Hacker News · 3 points · 1 day ago
Lexxy: A modern rich text editor for Railsjorgemanrubia.com
Hacker News · 1 points · 2 days ago
Nimble takes some text and a schema, and makes typed decisions about the textgithub.com
Hacker News · 1 points · 10 days ago
Show HN: Diff Pro Max – File Compare Apprkpg.net
Hacker News · 1 points · 7 days ago
Sifty, free text classification with one URL and no API keysifty.dev
Hacker News · 1 points · 10 days ago
Show HN: Morse Code Translator – real-time text-to-Morse with audiomorsetranslator.me
Hacker News · 2 points · 11 days ago