After seeing the claim, I knew it would work because of a few intuitive things:

1. Things that are the result of a very large number of small perturbations follow a Gaussian distribution.

1a. Word embeddings are the result of millions/billions/trillions of small perturbations (i.e. gradient descent)

2. As you increase the dimension, d, of a multivariate Gaussian distribution, the actual mass of the data is increasingly concentrated in the thin spherical shell. For MVN distributions with standardized marginal dimensions, this is on the sphere of radius \sqrt{d}.

In fact when I searched "Gaussian curse of dimensionality" I got this at the top result: [1] which shows this phenomena perfectly. Or this one with pretty interactive pictures [2]

[1]: https://www.miryusupov.com/blog/posts/thin-shells/index.html

[2]: https://aseemrb.me/blog/high-dimensional-gaussians-on-a-sphe...

Does the mapping depend on the chose set, or could you just map all 25k words and pick 10 points that form a circle? From the process, they seemingly replaced single words in the set that didn't fit, which makes it sound like the position of the remaining words didn't change much from changing the set.

I recently did some analysis on how different style prompts like "be succinct" and "avoid mannered prose" affect LLM outputs and I also saw a ~circular relationship with a 2d dimensional reduction (MDS) of the similarity matrix. I wonder if it's some artifact of PCA/MDS + Gemma's output dist

The best bar bets are the ones where you're technically not cheating, just abusing linear algebra.

Using high-dimensional statistics for beer money, not academic papers" is the correct use of the degree. The papers are where you prove the circle exists. The bar is where you find out it exists for everything