Faster prompt lookup drafting in llama.cpp

Faster prompt lookup drafting in llama.cpp jadidbourbaki.github.io

Four changes to the n-gram caches of llama.cpp make drafting up to 41.6x faster, load the static cache up to 23.5x faster, and lower peak memory up to 2.65x.

87 points•pptadversary•4 days ago•12 comments•

#faster #drafting #llama #cpp #prompt #lookup #faster prompt #prompt lookup

12 comments

jadidbourbaki4 days ago

Btw, if anyone has experience with the open source community in general and llama.cpp in specific, I would greatly appreciate some advice. I’m facing a bit of an interpersonal issue that I really hope is resolved without any ill will. Here is the context:

https://www.reddit.com/r/LocalLLaMA/comments/1wr5ylm/comment...

Any advice for what I can do? Due to this, I cannot create a PR or issue in the llama.cpp repository. However, I am worried about bothering the maintainers on other channels in case it aggravates them further. Thank you for your help!

embedding-shape3 days ago

> Any advice for what I can do?

Email that author (email can usually be found via GitHub, or contact via other private way they've shared somewhere) and explain the situation, don't lambast them publicly on social media or similar ways. If that doesn't work, do the same but for another maintainer. Don't spam all of them straight up, wait a week or something before contacting someone else.

netburst3 days ago

That would be inadvisable as the maintainer (?) in question's GitHub bio states:

> Emailing me about a temporary ban will result in a permanent one instead.

As far as I can tell the ban was given based on a single accidental ping to the maintainer, who then reviewed the draft PR before realizing it was not for the mainline llama.cpp repo.

electroglyph3 days ago

hopefully someone on the team sees this and fixes the situation, otherwise, if it's good, i'm sure somebody else will open the PR. thanks for your contribution.

wronglebowski3 days ago

I understand this repository must be under siege effectively with how popular it is, but the team needs to come up with a decent way to handle this. The current climate with Nvidia buying Hugging Face who effectively owns the llama.cpp team there’s a lot of discourse around their control of the team that’s negative.

rfgplk3 days ago

Just hard fork the project. Frankly, llama.cpp is so badly written that these kind of speedups are trivial, and a hard fork (or a total rewrite) has been needed for the longest time.

jadidbourbaki4 days ago

Fun update to this: Daniel Lemire added another optimization to make this even faster. https://github.com/jadidbourbaki/llama.cpp/pull/12

I’ll benchmark his change and add it to the article, crediting him for this improvement.

jadidbourbaki3 days ago

As an update to this, Lemire absolutely crushed it! I benchmarked his change today. Here is my PR comment:

https://github.com/jadidbourbaki/llama.cpp/pull/12#issuecomm...

S0y3 days ago

How much does this speedup inference for the end user in terms of tk/s ?

PicardsFlute3 days ago

So, instead of resolving it privately like you were asked, you come here and complain? Was the bot posts on Reddit not enough for you? Pro tip: There is a right way and a wrong way to approach these things and you are MOST certainly approaching this the wrong way. And before you go accusing me of anything: I am NOT associated with that project, but I DID see what happened. You are acting like a damned child and should be ashamed of yourself.

ricardobeat3 days ago

Is there some missing context here? Where was the author asked to resolve issues in private? And what 'private' channels would be available to them?

frazar03 days ago

I think they meant to reply to this comment

https://news.ycombinator.com/item?id=49859982#49863097

but commented on the post.

Read the full thread on Hacker News →

12 comments

Related stories