Local decision model with calibrated probabilities: send a state and yes/no, choice or score questions, get a probability for every option. 0.8B GGUF on CPU, Jev-style API. - kouhxp/gutsy
1 comment
mrkn1about 4 hours ago
I made Gutsy to see what quality of inferencing I could get for a Jev style model on laptops and cheap VMs. The result is pretty good for triaging, relevance scoring and yes no type questions. I'd love to hear what latency others are getting. For me it was subsecond inference time for short queries.
Read the full thread on Hacker News →
Related stories
- Hacker News · 8 points · 2 days ago
- Hacker News · 2 points · 6 days ago
- Hacker News · 1 points · 12 days ago
- Hacker News · 22 points · 10 days ago
- Hacker News · 1 points · 11 days ago
- DEV Community · 10 points · 10 days ago