Benchmarks are more broken than we could have imagined

Benchmarks are more broken than we could have imagined horizonanalyticslabs.com

We audited public AI benchmark datasets from well-known vendors and found leaked answers, incomplete tests, gameable graders, and tasks their own solutions could not pass.

4 points•chillacy•9 days ago•1 comment•

#imagined #broken #benchmarks #audited #ai #vendors #public #datasets

1 comment

PartialSolution9 days ago

It's kind of ridiculous that datasets from these outfits are leaking fixes through git history

Read the full thread on Hacker News →

1 comment

Related stories