Select * the internet: DuckDB queries 100B rows in 30s

Select * the internet: DuckDB queries 100B rows in 30s motherduck.com

Common Crawl publishes petabytes of web crawl data on S3. With DuckDB and MotherDuck you can query the Common Crawl dataset directly, no cluster and no download, and measure how fast the vibe-coded web is growing. |…

4 points•dmkii•3 days ago•1 comment•

#duckdb #select #queries #rows #crawl #100b #internet #30s

1 comment

dmkii3 days ago

You can basically SELECT * the internet by querying the public Common Crawl dataset on S3 with DuckDB. It's a great way to explore how the AI-pilled web (Vercel, Lovable, Replit, Bolt, Cloudflare, GitHub) compare to the traditional Wordpress-Blogspot hegemony.

Read the full thread on Hacker News →

1 comment

Related stories