smallpond

About this project

smallpond A lightweight data processing framework built on [DuckDB] and [3FS]. Features — 🚀 High-performance data processing powered by DuckDB — 🌍 Scalable to handle PB-scale datasets — 🛠️ Easy operations with no long-running services Installation Python 3.8 to 3.12 is supported. Quick Start Documentation For detailed guides and API reference: — Getting Started — API Reference Performance We evaluated smallpond using the [GraySort benchmark] ([script]) on a cluster comprising 50 compute nodes and 25 storage nodes running [3FS]. The benchmark sorted 110.5TiB of data in 30 minutes and 14 seconds, achieving an average throughput of 3.66TiB/min. Details can be…

GitHub

Reviews

Built by

deepseek-ai

Imported from GitHub · not yet claimed on GitPalace

View developer pageSign in with GitHub to claim

Maintain deepseek-ai/smallpond? Claiming verifies admin access through your GitHub account and gives you control of this listing.

You might also like

build-your-own-x

Developer Tools

Master programming by recreating your favorite technologies from scratch.

★ New545.6K stars

by codecrafters-io

awesome

Developer Tools

😎 Awesome lists about all kinds of interesting topics [NOTE: Pull requests are temporarily disabled until I have a chan

★ New503.7K stars

by sindresorhus

public-apis

Developer Tools

A collective list of free APIs

★ New476.5K stars

by public-apis

free-programming-books

Developer Tools

:books: Freely available programming books

★ New396.1K stars

by EbookFoundation