HuskHoard uses Zstd frames to cut Parquet egress costs by a claimed 99.9%

A new open-source tool, HuskHoard, proposes a storage-level architecture using seekable Zstd frames and jump tables to dramatically reduce cloud egress costs for partial Parquet file reads.

The Answer Up Front

HuskHoard is for data engineering teams designing cold storage archives where cloud egress costs for partial reads are a primary design constraint. If you frequently run schema discovery or single-column scans on petabyte-scale Parquet datasets in S3 or GCS, its architecture is worth studying. Teams already using modern query engines like DuckDB, which perform partial remote reads at the application layer, should wait for comparative benchmarks. The bottom line: HuskHoard proposes an elegant, storage-level fix for a costly problem, but its claimed 99.9% cost reduction is a compelling calculation, not yet a measured result from a production workload.

Methodology

This is a v0 review based exclusively on the author's technical blog post and the associated GitHub repository for HuskHoard, observed on June 24, 2026. The source material is available at https://dev.to/coldstorage/using-zstd-frames-to-egress-partial-parquet-files-1fdd. This analysis covers the tool's proposed architecture, its core concepts (Zstd frames, jump tables), and the author's cost-saving calculations. It does not include independent performance benchmarks, a hands-on usability assessment, or a direct comparison against alternative tools that also perform partial reads, such as DuckDB's HTTPFS. The cost-saving figures presented here are claims made by the author based on public cloud pricing and have not been independently verified. Update cadence: this review will be updated to a v1 once we can perform independent benchmarks.

What It Does

The problem: paying to download bytes you don't need

Many data operations only require a small fraction of a large Parquet file. Reading a 50 KB footer to discover a schema, for example, might trigger a full download of a 4 GB file from cloud storage. At S3's standard egress rate of $0.09/GB, these reads become prohibitively expensive at scale. The author provides a calculation for a data catalog sync across 100 TB of data, where fetching full files would cost $9,216. In contrast, fetching only the required footers would cost just $9.00. The challenge is telling the storage system which specific bytes to send.

Zstd frames for seekable archives

HuskHoard's solution begins with how it compresses data. Instead of creating a single, monolithic compressed file, it concatenates multiple, independent Zstd frames. Each frame is a self-contained compressed unit. If you know the compressed size of each preceding frame, you can calculate the exact byte offset to the beginning of any specific frame. This allows a client to use an HTTP Range request to fetch just one frame from a massive file without downloading the entire object.

Jump tables as a file index

To make frames seekable, HuskHoard creates a

The investor read

HuskHoard targets a single, painful line item in the modern data stack: cloud egress costs. This signals a market mature enough for second-order cost optimization to become a primary feature. The approach is sound, but its strategic moat is questionable. The direct competitors are not other archival tools, but rather features within query engines like DuckDB and Polars that are rapidly improving their ability to perform intelligent partial reads on standard remote files. For HuskHoard to be investable, it must demonstrate a 10x advantage in cost, speed, or simplicity over these increasingly capable, open-source incumbents. Without that, it risks being a clever feature, not a standalone company. A potential path to commercialization could be a managed service that transparently applies this optimization, but the underlying technology first needs to prove its superiority in open benchmarks.