I agree with the points above and specifically for self-reference/in-file, the output stream is practically owned by the parquet writer. The writer would need to surface and coordinate writing to the target file, so should also be able to manage the compression. Uncompressed is always an option if there are concerns about efficiency, but the impact of not having compression for common cases (like large text blocks), would be much worse.
I'm a strong +1 on reviving this from the earlier proposal. -Dan On Wed, Jul 29, 2026 at 7:12 AM Alkis Evlogimenos via dev < [email protected]> wrote: > Hello, > > Now that FILE [1] is merged I'd like to reopen one point from the original > proposal that got dropped before merge: the bytes of a self-reference > should use the same CompressionCodec as the column's inline field. Removing > it was a mistake, and it's cheap to fix while FILE hasn't shipped in a > release. > > As merged, a self-reference can only be stored uncompressed. That's ok for > data like images and video, but it makes self-references useless for text > blobs (html, json, logs). Values too big to be inline yet small enough to > want as self-references are exactly where PLAIN blows up the storage cost. > A self-reference is the Parquet writer's decision to store a large value > out-of-band, so it should be compressed consistently with the inline values > it came from. > > To the objections from the PR: > > 1. Apply it uniformly to external refs too. > > The asymmetry is the point. An external s3://… can be referenced by many > files and systems, including ones that know nothing of Parquet; its > encoding is decided above Parquet, sometimes outside any engine. A > self-reference lives inside Parquet and is written by the Parquet writer. > Parquet owns those bytes, so Parquet compresses them. > > 2. Let the engine own it. > > The engine already owns the blob's own compression via content_type and can > pass the writer pre-compressed bytes. The inline codec is a different > thing: the storage compression Parquet applies to the column. A > self-reference is the same bytes spilled out-of-band, so it belongs to the > same storage and codec. > > 3. Force-compressing images wastes CPU. > > It doesn't. The writer picks the codec per column chunk (and per page in > v2), same as it already does for inline. Inheritance just propagates that > choice. A column chunk written uncompressed stays uncompressed. > > 4. Compaction would force decompress/recompress. > > Only an issue for external refs. Compaction rewrites the whole file, so > self-referenced bytes re-encode in the same pass, exactly like inline > values. > > Proposed wording: > > > The bytes referenced by a self-reference (a FILE with no uri) are > compressed with the CompressionCodec of the inline column chunk's > ColumnMetadata. This does not apply to external references. > > Cheers, > > [1] https://github.com/apache/parquet-format/pull/585 >
