I'm a little worried about adding this in, Self-references are already specified as [offset, offset+size) ranges: the resolved bytes are that range, and no compression transform is defined. Adding compression isn't a one-line codec inheritance rule. If we reuse Parquet's CompressionCodec model, readers also need framing (at least uncompressed length). "The inline column chunk's CompressionCodec" is underspecified too: a FILE group may omit inline, codecs are per column chunk / row group, and v2 can leave data uncompressed under the same chunk codec via is_compressed.
Encryption is a good example of the same gap. The merged text says self-ref files must not use modular encryption, but we never really worked through how those ranges would interact with features that need page/module structure. We shouldn't bolt compression onto the same ranges without defining how compressed self-ref bytes are laid out. IMHO, we should first strictly define how these out-of-band bytes are laid out and framed, then we can make more concrete decisions about inheriting codecs, encryption, and so on. On Wed, Jul 29, 2026 at 9:11 AM Alkis Evlogimenos via dev < [email protected]> wrote: > Hello, > > Now that FILE [1] is merged I'd like to reopen one point from the original > proposal that got dropped before merge: the bytes of a self-reference > should use the same CompressionCodec as the column's inline field. Removing > it was a mistake, and it's cheap to fix while FILE hasn't shipped in a > release. > > As merged, a self-reference can only be stored uncompressed. That's ok for > data like images and video, but it makes self-references useless for text > blobs (html, json, logs). Values too big to be inline yet small enough to > want as self-references are exactly where PLAIN blows up the storage cost. > A self-reference is the Parquet writer's decision to store a large value > out-of-band, so it should be compressed consistently with the inline values > it came from. > > To the objections from the PR: > > 1. Apply it uniformly to external refs too. > > The asymmetry is the point. An external s3://… can be referenced by many > files and systems, including ones that know nothing of Parquet; its > encoding is decided above Parquet, sometimes outside any engine. A > self-reference lives inside Parquet and is written by the Parquet writer. > Parquet owns those bytes, so Parquet compresses them. > > 2. Let the engine own it. > > The engine already owns the blob's own compression via content_type and can > pass the writer pre-compressed bytes. The inline codec is a different > thing: the storage compression Parquet applies to the column. A > self-reference is the same bytes spilled out-of-band, so it belongs to the > same storage and codec. > > 3. Force-compressing images wastes CPU. > > It doesn't. The writer picks the codec per column chunk (and per page in > v2), same as it already does for inline. Inheritance just propagates that > choice. A column chunk written uncompressed stays uncompressed. > > 4. Compaction would force decompress/recompress. > > Only an issue for external refs. Compaction rewrites the whole file, so > self-referenced bytes re-encode in the same pass, exactly like inline > values. > > Proposed wording: > > > The bytes referenced by a self-reference (a FILE with no uri) are > compressed with the CompressionCodec of the inline column chunk's > ColumnMetadata. This does not apply to external references. > > Cheers, > > [1] https://github.com/apache/parquet-format/pull/585 >
