Russell, I'm not sure I follow your points here. With inline, the data is compressed using the column compression defined in the writer. This is equivalent to any data written in parquet and compressed as a page, so I don't see the issue there.
The self-ref/in-file offset+size would represent the compressed size (when compressed). Having the uncompressed size is nice, but technically not guaranteed to be accurate because uncompressed sizes rely heavily on the memory layout which can vary by language/implementation. I think this approach largely aligns with how parquet handles data managed by the writer. -Dan On Wed, Jul 29, 2026 at 9:37 AM Russell Spitzer <[email protected]> wrote: > I'm a little worried about adding this in, > > Self-references are already specified as [offset, offset+size) ranges: the > resolved bytes are that range, and no compression transform is defined. > Adding compression isn't a one-line codec inheritance rule. If we reuse > Parquet's CompressionCodec model, readers also need framing (at least > uncompressed length). "The inline column chunk's CompressionCodec" is > underspecified too: a FILE group may omit inline, codecs are per column > chunk / row group, and v2 can leave data uncompressed under the same chunk > codec via is_compressed. > > Encryption is a good example of the same gap. The merged text says self-ref > files must not use modular encryption, but we never really worked through > how those ranges would interact with features that need page/module > structure. We shouldn't bolt compression onto the same ranges without > defining how compressed self-ref bytes are laid out. > > IMHO, we should first strictly define how these out-of-band bytes are laid > out and framed, then we can make more concrete decisions about inheriting > codecs, encryption, and so on. > > On Wed, Jul 29, 2026 at 9:11 AM Alkis Evlogimenos via dev < > [email protected]> wrote: > > > Hello, > > > > Now that FILE [1] is merged I'd like to reopen one point from the > original > > proposal that got dropped before merge: the bytes of a self-reference > > should use the same CompressionCodec as the column's inline field. > Removing > > it was a mistake, and it's cheap to fix while FILE hasn't shipped in a > > release. > > > > As merged, a self-reference can only be stored uncompressed. That's ok > for > > data like images and video, but it makes self-references useless for text > > blobs (html, json, logs). Values too big to be inline yet small enough to > > want as self-references are exactly where PLAIN blows up the storage > cost. > > A self-reference is the Parquet writer's decision to store a large value > > out-of-band, so it should be compressed consistently with the inline > values > > it came from. > > > > To the objections from the PR: > > > > 1. Apply it uniformly to external refs too. > > > > The asymmetry is the point. An external s3://… can be referenced by many > > files and systems, including ones that know nothing of Parquet; its > > encoding is decided above Parquet, sometimes outside any engine. A > > self-reference lives inside Parquet and is written by the Parquet writer. > > Parquet owns those bytes, so Parquet compresses them. > > > > 2. Let the engine own it. > > > > The engine already owns the blob's own compression via content_type and > can > > pass the writer pre-compressed bytes. The inline codec is a different > > thing: the storage compression Parquet applies to the column. A > > self-reference is the same bytes spilled out-of-band, so it belongs to > the > > same storage and codec. > > > > 3. Force-compressing images wastes CPU. > > > > It doesn't. The writer picks the codec per column chunk (and per page in > > v2), same as it already does for inline. Inheritance just propagates that > > choice. A column chunk written uncompressed stays uncompressed. > > > > 4. Compaction would force decompress/recompress. > > > > Only an issue for external refs. Compaction rewrites the whole file, so > > self-referenced bytes re-encode in the same pass, exactly like inline > > values. > > > > Proposed wording: > > > > > The bytes referenced by a self-reference (a FILE with no uri) are > > compressed with the CompressionCodec of the inline column chunk's > > ColumnMetadata. This does not apply to external references. > > > > Cheers, > > > > [1] https://github.com/apache/parquet-format/pull/585 > > >
