Hello, Now that FILE [1] is merged I'd like to reopen one point from the original proposal that got dropped before merge: the bytes of a self-reference should use the same CompressionCodec as the column's inline field. Removing it was a mistake, and it's cheap to fix while FILE hasn't shipped in a release.
As merged, a self-reference can only be stored uncompressed. That's ok for data like images and video, but it makes self-references useless for text blobs (html, json, logs). Values too big to be inline yet small enough to want as self-references are exactly where PLAIN blows up the storage cost. A self-reference is the Parquet writer's decision to store a large value out-of-band, so it should be compressed consistently with the inline values it came from. To the objections from the PR: 1. Apply it uniformly to external refs too. The asymmetry is the point. An external s3://… can be referenced by many files and systems, including ones that know nothing of Parquet; its encoding is decided above Parquet, sometimes outside any engine. A self-reference lives inside Parquet and is written by the Parquet writer. Parquet owns those bytes, so Parquet compresses them. 2. Let the engine own it. The engine already owns the blob's own compression via content_type and can pass the writer pre-compressed bytes. The inline codec is a different thing: the storage compression Parquet applies to the column. A self-reference is the same bytes spilled out-of-band, so it belongs to the same storage and codec. 3. Force-compressing images wastes CPU. It doesn't. The writer picks the codec per column chunk (and per page in v2), same as it already does for inline. Inheritance just propagates that choice. A column chunk written uncompressed stays uncompressed. 4. Compaction would force decompress/recompress. Only an issue for external refs. Compaction rewrites the whole file, so self-referenced bytes re-encode in the same pass, exactly like inline values. Proposed wording: > The bytes referenced by a self-reference (a FILE with no uri) are compressed with the CompressionCodec of the inline column chunk's ColumnMetadata. This does not apply to external references. Cheers, [1] https://github.com/apache/parquet-format/pull/585
