emkornfield commented on code in PR #3608:
URL: https://github.com/apache/parquet-java/pull/3608#discussion_r3755214618
##########
parquet-hadoop/src/main/java/org/apache/parquet/hadoop/ParquetFileWriter.java:
##########
@@ -609,6 +609,50 @@ public InternalFileEncryptor getEncryptor() {
return fileEncryptor;
}
+ /**
+ * Writes a {@code FILE} self-reference payload into the file body,
inheriting the compression and
+ * encryption of the {@code inline} column chunk, and returns the {@code
offset} and {@code size} a
+ * writer records in the self-reference's {@code offset} and {@code size}
fields. See
+ * {@link SelfReferenceStorage} for the layout and the Parquet format's
"FILE" logical type
+ * specification for the storage-inheritance semantics.
+ *
+ * <p>The payload is compressed as an independent compression block using
{@code compressor} (the
+ * compressor for the {@code inline} column chunk's {@link
CompressionCodecName}) and, when
+ * {@code pageBlockEncryptor} is non-null, encrypted as an independent
module with the
+ * {@code Self-Reference} module type. The row group ordinal is that of the
block currently being
+ * written.
+ *
+ * <p>Payloads are written while a block is open but before its column
chunks are flushed, so they
+ * land in a contiguous run ahead of the row group's chunks. This keeps each
column chunk
+ * contiguous on disk, which the read path relies on when coalescing
adjacent chunks into a single
+ * range read.
+ *
+ * <p>This must be called while a block is open (after {@link
#startBlock(long)} and before
+ * {@link #endBlock()}) so that the returned offset falls within the file
body.
+ *
+ * @param resolvedBytes the resolved (logical) bytes of the self-reference
+ * @param compressor the compressor for the {@code inline} column chunk's
codec
+ * @param pageBlockEncryptor the data-module encryptor of the {@code inline}
column chunk, or
+ * {@code null} if the column chunk is not encrypted
+ * @param columnOrdinal the ordinal of the {@code inline} column the
self-reference inherits from
+ * @return the offset and size of the stored representation
+ * @throws IOException if writing or compression fails
+ */
+ public SelfReferenceStorage.StoredRange writeSelfReference(
+ BytesInput resolvedBytes,
+ CodecFactory.BytesCompressor compressor,
Review Comment:
this seems like an awkward API, shold compressor and pageBlockEncryptor be
abstracted in some way (so all that is needed is column Ordinal)? or should all
three be grouped together?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
