rok commented on code in PR #603:
URL: https://github.com/apache/parquet-format/pull/603#discussion_r3692639460
##########
Encryption.md:
##########
@@ -270,43 +272,54 @@ The following module types are defined:
* OffsetIndex (7)
* BloomFilter Header (8)
* BloomFilter Bitset (9)
+ * Self-Reference (10)
-| | Internal File ID | Module type | Row group ordinal |
Column ordinal | Page ordinal|
-|----------------------|------------------|-------------|-------------------|----------------|-------------|
-| Footer | yes | yes (0) | no |
no | no |
-| ColumnMetaData | yes | yes (1) | yes |
yes | no |
-| Data Page | yes | yes (2) | yes |
yes | yes |
-| Dictionary Page | yes | yes (3) | yes |
yes | no |
-| Data Page Header | yes | yes (4) | yes |
yes | yes |
-| Dictionary Page Header| yes | yes (5) | yes |
yes | no |
-| ColumnIndex | yes | yes (6) | yes |
yes | no |
-| OffsetIndex | yes | yes (7) | yes |
yes | no |
-| BloomFilter Header | yes | yes (8) | yes |
yes | no |
-| BloomFilter Bitset | yes | yes (9) | yes |
yes | no |
+| | Internal File ID | Module type | Row group ordinal |
Column ordinal | Page ordinal | Self-reference ordinal |
+|-----------------------|------------------|-------------|-------------------|----------------|--------------|------------------------|
+| Footer | yes | yes (0) | no |
no | no | no |
+| ColumnMetaData | yes | yes (1) | yes |
yes | no | no |
+| Data Page | yes | yes (2) | yes |
yes | yes | no |
+| Dictionary Page | yes | yes (3) | yes |
yes | no | no |
+| Data Page Header | yes | yes (4) | yes |
yes | yes | no |
+| Dictionary Page Header| yes | yes (5) | yes |
yes | no | no |
+| ColumnIndex | yes | yes (6) | yes |
yes | no | no |
+| OffsetIndex | yes | yes (7) | yes |
yes | no | no |
+| BloomFilter Header | yes | yes (8) | yes |
yes | no | no |
+| BloomFilter Bitset | yes | yes (9) | yes |
yes | no | no |
+| Self-Reference | yes | yes (10) | yes |
yes | no | yes |
+
+For a self-reference, the column ordinal is that of the `inline` column. The
+self-reference ordinal is the zero-based position representing the same `FILE`
value
+in that column chunk's repetition and definition level stream, including null
+positions. It is derived by the reader and is not stored separately.
## 5. File Format
### 5.1 Encrypted module serialization
-All modules, except column pages, are encrypted with the GCM cipher. In the
AES_GCM_V1 algorithm,
-the column pages are also encrypted with AES GCM. For each module, the GCM
encryption
+All modules, except column pages and self-references, are encrypted with the
GCM cipher. In the
+AES_GCM_V1 algorithm, column pages and self-references are also encrypted with
AES GCM. For each module, the GCM encryption
buffer is comprised of a nonce, ciphertext and tag, described in the
Algorithms section. The length of
the encryption buffer (a 4-byte little endian) is written to the output
stream, followed by the buffer itself.
|length (4 bytes) | nonce (12 bytes) | ciphertext (length-28 bytes) | tag (16
bytes) |
|-----------------|------------------|------------------------------|----------------|
-In the AES_GCM_CTR_V1 algorithm, the column pages are encrypted with AES CTR.
-For each page, the CTR encryption buffer is comprised of a nonce and
ciphertext,
+In the AES_GCM_CTR_V1 algorithm, column pages and self-references are
encrypted with AES CTR.
+For each page or self-reference, the CTR encryption buffer is comprised of a
nonce and ciphertext,
described in the Algorithms section. The length of the encryption buffer
(a 4-byte little endian) is written to the output stream, followed by the
buffer itself.
|length (4 bytes) | nonce (12 bytes) | ciphertext (length-12 bytes) |
|-----------------|------------------|------------------------------|
+For a self-reference, `offset` points to the beginning of the 4-byte length
and `size`
+is the size of the complete encrypted module, including the length, nonce,
ciphertext,
+and GCM tag when present.
Review Comment:
4-byte length limits us to 2GiB payload. Are we expecting use cases with
greater size requirements?
##########
LogicalTypes.md:
##########
@@ -756,10 +756,44 @@ A self-reference points within the same Parquet file
using `offset` and `size` (
required). A self-reference is when `uri` is not set. A file containing
self-references
can be renamed or relocated as a single unit.
-Parquet files containing self-references must not use Parquet modular
encryption.
-Self-referenced byte ranges are not Parquet encryption modules and therefore
cannot
-be encrypted or authenticated independently. Encryption of external files
referenced
-by `uri` is outside the scope of the Parquet format.
+A schema that permits self-references must include the `inline` field.
+
+Each self-reference inherits the compression and encryption settings of the
`inline`
+column chunk in the same row group. The corresponding position is the position
+representing the same `FILE` value in the `inline` column's repetition and
definition
+level stream.
+
+Each referenced byte range is compressed independently using the
`CompressionCodec`
+of the `inline` column chunk. `UNCOMPRESSED` leaves the referenced bytes
uncompressed.
+
+Each compressed byte range is an independent compression block. Compression
state is
+not shared with the data page or with other referenced ranges.
+
+For an unencrypted self-reference, `offset` and `size` identify either the
independent
+compressed block or the uncompressed bytes. For a compressed block, the
complete range
+is supplied to the codec, and its decompressed output is the resolved value.
Review Comment:
Should we also store the decompressed size somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
