wgtmac commented on code in PR #3750:
URL: https://github.com/apache/parquet-java/pull/3750#discussion_r4081074738
##########
parquet-protobuf/README.md:
##########
@@ -21,3 +21,45 @@ parquet-protobuf
================
Protocol Buffer support for Parquet columnar format.
+
+## Message fields stored as proto bytes
+
+Two kinds of message fields cannot be mapped to a Parquet group, so
`ProtoSchemaConverter`
+terminates them as the **serialized protobuf message** instead:
+
+* **Fields of an empty message type** (`message Stub {}`) — Parquet
forbids empty groups.
+ An empty message serializes to zero bytes, so the column is cheap, and field
presence still
+ round-trips: `null` means the field was unset, an empty value means it was
set.
+* **Recursive fields beyond `parquet.proto.maxRecursion`** (default 5) —
the remaining
+ sub-tree is stored as the serialized message instead of expanding the schema
forever.
+
+The Parquet type is an unannotated `BINARY` column that keeps the field's own
repetition (or,
+for repeated fields and map values, sits inside the standard `LIST` / `MAP`
wrappers when
+`parquet.proto.writeSpecsCompliant` is set):
+
+```
+message Trees.StubBox {
+ optional binary stub = 1; // Stub stub = 1;
+ optional group stubs (LIST) = 2 { // repeated Stub stubs = 2;
+ repeated group list {
+ required binary element;
+ }
+ }
+ optional group stub_map (MAP) = 3 { // map<string, Stub> stub_map =
3;
+ repeated group key_value {
+ required binary key (STRING);
+ optional binary value;
+ }
+ }
+}
+```
+
+Readers that do not know about protobuf simply see opaque bytes (all of them
empty for an
+empty message type). Readers that do can parse the bytes with the message's
descriptor, which
+the writer stores in the file footer under the `parquet.proto.descriptor` key;
+`ProtoParquetReader` does this automatically and materializes the original
message.
Review Comment:
nit: `ProtoParquetReader` seems to load the protobuf class from
`parquet.proto.class` (or the configured class), rather than parse
`parquet.proto.descriptor` from the footer. Could we clarify that here so it
doesn’t imply the footer descriptor alone is sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
