This is an automated email from the ASF dual-hosted git repository.
wgtmac pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/parquet-java.git
The following commit(s) were added to refs/heads/master by this push:
new c3def131d GH-2142: Write fields of empty message types as proto bytes
(#3750)
c3def131d is described below
commit c3def131da9f09d8bcf3968cf076f410fdcf5309
Author: Peter Puškár <[email protected]>
AuthorDate: Wed Sep 23 17:33:48 2026 +0200
GH-2142: Write fields of empty message types as proto bytes (#3750)
### Rationale for this change
Protobuf allows empty message definitions, but Parquet forbids empty
groups. Converting a message
that merely *contains* a field of an empty message type produces a schema
with an empty group,
which writer construction rejects with `InvalidSchemaException: Cannot
write a schema with an
empty group`. Such fields appear in real-world schemas (deprecated stubs,
marker/placeholder
messages), and a single one makes the whole message type unwritable.
### What changes are included in this PR?
`ProtoSchemaConverter.addMessageField` terminates a field whose message
type has no fields as a
`BINARY` column holding the serialized message — the same mechanism
PARQUET-1711 uses for
recursion beyond `maxRecursion`. Since an empty message serializes to zero
bytes, the column is
cheap, and field **presence** still round-trips (null = unset vs empty
bytes = set):
- singular field → `optional binary stub` (or `required`, per the field);
- repeated field, parquet-specs mode → LIST-wrapped binary via the existing
`addRepeatedPrimitive`, so element cardinality survives;
- repeated field, old style → `repeated binary`;
- map value type → `optional binary value` inside the `key_value` group
(keys stay typed).
`ProtoWriteSupport.createMessageWriter`'s existing truncated-field check
(primitive BINARY where a
message field was declared → `BinaryWriter`) is generalized to look through
the LIST/MAP wrapper
(`getGroupType` → `getContentType`), so the writer tree lines up with these
schemas.
A message that is empty at the **root** is still rejected — there is no
parent field to hold the
bytes, and a Parquet file with zero columns is not representable.
**Read side** (second commit, after review): `ProtoMessageConverter` used
to cast every message
field's Parquet type to a group, so a message field stored as `BINARY`
failed with a
`ClassCastException` while the converter tree was built — which also means
files with
PARQUET-1711 recursion truncation have been unreadable by
`ProtoParquetReader` since 1.13.0 (#995
shipped with an explicit "TODO: ReadSupport"). A new
`ProtoBinaryMessageConverter` parses the
bytes back through `parentBuilder.newBuilderForField(field)` and hands the
message to the existing
parent container, so singular fields, LIST elements, old-style repeated
fields and map values all
round-trip without touching `ListConverter`/`MapConverter`. Parse failures
surface as
`ParquetDecodingException`.
### Are these changes tested?
Yes. New `ProtoEmptyMessageTest` (new test messages `Stub`/`StubBox` in
`Trees.proto`) writes
through the real write path (`ProtoParquetWriter` → `MessageColumnIO`, both
specs-compliant and
old style) and reads back with `GroupReadSupport` and with
`ProtoParquetReader`:
- singular / repeated / map-value empty-message fields round-trip with
correct cardinality and
zero-byte values;
- presence round-trips (set empty message vs unset field);
- `ProtoParquetReader` reads the written messages back equal to the
originals (specs-compliant and
old style), and the same read path round-trips a `BinaryTree` truncated
at `maxRecursion`;
- an empty root message still fails with `InvalidSchemaException` ("Cannot
write a schema with an
empty group").
`ProtoSchemaConverterTest.testEmptyMessageFields` pins the converted
schema. The full
parquet-protobuf suite passes (117 tests).
### Are there any user-facing changes?
Message types that previously could not be written to Parquet at all now
can; fields of empty
message types appear as (possibly LIST/MAP-wrapped) `binary` columns and
read back into the
original messages with `ProtoParquetReader`. Files with `maxRecursion`
truncation, previously
unreadable by `ProtoParquetReader`, now read back as well. No change for
schemas that were
previously writable. Error behavior for an empty root message is unchanged.
Closes #2142
---
parquet-protobuf/README.md | 45 ++++
.../parquet/proto/ProtoMessageConverter.java | 37 +++
.../apache/parquet/proto/ProtoSchemaConverter.java | 28 +++
.../apache/parquet/proto/ProtoWriteSupport.java | 33 +--
.../parquet/proto/ProtoEmptyMessageTest.java | 250 +++++++++++++++++++++
.../parquet/proto/ProtoSchemaConverterTest.java | 21 ++
parquet-protobuf/src/test/resources/Trees.proto | 12 +
7 files changed, 411 insertions(+), 15 deletions(-)
diff --git a/parquet-protobuf/README.md b/parquet-protobuf/README.md
index c073b93bc..34981065f 100644
--- a/parquet-protobuf/README.md
+++ b/parquet-protobuf/README.md
@@ -21,3 +21,48 @@ parquet-protobuf
================
Protocol Buffer support for Parquet columnar format.
+
+## Message fields stored as proto bytes
+
+Two kinds of message fields cannot be mapped to a Parquet group, so
`ProtoSchemaConverter`
+terminates them as the **serialized protobuf message** instead:
+
+* **Fields of an empty message type** (`message Stub {}`) — Parquet
forbids empty groups.
+ An empty message serializes to zero bytes, so the column is cheap, and field
presence still
+ round-trips: `null` means the field was unset, an empty value means it was
set.
+* **Recursive fields beyond `parquet.proto.maxRecursion`** (default 5) —
the remaining
+ sub-tree is stored as the serialized message instead of expanding the schema
forever.
+
+The Parquet type is an unannotated `BINARY` column that keeps the field's own
repetition (or,
+for repeated fields and map values, sits inside the standard `LIST` / `MAP`
wrappers when
+`parquet.proto.writeSpecsCompliant` is set):
+
+```
+message Trees.StubBox {
+ optional binary stub = 1; // Stub stub = 1;
+ optional group stubs (LIST) = 2 { // repeated Stub stubs = 2;
+ repeated group list {
+ required binary element;
+ }
+ }
+ optional group stub_map (MAP) = 3 { // map<string, Stub> stub_map =
3;
+ repeated group key_value {
+ required binary key (STRING);
+ optional binary value;
+ }
+ }
+}
+```
+
+Readers that do not know about protobuf simply see opaque bytes (all of them
empty for an
+empty message type). Readers that have the generated message class can parse
the bytes back into
+the message; `ProtoParquetReader` does this automatically, resolving the class
from the
+`parquet.proto.class` footer key (or from the class configured for reading).
The writer also stores
+the message descriptor in the footer under `parquet.proto.descriptor`, which
tools that do not
+have the generated class can use to interpret the bytes; `ProtoParquetReader`
itself does not read
+it.
+
+Note that the column type follows the proto schema at write time: if an empty
message type later
+gains fields, or `parquet.proto.maxRecursion` is changed, new files store the
field as a group
+where old files store `BINARY`. Tools that merge schemas across such files
will report a type
+conflict, the same way they do for any other field whose type changed.
diff --git
a/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoMessageConverter.java
b/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoMessageConverter.java
index ff9e4ca33..0da0dfca9 100644
---
a/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoMessageConverter.java
+++
b/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoMessageConverter.java
@@ -36,6 +36,7 @@ import com.google.protobuf.DoubleValue;
import com.google.protobuf.FloatValue;
import com.google.protobuf.Int32Value;
import com.google.protobuf.Int64Value;
+import com.google.protobuf.InvalidProtocolBufferException;
import com.google.protobuf.Message;
import com.google.protobuf.StringValue;
import com.google.protobuf.UInt32Value;
@@ -347,6 +348,9 @@ class ProtoMessageConverter extends GroupConverter {
if (messageType.equals(BytesValue.getDescriptor())) {
return new ProtoBytesValueConverter(pvc);
}
+ // Otherwise the column holds the serialized message itself:
ProtoSchemaConverter stores
+ // fields of empty message types and recursion beyond maxRecursion
as proto bytes.
+ return new ProtoBinaryMessageConverter(pvc, parentBuilder,
fieldDescriptor);
}
Message.Builder subBuilder =
parentBuilder.newBuilderForField(fieldDescriptor);
return new ProtoMessageConverter(conf, pvc, subBuilder,
parquetType.asGroupType(), extraMetadata);
@@ -495,6 +499,39 @@ class ProtoMessageConverter extends GroupConverter {
}
}
+ /**
+ * Reads a message field that {@link ProtoSchemaConverter} stored as the
serialized proto bytes
+ * (a field of an empty message type, or recursion truncated at
maxRecursion) back into the
+ * message.
+ */
+ static final class ProtoBinaryMessageConverter extends PrimitiveConverter {
+
+ private final ParentValueContainer parent;
+ private final Message.Builder parentBuilder;
+ private final Descriptors.FieldDescriptor fieldDescriptor;
+
+ ProtoBinaryMessageConverter(
+ ParentValueContainer parent,
+ Message.Builder parentBuilder,
+ Descriptors.FieldDescriptor fieldDescriptor) {
+ this.parent = parent;
+ this.parentBuilder = parentBuilder;
+ this.fieldDescriptor = fieldDescriptor;
+ }
+
+ @Override
+ public void addBinary(Binary binary) {
+ Message.Builder builder =
parentBuilder.newBuilderForField(fieldDescriptor);
+ try {
+ builder.mergeFrom(ByteString.copyFrom(binary.toByteBuffer()));
+ } catch (InvalidProtocolBufferException e) {
+ throw new ParquetDecodingException(
+ "Cannot parse field " + fieldDescriptor.getFullName() + " from its
serialized proto bytes", e);
+ }
+ parent.add(builder.build());
+ }
+ }
+
static final class ProtoBinaryConverter extends PrimitiveConverter {
final ParentValueContainer parent;
diff --git
a/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoSchemaConverter.java
b/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoSchemaConverter.java
index ff27b263e..dc0c97b7f 100644
---
a/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoSchemaConverter.java
+++
b/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoSchemaConverter.java
@@ -70,6 +70,22 @@ import org.slf4j.LoggerFactory;
/**
* Converts a Protocol Buffer Descriptor into a Parquet schema.
+ * <p>
+ * Message fields normally become Parquet groups. Two kinds of message fields
cannot, and are
+ * instead terminated as an unannotated {@code BINARY} column holding the
serialized proto message
+ * (keeping the field's repetition, or sitting inside the usual LIST/MAP
wrappers):
+ * <ul>
+ * <li>fields of an <em>empty</em> message type, because Parquet forbids
empty groups; the value is
+ * zero bytes when the field is set and {@code null} when it is not, so
presence still
+ * round-trips;</li>
+ * <li>recursive fields nested deeper than {@code maxRecursion}.</li>
+ * </ul>
+ * Readers unaware of protobuf see opaque bytes. {@code ProtoParquetReader}
parses them back into the
+ * message using the generated class it resolves from the {@code
parquet.proto.class} footer key (or
+ * the class configured for reading). Since the column type follows
+ * the proto schema at write time, an empty message type that later gains
fields (or a changed
+ * {@code maxRecursion}) produces a group where older files hold {@code
BINARY}, like any other
+ * field whose type changed. See the parquet-protobuf README for details.
*/
public class ProtoSchemaConverter {
@@ -327,6 +343,18 @@ public class ProtoSchemaConverter {
return addMapField(descriptor, builder, seen, depth);
}
+ // Parquet forbids empty groups, so a field of an empty message type is
terminated as proto
+ // bytes (zero bytes when the message is set - presence still
round-trips), preserving the
+ // field's repetition so the write path (Array/Repeated/MapWriter) still
matches the schema.
+ if (descriptor.getMessageType().getFields().isEmpty()) {
+ if (descriptor.isRepeated() && parquetSpecsCompliant) {
+ // LIST-wrap the truncated bytes the same way any repeated primitive
is wrapped
+ return addRepeatedPrimitive(BINARY, null, builder);
+ }
+ // optional, required, or repeated in the old schema style
+ return builder.primitive(BINARY,
getRepetition(descriptor)).as((LogicalTypeAnnotation) null);
+ }
+
seen = ImmutableSetMultimap.<String, Integer>builder()
.putAll(seen)
.put(typeName, depth)
diff --git
a/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoWriteSupport.java
b/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoWriteSupport.java
index 51e2d7e25..c6109468f 100644
---
a/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoWriteSupport.java
+++
b/parquet-protobuf/src/main/java/org/apache/parquet/proto/ProtoWriteSupport.java
@@ -356,42 +356,45 @@ public class ProtoWriteSupport<T extends
MessageOrBuilder> extends WriteSupport<
}
// This can happen now that recursive schemas get truncated to bytes.
Write the bytes.
- if (type.isPrimitive()
- && type.asPrimitiveType().getPrimitiveTypeName() ==
PrimitiveType.PrimitiveTypeName.BINARY) {
+ // The truncated type keeps the field's shape, so it may sit behind a
LIST wrapper
+ // (repeated field) or be the value inside a MAP's key_value group.
+ Type contentType = getContentType(type);
+ if (contentType.isPrimitive()
+ && contentType.asPrimitiveType().getPrimitiveTypeName() ==
PrimitiveType.PrimitiveTypeName.BINARY) {
return new BinaryWriter();
}
- return new MessageWriter(fieldDescriptor.getMessageType(),
getGroupType(type));
+ return new MessageWriter(fieldDescriptor.getMessageType(),
contentType.asGroupType());
}
- private GroupType getGroupType(Type type) {
+ /** Unwraps the LIST/MAP wrapper groups to the type holding the message
content itself. */
+ private Type getContentType(Type type) {
+ if (type.isPrimitive()) {
+ return type;
+ }
LogicalTypeAnnotation logicalTypeAnnotation =
type.getLogicalTypeAnnotation();
if (logicalTypeAnnotation == null) {
- return type.asGroupType();
+ return type;
}
return logicalTypeAnnotation
- .accept(new
LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<GroupType>() {
+ .accept(new
LogicalTypeAnnotation.LogicalTypeAnnotationVisitor<Type>() {
@Override
- public Optional<GroupType> visit(
- LogicalTypeAnnotation.ListLogicalTypeAnnotation
listLogicalType) {
+ public Optional<Type>
visit(LogicalTypeAnnotation.ListLogicalTypeAnnotation listLogicalType) {
return ofNullable(type.asGroupType()
.getType("list")
.asGroupType()
- .getType("element")
- .asGroupType());
+ .getType("element"));
}
@Override
- public Optional<GroupType> visit(
- LogicalTypeAnnotation.MapLogicalTypeAnnotation mapLogicalType)
{
+ public Optional<Type>
visit(LogicalTypeAnnotation.MapLogicalTypeAnnotation mapLogicalType) {
return ofNullable(type.asGroupType()
.getType("key_value")
.asGroupType()
- .getType("value")
- .asGroupType());
+ .getType("value"));
}
})
- .orElse(type.asGroupType());
+ .orElse(type);
}
private MapWriter createMapWriter(FieldDescriptor fieldDescriptor, Type
type) {
diff --git
a/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoEmptyMessageTest.java
b/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoEmptyMessageTest.java
new file mode 100644
index 000000000..1d70d9b17
--- /dev/null
+++
b/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoEmptyMessageTest.java
@@ -0,0 +1,250 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.parquet.proto;
+
+import static org.assertj.core.api.Assertions.assertThat;
+import static org.assertj.core.api.Assertions.assertThatThrownBy;
+
+import com.google.protobuf.Message;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.example.data.Group;
+import org.apache.parquet.hadoop.ParquetFileReader;
+import org.apache.parquet.hadoop.ParquetReader;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.example.GroupReadSupport;
+import org.apache.parquet.proto.test.Trees;
+import org.apache.parquet.schema.InvalidSchemaException;
+import org.apache.parquet.schema.MessageType;
+import org.apache.parquet.schema.MessageTypeParser;
+import org.apache.parquet.schema.PrimitiveType;
+import org.junit.jupiter.api.Test;
+
+/**
+ * Fields typed as an EMPTY proto message cannot map to a parquet group
(parquet forbids empty
+ * groups, so writer construction used to fail with an {@code
InvalidSchemaException}). They are
+ * now terminated as proto bytes, like recursion beyond maxRecursion, which
also keeps the field's
+ * presence observable (null vs an empty byte array).
+ * <p>
+ * The written files are read back both with {@link ProtoParquetReader} (which
parses the bytes
+ * back into the message) and with parquet-hadoop's protobuf-agnostic {@link
GroupReadSupport},
+ * to show that a reader without protobuf knowledge sees plain, unannotated
binary columns.
+ */
+public class ProtoEmptyMessageTest {
+
+ private static Path write(boolean specsCompliant, Message... messages)
throws IOException {
+ Path file = TestUtils.someTemporaryFilePath();
+ Configuration conf = new Configuration();
+ ProtoWriteSupport.setWriteSpecsCompliant(conf, specsCompliant);
+ try (ParquetWriter<Message> writer =
ProtoParquetWriter.<Message>builder(file)
+ .withMessage(messages[0].getClass())
+ .withConf(conf)
+ .build()) {
+ for (Message message : messages) {
+ writer.write(message);
+ }
+ }
+ return file;
+ }
+
+ /** Reads with parquet-hadoop's generic {@link GroupReadSupport}, which
knows nothing about protobuf. */
+ private static List<Group> readWithGenericReader(Path file) throws
IOException {
+ List<Group> rows = new ArrayList<>();
+ try (ParquetReader<Group> reader =
+ ParquetReader.builder(new GroupReadSupport(), file).build()) {
+ for (Group group = reader.read(); group != null; group = reader.read()) {
+ rows.add(group);
+ }
+ }
+ return rows;
+ }
+
+ private static MessageType readFooterSchema(Path file) throws IOException {
+ try (ParquetFileReader reader = ParquetFileReader.open(new
Configuration(), file)) {
+ return reader.getFileMetaData().getSchema();
+ }
+ }
+
+ @Test
+ public void emptyMessageFieldsAreUnannotatedBinaryColumns() throws Exception
{
+ // What a reader without protobuf knowledge sees in the footer: plain
BINARY columns with no
+ // logical type, keeping the field's repetition (or the standard LIST/MAP
wrappers).
+ MessageType schema = readFooterSchema(write(true,
Trees.StubBox.getDefaultInstance()));
+
+ assertThat(schema)
+ .isEqualTo(MessageTypeParser.parseMessageType("message Trees.StubBox
{\n"
+ + " optional binary stub = 1;\n"
+ + " optional group stubs (LIST) = 2 {\n"
+ + " repeated group list {\n"
+ + " required binary element;\n"
+ + " }\n"
+ + " }\n"
+ + " optional group stub_map (MAP) = 3 {\n"
+ + " repeated group key_value {\n"
+ + " required binary key (STRING);\n"
+ + " optional binary value;\n"
+ + " }\n"
+ + " }\n"
+ + " optional binary name (STRING) = 4;\n"
+ + "}"));
+
+ PrimitiveType stub = schema.getType("stub").asPrimitiveType();
+
assertThat(stub.getPrimitiveTypeName()).isEqualTo(PrimitiveType.PrimitiveTypeName.BINARY);
+ assertThat(stub.getLogicalTypeAnnotation())
+ .as("no logical type: the bytes are opaque to non-protobuf readers")
+ .isNull();
+
+ PrimitiveType element = schema.getType("stubs", "list",
"element").asPrimitiveType();
+
assertThat(element.getPrimitiveTypeName()).isEqualTo(PrimitiveType.PrimitiveTypeName.BINARY);
+ assertThat(element.getLogicalTypeAnnotation()).isNull();
+
+ PrimitiveType value = schema.getType("stub_map", "key_value",
"value").asPrimitiveType();
+
assertThat(value.getPrimitiveTypeName()).isEqualTo(PrimitiveType.PrimitiveTypeName.BINARY);
+ assertThat(value.getLogicalTypeAnnotation()).isNull();
+ }
+
+ @Test
+ public void emptyMessageFieldsAreUnannotatedBinaryColumnsOldStyle() throws
Exception {
+ MessageType schema = readFooterSchema(write(false,
Trees.StubBox.getDefaultInstance()));
+
+ PrimitiveType stubs = schema.getType("stubs").asPrimitiveType();
+ assertThat(stubs.getRepetition())
+ .as("old style keeps the repeated field itself")
+ .isEqualTo(PrimitiveType.Repetition.REPEATED);
+
assertThat(stubs.getPrimitiveTypeName()).isEqualTo(PrimitiveType.PrimitiveTypeName.BINARY);
+ assertThat(stubs.getLogicalTypeAnnotation()).isNull();
+ }
+
+ @Test
+ public void emptyMessageFieldsWriteAsBytes() throws Exception {
+ Trees.StubBox box = Trees.StubBox.newBuilder()
+ .setStub(Trees.Stub.getDefaultInstance())
+ .addStubs(Trees.Stub.getDefaultInstance())
+ .addStubs(Trees.Stub.getDefaultInstance())
+ .putStubMap("k", Trees.Stub.getDefaultInstance())
+ .setName("x")
+ .build();
+
+ Group row = readWithGenericReader(write(true, box)).get(0);
+ assertThat(row.getBinary("stub", 0).length())
+ .as("optional empty message present as zero bytes")
+ .isEqualTo(0);
+ assertThat(row.getGroup("stubs", 0).getFieldRepetitionCount("list"))
+ .as("repeated empty messages keep their cardinality")
+ .isEqualTo(2);
+ Group entry = row.getGroup("stub_map", 0).getGroup("key_value", 0);
+ assertThat(entry.getString("key", 0)).as("map keys stay
typed").isEqualTo("k");
+ assertThat(entry.getBinary("value", 0).length())
+ .as("map value is zero bytes")
+ .isEqualTo(0);
+ assertThat(row.getString("name", 0)).isEqualTo("x");
+ }
+
+ @Test
+ public void emptyMessagePresenceRoundTrips() throws Exception {
+ Trees.StubBox with = Trees.StubBox.newBuilder()
+ .setStub(Trees.Stub.getDefaultInstance())
+ .build();
+ Trees.StubBox without = Trees.StubBox.getDefaultInstance();
+
+ List<Group> rows = readWithGenericReader(write(true, with, without));
+ assertThat(rows.get(0).getFieldRepetitionCount("stub"))
+ .as("set empty message is present")
+ .isEqualTo(1);
+ assertThat(rows.get(1).getFieldRepetitionCount("stub"))
+ .as("unset empty message is null")
+ .isEqualTo(0);
+ }
+
+ @Test
+ public void emptyMessageFieldsWriteAsBytesOldStyle() throws Exception {
+ Trees.StubBox box = Trees.StubBox.newBuilder()
+ .addStubs(Trees.Stub.getDefaultInstance())
+ .addStubs(Trees.Stub.getDefaultInstance())
+ .build();
+
+ Group row = readWithGenericReader(write(false, box)).get(0);
+ assertThat(row.getFieldRepetitionCount("stubs"))
+ .as("repeated empty messages keep their cardinality")
+ .isEqualTo(2);
+ }
+
+ @Test
+ public void emptyMessageFieldsRoundTripThroughProtoReader() throws Exception
{
+ Trees.StubBox box = Trees.StubBox.newBuilder()
+ .setStub(Trees.Stub.getDefaultInstance())
+ .addStubs(Trees.Stub.getDefaultInstance())
+ .addStubs(Trees.Stub.getDefaultInstance())
+ .putStubMap("k", Trees.Stub.getDefaultInstance())
+ .setName("x")
+ .build();
+ Trees.StubBox without = Trees.StubBox.newBuilder().setName("y").build();
+
+ assertThat(TestUtils.readMessages(write(true, box, without),
Trees.StubBox.class))
+ .as("ProtoParquetReader parses the proto bytes back, presence
included")
+ .containsExactly(box, without);
+ }
+
+ @Test
+ public void emptyMessageFieldsRoundTripThroughProtoReaderOldStyle() throws
Exception {
+ Trees.StubBox box = Trees.StubBox.newBuilder()
+ .setStub(Trees.Stub.getDefaultInstance())
+ .addStubs(Trees.Stub.getDefaultInstance())
+ .addStubs(Trees.Stub.getDefaultInstance())
+ .putStubMap("k", Trees.Stub.getDefaultInstance())
+ .build();
+
+ assertThat(TestUtils.readMessages(write(false, box), Trees.StubBox.class))
+ .containsExactly(box);
+ }
+
+ @Test
+ public void truncatedRecursionRoundTripsThroughProtoReader() throws
Exception {
+ // the same binary-to-message read path serves recursion truncated at
maxRecursion
+ Trees.BinaryTree.Builder tree = Trees.BinaryTree.newBuilder();
+ Trees.BinaryTree.Builder cursor = tree;
+ for (int i = 0; i < 6; i++) {
+ cursor.getValueBuilder().setTypeUrl("level-" + i);
+ cursor = cursor.getLeftBuilder();
+ }
+ Path file = TestUtils.someTemporaryFilePath();
+ Configuration conf = new Configuration();
+ ProtoWriteSupport.setWriteSpecsCompliant(conf, true);
+ ProtoSchemaConverter.setMaxRecursion(conf, 2);
+ try (ParquetWriter<Message> writer =
ProtoParquetWriter.<Message>builder(file)
+ .withMessage(Trees.BinaryTree.class)
+ .withConf(conf)
+ .build()) {
+ writer.write(tree.build());
+ }
+
+ assertThat(TestUtils.readMessages(file,
Trees.BinaryTree.class)).containsExactly(tree.build());
+ }
+
+ @Test
+ public void emptyRootMessageStillRejected() {
+ // the root message itself cannot be terminated as bytes - there is no
field to hold them
+ assertThatThrownBy(() -> write(true, Trees.Stub.getDefaultInstance()))
+ .isInstanceOf(InvalidSchemaException.class)
+ .hasMessageContaining("Cannot write a schema with an empty group");
+ }
+}
diff --git
a/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoSchemaConverterTest.java
b/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoSchemaConverterTest.java
index a4539e339..5c8bc0b7e 100644
---
a/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoSchemaConverterTest.java
+++
b/parquet-protobuf/src/test/java/org/apache/parquet/proto/ProtoSchemaConverterTest.java
@@ -579,6 +579,27 @@ public class ProtoSchemaConverterTest {
new ProtoSchemaConverter(true, PAR_RECURSION_DEPTH, false));
}
+ @Test
+ public void testEmptyMessageFields() throws Exception {
+ String expectedSchema = JOINER.join(
+ "message Trees.StubBox {",
+ " optional binary stub = 1;",
+ " optional group stubs (LIST) = 2 {",
+ " repeated group list {",
+ " required binary element;",
+ " }",
+ " }",
+ " optional group stub_map (MAP) = 3 {",
+ " repeated group key_value {",
+ " required binary key (STRING);",
+ " optional binary value;",
+ " }",
+ " }",
+ " optional binary name (STRING) = 4;",
+ "}");
+ testConversion(Trees.StubBox.class, expectedSchema, new
ProtoSchemaConverter(true, 5, false));
+ }
+
@Test
public void testDeepRecursion() {
// The general idea is to test the fanout of the schema.
diff --git a/parquet-protobuf/src/test/resources/Trees.proto
b/parquet-protobuf/src/test/resources/Trees.proto
index c62754d6b..342f6b094 100644
--- a/parquet-protobuf/src/test/resources/Trees.proto
+++ b/parquet-protobuf/src/test/resources/Trees.proto
@@ -35,3 +35,15 @@ message WideTree {
google.protobuf.Any value = 1;
repeated WideTree children = 2;
}
+
+// An empty message: parquet cannot represent an empty group, so fields of this
+// type are terminated as proto bytes (like recursion beyond maxRecursion).
+message Stub {
+}
+
+message StubBox {
+ Stub stub = 1;
+ repeated Stub stubs = 2;
+ map<string, Stub> stub_map = 3;
+ string name = 4;
+}
