Hi Andrew, >From the documentation ( https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.html ):
Protobuf wrapper classes are translated to nullable types, as follows. > > - google.protobuf.Int32Value maps to a nullable FieldType.INT32 > - google.protobuf.Int64Value maps to a nullable FieldType.INT64 > - google.protobuf.UInt32Value maps to a nullable > FieldType.logicalType(new UInt32()) > - google.protobuf.UInt64Value maps to a nullable Field.logicalType(new > UInt64()) > - google.protobuf.FloatValue maps to a nullable FieldType.FLOAT > - google.protobuf.DoubleValue maps to a nullable FieldType.DOUBLE > - google.protobuf.BoolValue maps to a nullable FieldType.BOOLEAN > - google.protobuf.StringValue maps to a nullable FieldType.STRING > - google.protobuf.BytesValue maps to a nullable FieldType.BYTES > > This means that you should use the google wrapper-types in order to achieve nullable fields. The wrapper is available here: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto . Hope it helps :) On Thu, Jun 3, 2021 at 6:48 PM Andrew Kettmann <[email protected]> wrote: > Using org.apache.beam.sdk.extensions.protobuf.ProtoMessageSchema to create > a beam schema from generated protobuf3 classes. However, > org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator#beamFieldTypeFromSingularProtoField > doesn't apply nullable to fields in the message. My understanding is that > by default protobuf fields ARE optional, is that incorrect? Converting from > a serialized message without values for some fields crashes when it tries > to cast them to a Row since the Row is not expecting a field as nullable. > > Anyone have any advice regarding this? Modify the schema after it is > generated by ProtoMessageSchema or is there another method/option I am > missing? > > <https://www.evolve24.com> *Andrew Kettmann* > DevOps Engineer > P: 1.314.596.2836 > [image: LinkedIn] <https://linkedin.com/company/evolve24> [image: Twitter] > <https://twitter.com/evolve24> [image: Instagram] > <https://www.instagram.com/evolve_24> > > evolve24 Confidential & Proprietary Statement: This email and any > attachments are confidential and may contain information that is > privileged, confidential or exempt from disclosure under applicable law. It > is intended for the use of the recipients. If you are not the intended > recipient, or believe that you have received this communication in error, > please do not read, print, copy, retransmit, disseminate, or otherwise use > the information. Please delete this email and attachments, without reading, > printing, copying, forwarding or saving them, and notify the Sender > immediately by reply email. No confidentiality or privilege is waived or > lost by any transmission in error. >
