The problem is that protobuf primitives are represented in Java as primitives, which are not nullable.
Ideally, they should be objects instead, but alas - no. The wrapper is a decent (but not perfect) workaround. On Mon, Jun 7, 2021, 18:01 Reuven Lax <[email protected]> wrote: > I believe that as of proto 3.12, optional fields are supported directly - > https://github.com/pseudomuto/protoc-gen-doc/issues/422 . _think_ this > should be supported by Beam (assuming Beam uses a new-enough proto > library), but I'm not sure if it's been tested. > > On Mon, Jun 7, 2021 at 8:53 AM Andrew Kettmann <[email protected]> > wrote: > >> Thanks that looks like it is the issue, I appreciate the help. >> ------------------------------ >> *From:* Thomas Fredriksen(External) <[email protected]> >> *Sent:* Monday, June 7, 2021 12:53 AM >> *To:* [email protected] <[email protected]> >> *Subject:* Re: [2.28.0] [Java] [protobuf] ProtoMessageSchema doesn't >> create fields as nullable >> >> Hi Andrew, >> >> From the documentation ( >> https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.html >> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Freleases%2Fjavadoc%2F2.19.0%2Forg%2Fapache%2Fbeam%2Fsdk%2Fextensions%2Fprotobuf%2FProtoSchemaTranslator.html&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013163228%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=iFaa1RyJbbRVIU%2FqnDoxvvJzXuxSrkMwilc8jYFh%2Btc%3D&reserved=0> >> ): >> >> Protobuf wrapper classes are translated to nullable types, as follows. >> >> - google.protobuf.Int32Value maps to a nullable FieldType.INT32 >> - google.protobuf.Int64Value maps to a nullable FieldType.INT64 >> - google.protobuf.UInt32Value maps to a nullable >> FieldType.logicalType(new UInt32()) >> - google.protobuf.UInt64Value maps to a nullable >> Field.logicalType(new UInt64()) >> - google.protobuf.FloatValue maps to a nullable FieldType.FLOAT >> - google.protobuf.DoubleValue maps to a nullable FieldType.DOUBLE >> - google.protobuf.BoolValue maps to a nullable FieldType.BOOLEAN >> - google.protobuf.StringValue maps to a nullable FieldType.STRING >> - google.protobuf.BytesValue maps to a nullable FieldType.BYTES >> >> This means that you should use the google wrapper-types in order to >> achieve nullable fields. >> The wrapper is available here: >> https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto >> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprotocolbuffers%2Fprotobuf%2Fblob%2Fmaster%2Fsrc%2Fgoogle%2Fprotobuf%2Fwrappers.proto&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013173227%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ym8iX%2F4BnQsnbDfsj4sSa6F4xJEIinWSsPGnDsIc8ng%3D&reserved=0> >> . >> >> Hope it helps :) >> >> On Thu, Jun 3, 2021 at 6:48 PM Andrew Kettmann <[email protected]> >> wrote: >> >> Using org.apache.beam.sdk.extensions.protobuf.ProtoMessageSchema to >> create a beam schema from generated protobuf3 classes. However, >> org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator#beamFieldTypeFromSingularProtoField >> doesn't apply nullable to fields in the message. My understanding is that >> by default protobuf fields ARE optional, is that incorrect? Converting from >> a serialized message without values for some fields crashes when it tries >> to cast them to a Row since the Row is not expecting a field as nullable. >> >> Anyone have any advice regarding this? Modify the schema after it is >> generated by ProtoMessageSchema or is there another method/option I am >> missing? >> >> >> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.evolve24.com%2F&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013183214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Rwn8bQVGLtAX5efP0JWFaABkvFRDw6Ql0WizkQ6ouR0%3D&reserved=0> >> *Andrew >> Kettmann* >> DevOps Engineer >> P: 1.314.596.2836 <(314)%20596-2836> >> [image: LinkedIn] >> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkedin.com%2Fcompany%2Fevolve24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013193221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fFRlXtpBwHpYmeVwJL5DZ4cjFEpnXfW%2FZDVUBVLYbLc%3D&reserved=0> >> [image: >> Twitter] >> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fevolve24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013193221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sOKlR8T6dH88boG5C6JS3muO3AQxBUWr%2B9Q0w2AG934%3D&reserved=0> >> [image: >> Instagram] >> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fevolve_24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013203210%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=yBQR%2BPmDOURPaeHR5lVY00vADdu1OYpxda64UaxOmzI%3D&reserved=0> >> >> evolve24 Confidential & Proprietary Statement: This email and any >> attachments are confidential and may contain information that is >> privileged, confidential or exempt from disclosure under applicable law. It >> is intended for the use of the recipients. If you are not the intended >> recipient, or believe that you have received this communication in error, >> please do not read, print, copy, retransmit, disseminate, or otherwise use >> the information. Please delete this email and attachments, without reading, >> printing, copying, forwarding or saving them, and notify the Sender >> immediately by reply email. No confidentiality or privilege is waived or >> lost by any transmission in error. >> >>
