I think the issue might actually be a field name mismatch. It looks like you have "knowsJavascript" in the table, but "knowsJs" in the proto.
You should be able to go from NOT NULL -> NULL without issue. On Wed, Apr 21, 2021 at 11:36 AM Fernando Morales Martinez < fernando.mora...@wizeline.com> wrote: > Thanks for the insight, Robert! > The Java class is autogenerated from a protobuf schema. I'm not very > knowledgeable regarding proto, but it looks like NOT NULLness can be > achieved through *oneof kind* and through implementation of a new data > type that wraps around the desired data. What are your thoughts on that? > I tried adding the NOT NULL keyword to each column when creating the > tables [1] > <https://github.com/fernando-wizeline/beam/blob/abc17db41b6aabf3f337c7742526e5ae9655f40b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubTableProviderIT.java#L658> > and [2] > <https://github.com/fernando-wizeline/beam/blob/abc17db41b6aabf3f337c7742526e5ae9655f40b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubTableProviderIT.java#L671> > but > that failed when executing the *executeDdl* method. > > Is it worth to try the *oneof kind *approach or is there a simpler way > I'm not seeing to accomplish the NOT NULLness? > > Thanks again! > > On Tue, Apr 20, 2021 at 2:08 PM Robert Burke <rob...@frantil.com> wrote: > >> It looks like it doesn't consider TYPE to match TYPE NOT NULL. I don't >> know how Beam Java handles that but I'd guess you'd need to annotate >> somehow the fields to make them match. >> >> On Tue, Apr 20, 2021, 12:24 PM Fernando Morales Martinez < >> fernando.mora...@wizeline.com> wrote: >> >>> sure thing! >>> This is the error from method *testSQLInsertRowsToPubsubFlat* >>> >>> Given message schema: 'Fields: >>> Field{name=name, description=, type=STRING, options={{}}} >>> Field{name=height, description=, type=INT32, options={{}}} >>> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}} >>> Options:{{}}' >>> does not match schema inferred from protobuf class. >>> Protobuf class: >>> 'org.apache.beam.sdk.extensions.protobuf.PayloadMessages$NameHeightKnowsJSMessage' >>> Inferred schema: 'Fields: >>> Field{name=name, description=, type=STRING NOT NULL, >>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>> value=1}}}} >>> Field{name=height, description=, type=INT32 NOT NULL, >>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>> value=2}}}} >>> Field{name=knowsJs, description=, type=BOOLEAN NOT NULL, >>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>> value=3}}}} >>> Options:{{beam:option:proto:meta:type_name=Option{type=STRING NOT NULL, >>> value=NameHeightKnowsJSMessage}}}' >>> >>> And this is the message from >>> *testSQLInsertRowsToPubsubWithTimestampAttributeFlat*. >>> >>> Given message schema: 'Fields: >>> Field{name=name, description=, type=STRING, options={{}}} >>> Field{name=height, description=, type=INT32, options={{}}} >>> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}} >>> Options:{{}}' >>> does not match schema inferred from protobuf class. >>> Protobuf class: >>> 'org.apache.beam.sdk.extensions.protobuf.PayloadMessages$NameHeightKnowsJSMessage' >>> Inferred schema: 'Fields: >>> Field{name=name, description=, type=STRING NOT NULL, >>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>> value=1}}}} >>> Field{name=height, description=, type=INT32 NOT NULL, >>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>> value=2}}}} >>> Field{name=knowsJs, description=, type=BOOLEAN NOT NULL, >>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>> value=3}}}} >>> Options:{{beam:option:proto:meta:type_name=Option{type=STRING NOT NULL, >>> value=NameHeightKnowsJSMessage}}}' >>> >>> Maybe the proto variable *knowsJs* should be called *knowsJavascript*? >>> >>> >>> On Tue, Apr 20, 2021 at 12:26 PM Daniel Collins <dpcoll...@google.com> >>> wrote: >>> >>>> The error includes "NameHeightMessage". Can you provide an example >>>> error from one of the other methods? >>>> >>>> On Tue, Apr 20, 2021 at 1:56 PM Fernando Morales Martinez < >>>> fernando.mora...@wizeline.com> wrote: >>>> >>>>> Thanks for the heads up, Daniel! >>>>> >>>>> I missed changing that one, but even after making the change, I'm >>>>> getting the same error. >>>>> >>>>> The other two methods, *testSQLInsertRowsToPubsubFlat* and >>>>> *testSQLInsertRowsToPubsubWithTimestampAttributeFlat*, were already >>>>> using NameHeightKnowsJSMessage class but are still throwing the same >>>>> error. >>>>> >>>>> Any idea what else might be going on? >>>>> >>>>> On Tue, Apr 20, 2021 at 11:13 AM Daniel Collins <dpcoll...@google.com> >>>>> wrote: >>>>> >>>>>> Thanks for working on this! It looks to me like the schemas don't >>>>>> match: you appear to be using NameHeightMessage defined as: >>>>>> >>>>>> ``` >>>>>> message NameHeightMessage { >>>>>> string name = 1; >>>>>> int32 height = 2; >>>>>> } >>>>>> ``` >>>>>> >>>>>> And expecting it to work with a table schema that has a "BOOL >>>>>> knowsJavascript" field. Did you mean to use the >>>>>> "NameHeightKnowsJSMessage" >>>>>> class? >>>>>> >>>>>> -Daniel >>>>>> >>>>>> On Tue, Apr 20, 2021 at 1:02 PM Fernando Morales Martinez < >>>>>> fernando.mora...@wizeline.com> wrote: >>>>>> >>>>>>> Sorry for the spam, forgot to add the pertinent link to the code >>>>>>> change. >>>>>>> >>>>>>> >>>>>>> https://github.com/fernando-wizeline/beam/commit/abc17db41b6aabf3f337c7742526e5ae9655f40b >>>>>>> >>>>>>> Thanks! >>>>>>> >>>>>>> On Tue, Apr 20, 2021 at 10:17 AM Fernando Morales Martinez < >>>>>>> fernando.mora...@wizeline.com> wrote: >>>>>>> >>>>>>>> Hi team, >>>>>>>> >>>>>>>> I'm working on adding tests to PubsubTableProviderIT class to test >>>>>>>> the proto support added to Pubsub. >>>>>>>> >>>>>>>> The issue below occurs when running >>>>>>>> *testSQLReadAndWriteWithSameFlatTableDefinition*, >>>>>>>> *testSQLInsertRowsToPubsubFlat* and >>>>>>>> *testSQLInsertRowsToPubsubWithTimestampAttributeFlat* >>>>>>>> of PubsubTableProviderIT. >>>>>>>> >>>>>>>> Right now I'm facing an issue when executing method >>>>>>>> inferAndVerifySchema of class ProtoPayloadSerializerProvider. The >>>>>>>> expected >>>>>>>> schema is set as >>>>>>>> >>>>>>>> 'Fields: >>>>>>>> Field{name=name, description=, type=STRING, options={{}}} >>>>>>>> Field{name=height, description=, type=INT32, options={{}}} >>>>>>>> Field{name=knowsJavascript, description=, type=BOOLEAN, >>>>>>>> options={{}}} >>>>>>>> Options:{{}}' >>>>>>>> >>>>>>>> and the schema obtained from the protoclass is: >>>>>>>> >>>>>>>> 'Fields: >>>>>>>> Field{name=name, description=, type=STRING NOT NULL, >>>>>>>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>>>>>>> value=1}}}} >>>>>>>> Field{name=height, description=, type=INT32 NOT NULL, >>>>>>>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>>>>>>> value=2}}}} >>>>>>>> Options:{{beam:option:proto:meta:type_name=Option{type=STRING NOT >>>>>>>> NULL, value=NameHeightMessage}}}' >>>>>>>> java.lang.IllegalArgumentException: Given message schema: 'Fields: >>>>>>>> Field{name=name, description=, type=STRING, options={{}}} >>>>>>>> Field{name=height, description=, type=INT32, options={{}}} >>>>>>>> Field{name=knowsJavascript, description=, type=BOOLEAN, >>>>>>>> options={{}}} >>>>>>>> Options:{{}}' >>>>>>>> >>>>>>>> I'm guessing, by the name of the tests, the idea is to compare the >>>>>>>> payload (protoclass in this case) against the flat schema, but then the >>>>>>>> validation in inferAndVerifySchema fails. >>>>>>>> >>>>>>>> Should I be looking for a workaround for that and if so, can you >>>>>>>> shed some light on how to proceed? >>>>>>>> >>>>>>>> Thanks for the help! >>>>>>>> - Fernando Morales >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> *This email and its contents (including any attachments) are being >>>>>>> sent toyou on the condition of confidentiality and may be protected by >>>>>>> legalprivilege. Access to this email by anyone other than the intended >>>>>>> recipientis unauthorized. If you are not the intended recipient, please >>>>>>> immediatelynotify the sender by replying to this message and delete the >>>>>>> materialimmediately from your system. Any further use, dissemination, >>>>>>> distributionor reproduction of this email is strictly prohibited. >>>>>>> Further, >>>>>>> norepresentation is made with respect to any content contained in this >>>>>>> email.* >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *This email and its contents (including any attachments) are being >>>>> sent toyou on the condition of confidentiality and may be protected by >>>>> legalprivilege. Access to this email by anyone other than the intended >>>>> recipientis unauthorized. If you are not the intended recipient, please >>>>> immediatelynotify the sender by replying to this message and delete the >>>>> materialimmediately from your system. Any further use, dissemination, >>>>> distributionor reproduction of this email is strictly prohibited. Further, >>>>> norepresentation is made with respect to any content contained in this >>>>> email.* >>>> >>>> >>> >>> >>> >>> >>> >>> >>> >>> *This email and its contents (including any attachments) are being sent >>> toyou on the condition of confidentiality and may be protected by >>> legalprivilege. Access to this email by anyone other than the intended >>> recipientis unauthorized. If you are not the intended recipient, please >>> immediatelynotify the sender by replying to this message and delete the >>> materialimmediately from your system. Any further use, dissemination, >>> distributionor reproduction of this email is strictly prohibited. Further, >>> norepresentation is made with respect to any content contained in this >>> email.* >> >> > > > > > > > > *This email and its contents (including any attachments) are being sent > toyou on the condition of confidentiality and may be protected by > legalprivilege. Access to this email by anyone other than the intended > recipientis unauthorized. If you are not the intended recipient, please > immediatelynotify the sender by replying to this message and delete the > materialimmediately from your system. Any further use, dissemination, > distributionor reproduction of this email is strictly prohibited. Further, > norepresentation is made with respect to any content contained in this > email.*
