Thanks for the insight, Robert! The Java class is autogenerated from a protobuf schema. I'm not very knowledgeable regarding proto, but it looks like NOT NULLness can be achieved through *oneof kind* and through implementation of a new data type that wraps around the desired data. What are your thoughts on that? I tried adding the NOT NULL keyword to each column when creating the tables [1] <https://github.com/fernando-wizeline/beam/blob/abc17db41b6aabf3f337c7742526e5ae9655f40b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubTableProviderIT.java#L658> and [2] <https://github.com/fernando-wizeline/beam/blob/abc17db41b6aabf3f337c7742526e5ae9655f40b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubTableProviderIT.java#L671> but that failed when executing the *executeDdl* method.
Is it worth to try the *oneof kind *approach or is there a simpler way I'm not seeing to accomplish the NOT NULLness? Thanks again! On Tue, Apr 20, 2021 at 2:08 PM Robert Burke <[email protected]> wrote: > It looks like it doesn't consider TYPE to match TYPE NOT NULL. I don't > know how Beam Java handles that but I'd guess you'd need to annotate > somehow the fields to make them match. > > On Tue, Apr 20, 2021, 12:24 PM Fernando Morales Martinez < > [email protected]> wrote: > >> sure thing! >> This is the error from method *testSQLInsertRowsToPubsubFlat* >> >> Given message schema: 'Fields: >> Field{name=name, description=, type=STRING, options={{}}} >> Field{name=height, description=, type=INT32, options={{}}} >> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}} >> Options:{{}}' >> does not match schema inferred from protobuf class. >> Protobuf class: >> 'org.apache.beam.sdk.extensions.protobuf.PayloadMessages$NameHeightKnowsJSMessage' >> Inferred schema: 'Fields: >> Field{name=name, description=, type=STRING NOT NULL, >> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >> value=1}}}} >> Field{name=height, description=, type=INT32 NOT NULL, >> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >> value=2}}}} >> Field{name=knowsJs, description=, type=BOOLEAN NOT NULL, >> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >> value=3}}}} >> Options:{{beam:option:proto:meta:type_name=Option{type=STRING NOT NULL, >> value=NameHeightKnowsJSMessage}}}' >> >> And this is the message from >> *testSQLInsertRowsToPubsubWithTimestampAttributeFlat*. >> >> Given message schema: 'Fields: >> Field{name=name, description=, type=STRING, options={{}}} >> Field{name=height, description=, type=INT32, options={{}}} >> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}} >> Options:{{}}' >> does not match schema inferred from protobuf class. >> Protobuf class: >> 'org.apache.beam.sdk.extensions.protobuf.PayloadMessages$NameHeightKnowsJSMessage' >> Inferred schema: 'Fields: >> Field{name=name, description=, type=STRING NOT NULL, >> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >> value=1}}}} >> Field{name=height, description=, type=INT32 NOT NULL, >> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >> value=2}}}} >> Field{name=knowsJs, description=, type=BOOLEAN NOT NULL, >> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >> value=3}}}} >> Options:{{beam:option:proto:meta:type_name=Option{type=STRING NOT NULL, >> value=NameHeightKnowsJSMessage}}}' >> >> Maybe the proto variable *knowsJs* should be called *knowsJavascript*? >> >> >> On Tue, Apr 20, 2021 at 12:26 PM Daniel Collins <[email protected]> >> wrote: >> >>> The error includes "NameHeightMessage". Can you provide an example error >>> from one of the other methods? >>> >>> On Tue, Apr 20, 2021 at 1:56 PM Fernando Morales Martinez < >>> [email protected]> wrote: >>> >>>> Thanks for the heads up, Daniel! >>>> >>>> I missed changing that one, but even after making the change, I'm >>>> getting the same error. >>>> >>>> The other two methods, *testSQLInsertRowsToPubsubFlat* and >>>> *testSQLInsertRowsToPubsubWithTimestampAttributeFlat*, were already >>>> using NameHeightKnowsJSMessage class but are still throwing the same error. >>>> >>>> Any idea what else might be going on? >>>> >>>> On Tue, Apr 20, 2021 at 11:13 AM Daniel Collins <[email protected]> >>>> wrote: >>>> >>>>> Thanks for working on this! It looks to me like the schemas don't >>>>> match: you appear to be using NameHeightMessage defined as: >>>>> >>>>> ``` >>>>> message NameHeightMessage { >>>>> string name = 1; >>>>> int32 height = 2; >>>>> } >>>>> ``` >>>>> >>>>> And expecting it to work with a table schema that has a "BOOL >>>>> knowsJavascript" field. Did you mean to use the "NameHeightKnowsJSMessage" >>>>> class? >>>>> >>>>> -Daniel >>>>> >>>>> On Tue, Apr 20, 2021 at 1:02 PM Fernando Morales Martinez < >>>>> [email protected]> wrote: >>>>> >>>>>> Sorry for the spam, forgot to add the pertinent link to the code >>>>>> change. >>>>>> >>>>>> >>>>>> https://github.com/fernando-wizeline/beam/commit/abc17db41b6aabf3f337c7742526e5ae9655f40b >>>>>> >>>>>> Thanks! >>>>>> >>>>>> On Tue, Apr 20, 2021 at 10:17 AM Fernando Morales Martinez < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi team, >>>>>>> >>>>>>> I'm working on adding tests to PubsubTableProviderIT class to test >>>>>>> the proto support added to Pubsub. >>>>>>> >>>>>>> The issue below occurs when running >>>>>>> *testSQLReadAndWriteWithSameFlatTableDefinition*, >>>>>>> *testSQLInsertRowsToPubsubFlat* and >>>>>>> *testSQLInsertRowsToPubsubWithTimestampAttributeFlat* >>>>>>> of PubsubTableProviderIT. >>>>>>> >>>>>>> Right now I'm facing an issue when executing method >>>>>>> inferAndVerifySchema of class ProtoPayloadSerializerProvider. The >>>>>>> expected >>>>>>> schema is set as >>>>>>> >>>>>>> 'Fields: >>>>>>> Field{name=name, description=, type=STRING, options={{}}} >>>>>>> Field{name=height, description=, type=INT32, options={{}}} >>>>>>> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}} >>>>>>> Options:{{}}' >>>>>>> >>>>>>> and the schema obtained from the protoclass is: >>>>>>> >>>>>>> 'Fields: >>>>>>> Field{name=name, description=, type=STRING NOT NULL, >>>>>>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>>>>>> value=1}}}} >>>>>>> Field{name=height, description=, type=INT32 NOT NULL, >>>>>>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL, >>>>>>> value=2}}}} >>>>>>> Options:{{beam:option:proto:meta:type_name=Option{type=STRING NOT >>>>>>> NULL, value=NameHeightMessage}}}' >>>>>>> java.lang.IllegalArgumentException: Given message schema: 'Fields: >>>>>>> Field{name=name, description=, type=STRING, options={{}}} >>>>>>> Field{name=height, description=, type=INT32, options={{}}} >>>>>>> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}} >>>>>>> Options:{{}}' >>>>>>> >>>>>>> I'm guessing, by the name of the tests, the idea is to compare the >>>>>>> payload (protoclass in this case) against the flat schema, but then the >>>>>>> validation in inferAndVerifySchema fails. >>>>>>> >>>>>>> Should I be looking for a workaround for that and if so, can you >>>>>>> shed some light on how to proceed? >>>>>>> >>>>>>> Thanks for the help! >>>>>>> - Fernando Morales >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> *This email and its contents (including any attachments) are being >>>>>> sent toyou on the condition of confidentiality and may be protected by >>>>>> legalprivilege. Access to this email by anyone other than the intended >>>>>> recipientis unauthorized. If you are not the intended recipient, please >>>>>> immediatelynotify the sender by replying to this message and delete the >>>>>> materialimmediately from your system. Any further use, dissemination, >>>>>> distributionor reproduction of this email is strictly prohibited. >>>>>> Further, >>>>>> norepresentation is made with respect to any content contained in this >>>>>> email.* >>>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> *This email and its contents (including any attachments) are being sent >>>> toyou on the condition of confidentiality and may be protected by >>>> legalprivilege. Access to this email by anyone other than the intended >>>> recipientis unauthorized. If you are not the intended recipient, please >>>> immediatelynotify the sender by replying to this message and delete the >>>> materialimmediately from your system. Any further use, dissemination, >>>> distributionor reproduction of this email is strictly prohibited. Further, >>>> norepresentation is made with respect to any content contained in this >>>> email.* >>> >>> >> >> >> >> >> >> >> >> *This email and its contents (including any attachments) are being sent >> toyou on the condition of confidentiality and may be protected by >> legalprivilege. Access to this email by anyone other than the intended >> recipientis unauthorized. If you are not the intended recipient, please >> immediatelynotify the sender by replying to this message and delete the >> materialimmediately from your system. Any further use, dissemination, >> distributionor reproduction of this email is strictly prohibited. Further, >> norepresentation is made with respect to any content contained in this >> email.* > > -- *This email and its contents (including any attachments) are being sent to you on the condition of confidentiality and may be protected by legal privilege. Access to this email by anyone other than the intended recipient is unauthorized. If you are not the intended recipient, please immediately notify the sender by replying to this message and delete the material immediately from your system. Any further use, dissemination, distribution or reproduction of this email is strictly prohibited. Further, no representation is made with respect to any content contained in this email.*
