It looks like it doesn't consider TYPE to match TYPE NOT NULL. I don't know
how Beam Java handles that but I'd guess you'd need to annotate somehow the
fields to make them match.
On Tue, Apr 20, 2021, 12:24 PM Fernando Morales Martinez <
fernando.mora...@wizeline.com> wrote:
> sure thing!
> This is the error from method *testSQLInsertRowsToPubsubFlat*
>
> Given message schema: 'Fields:
> Field{name=name, description=, type=STRING, options={{}}}
> Field{name=height, description=, type=INT32, options={{}}}
> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}}
> Options:{{}}'
> does not match schema inferred from protobuf class.
> Protobuf class:
> 'org.apache.beam.sdk.extensions.protobuf.PayloadMessages$NameHeightKnowsJSMessage'
> Inferred schema: 'Fields:
> Field{name=name, description=, type=STRING NOT NULL,
> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL,
> value=1}}}}
> Field{name=height, description=, type=INT32 NOT NULL,
> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL,
> value=2}}}}
> Field{name=knowsJs, description=, type=BOOLEAN NOT NULL,
> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL,
> value=3}}}}
> Options:{{beam:option:proto:meta:type_name=Option{type=STRING NOT NULL,
> value=NameHeightKnowsJSMessage}}}'
>
> And this is the message from
> *testSQLInsertRowsToPubsubWithTimestampAttributeFlat*.
>
> Given message schema: 'Fields:
> Field{name=name, description=, type=STRING, options={{}}}
> Field{name=height, description=, type=INT32, options={{}}}
> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}}
> Options:{{}}'
> does not match schema inferred from protobuf class.
> Protobuf class:
> 'org.apache.beam.sdk.extensions.protobuf.PayloadMessages$NameHeightKnowsJSMessage'
> Inferred schema: 'Fields:
> Field{name=name, description=, type=STRING NOT NULL,
> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL,
> value=1}}}}
> Field{name=height, description=, type=INT32 NOT NULL,
> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL,
> value=2}}}}
> Field{name=knowsJs, description=, type=BOOLEAN NOT NULL,
> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL,
> value=3}}}}
> Options:{{beam:option:proto:meta:type_name=Option{type=STRING NOT NULL,
> value=NameHeightKnowsJSMessage}}}'
>
> Maybe the proto variable *knowsJs* should be called *knowsJavascript*?
>
>
> On Tue, Apr 20, 2021 at 12:26 PM Daniel Collins <dpcoll...@google.com>
> wrote:
>
>> The error includes "NameHeightMessage". Can you provide an example error
>> from one of the other methods?
>>
>> On Tue, Apr 20, 2021 at 1:56 PM Fernando Morales Martinez <
>> fernando.mora...@wizeline.com> wrote:
>>
>>> Thanks for the heads up, Daniel!
>>>
>>> I missed changing that one, but even after making the change, I'm
>>> getting the same error.
>>>
>>> The other two methods, *testSQLInsertRowsToPubsubFlat* and
>>> *testSQLInsertRowsToPubsubWithTimestampAttributeFlat*, were already
>>> using NameHeightKnowsJSMessage class but are still throwing the same error.
>>>
>>> Any idea what else might be going on?
>>>
>>> On Tue, Apr 20, 2021 at 11:13 AM Daniel Collins <dpcoll...@google.com>
>>> wrote:
>>>
>>>> Thanks for working on this! It looks to me like the schemas don't
>>>> match: you appear to be using NameHeightMessage defined as:
>>>>
>>>> ```
>>>> message NameHeightMessage {
>>>> string name = 1;
>>>> int32 height = 2;
>>>> }
>>>> ```
>>>>
>>>> And expecting it to work with a table schema that has a "BOOL
>>>> knowsJavascript" field. Did you mean to use the "NameHeightKnowsJSMessage"
>>>> class?
>>>>
>>>> -Daniel
>>>>
>>>> On Tue, Apr 20, 2021 at 1:02 PM Fernando Morales Martinez <
>>>> fernando.mora...@wizeline.com> wrote:
>>>>
>>>>> Sorry for the spam, forgot to add the pertinent link to the code
>>>>> change.
>>>>>
>>>>>
>>>>> https://github.com/fernando-wizeline/beam/commit/abc17db41b6aabf3f337c7742526e5ae9655f40b
>>>>>
>>>>> Thanks!
>>>>>
>>>>> On Tue, Apr 20, 2021 at 10:17 AM Fernando Morales Martinez <
>>>>> fernando.mora...@wizeline.com> wrote:
>>>>>
>>>>>> Hi team,
>>>>>>
>>>>>> I'm working on adding tests to PubsubTableProviderIT class to test
>>>>>> the proto support added to Pubsub.
>>>>>>
>>>>>> The issue below occurs when running
>>>>>> *testSQLReadAndWriteWithSameFlatTableDefinition*,
>>>>>> *testSQLInsertRowsToPubsubFlat* and
>>>>>> *testSQLInsertRowsToPubsubWithTimestampAttributeFlat*
>>>>>> of PubsubTableProviderIT.
>>>>>>
>>>>>> Right now I'm facing an issue when executing method
>>>>>> inferAndVerifySchema of class ProtoPayloadSerializerProvider. The
>>>>>> expected
>>>>>> schema is set as
>>>>>>
>>>>>> 'Fields:
>>>>>> Field{name=name, description=, type=STRING, options={{}}}
>>>>>> Field{name=height, description=, type=INT32, options={{}}}
>>>>>> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}}
>>>>>> Options:{{}}'
>>>>>>
>>>>>> and the schema obtained from the protoclass is:
>>>>>>
>>>>>> 'Fields:
>>>>>> Field{name=name, description=, type=STRING NOT NULL,
>>>>>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL,
>>>>>> value=1}}}}
>>>>>> Field{name=height, description=, type=INT32 NOT NULL,
>>>>>> options={{beam:option:proto:meta:number=Option{type=INT32 NOT NULL,
>>>>>> value=2}}}}
>>>>>> Options:{{beam:option:proto:meta:type_name=Option{type=STRING NOT
>>>>>> NULL, value=NameHeightMessage}}}'
>>>>>> java.lang.IllegalArgumentException: Given message schema: 'Fields:
>>>>>> Field{name=name, description=, type=STRING, options={{}}}
>>>>>> Field{name=height, description=, type=INT32, options={{}}}
>>>>>> Field{name=knowsJavascript, description=, type=BOOLEAN, options={{}}}
>>>>>> Options:{{}}'
>>>>>>
>>>>>> I'm guessing, by the name of the tests, the idea is to compare the
>>>>>> payload (protoclass in this case) against the flat schema, but then the
>>>>>> validation in inferAndVerifySchema fails.
>>>>>>
>>>>>> Should I be looking for a workaround for that and if so, can you shed
>>>>>> some light on how to proceed?
>>>>>>
>>>>>> Thanks for the help!
>>>>>> - Fernando Morales
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *This email and its contents (including any attachments) are being
>>>>> sent toyou on the condition of confidentiality and may be protected by
>>>>> legalprivilege. Access to this email by anyone other than the intended
>>>>> recipientis unauthorized. If you are not the intended recipient, please
>>>>> immediatelynotify the sender by replying to this message and delete the
>>>>> materialimmediately from your system. Any further use, dissemination,
>>>>> distributionor reproduction of this email is strictly prohibited. Further,
>>>>> norepresentation is made with respect to any content contained in this
>>>>> email.*
>>>>
>>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> *This email and its contents (including any attachments) are being sent
>>> toyou on the condition of confidentiality and may be protected by
>>> legalprivilege. Access to this email by anyone other than the intended
>>> recipientis unauthorized. If you are not the intended recipient, please
>>> immediatelynotify the sender by replying to this message and delete the
>>> materialimmediately from your system. Any further use, dissemination,
>>> distributionor reproduction of this email is strictly prohibited. Further,
>>> norepresentation is made with respect to any content contained in this
>>> email.*
>>
>>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*
