Re: Flink State Processor API Example - Java

[Roman Khachatryan]

Hi Sandeep,

Could you provide the error stack trace and Flink version you are using?

Regards,
Roman

On Fri, Jul 2, 2021 at 6:42 PM Sandeep khanzode  wrote:
>
> Hi Guowei,
>
> I followed the document, but somehow, I am unable to get a working Java 
> example for Avro state.
>
> So, I tried to simply use the Java SpecificRecords created by Avro Maven 
> Plugin and inject. Now, that works correctly, but I use Avro 1.7.7 since it 
> is the last version that I saw which does not put a serialVersionUid in the 
> generated SpecificRecord.
>
> How can I use a Avro SpecificRecord generated by 1.8.0 if it use a 
> serialVersionUid because the moment I change something in the Avro schema and 
> regenerate the SpecificRecord, I will get a class incompatible error when 
> Flink tried to deserialise.
>
>
> Thanks,
> Sandip
>
>
>
> On 25-Jun-2021, at 10:25 AM, Guowei Ma  wrote:
>
> Hi Sandeep
>
> What I understand is that you want to manipulate the state. So I think you 
> could use the old schema to read the state first, and then write it to a new 
> schema, instead of using a new schema to read an old schema format data.
> In addition, I would like to ask, if you want to do "State Schema Evolution" 
> ? Flink currently supports avro+pojo's schema evolution[1], and you don't 
> need to do this manually.
>
> [1] 
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#supported-data-types-for-schema-evolution
>
> Best,
> Guowei
>
>
> On Fri, Jun 25, 2021 at 3:04 AM Sandeep khanzode  
> wrote:
>>
>> Hello,
>>
>> 1.] Can someone please share a working example of how to read 
>> ValueState and MapState from a checkpoint and update it? I tried 
>> to assemble a working Java example but there are bit and pieces of info 
>> around.
>>
>> 2.] I am using Avro 1.7.7 with Flink for state entities since versions 
>> belong Avro 1.7.7 add a serialVersionUid and then I cannot replace the class 
>> with a new Avro schema seamlessly. If I update the Avro schema, and the Avro 
>> Maven plugin runs, a new class with a new serialVersionUid is created and 
>> that cannot be replaced in the state with the Java exception stating that 
>> local and state copies are different.  Any example would be really 
>> appreciated.
>>
>> Thanks,
>> Sandip
>
>

Re: Flink State Processor API Example - Java

[Sandeep khanzode]

Hi Guowei,

I followed the document, but somehow, I am unable to get a working Java example 
for Avro state.

So, I tried to simply use the Java SpecificRecords created by Avro Maven Plugin 
and inject. Now, that works correctly, but I use Avro 1.7.7 since it is the 
last version that I saw which does not put a serialVersionUid in the generated 
SpecificRecord. 

How can I use a Avro SpecificRecord generated by 1.8.0 if it use a 
serialVersionUid because the moment I change something in the Avro schema and 
regenerate the SpecificRecord, I will get a class incompatible error when Flink 
tried to deserialise.


Thanks,
Sandip



> On 25-Jun-2021, at 10:25 AM, Guowei Ma  wrote:
> 
> Hi Sandeep
> 
> What I understand is that you want to manipulate the state. So I think you 
> could use the old schema to read the state first, and then write it to a new 
> schema, instead of using a new schema to read an old schema format data.
> In addition, I would like to ask, if you want to do "State Schema Evolution" 
> ? Flink currently supports avro+pojo's schema evolution[1], and you don't 
> need to do this manually.
> 
> [1] 
> https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#supported-data-types-for-schema-evolution
>  
> 
> 
> Best,
> Guowei
> 
> 
> On Fri, Jun 25, 2021 at 3:04 AM Sandeep khanzode  > wrote:
> Hello,
> 
> 1.] Can someone please share a working example of how to read 
> ValueState and MapState from a checkpoint and update it? I tried 
> to assemble a working Java example but there are bit and pieces of info 
> around. 
> 
> 2.] I am using Avro 1.7.7 with Flink for state entities since versions belong 
> Avro 1.7.7 add a serialVersionUid and then I cannot replace the class with a 
> new Avro schema seamlessly. If I update the Avro schema, and the Avro Maven 
> plugin runs, a new class with a new serialVersionUid is created and that 
> cannot be replaced in the state with the Java exception stating that local 
> and state copies are different.  Any example would be really appreciated.
> 
> Thanks,
> Sandip

Re: Flink State Processor API Example - Java

[Guowei Ma]

Hi Sandeep

What I understand is that you want to manipulate the state. So I think you
could use the old schema to read the state first, and then write it to a
new schema, instead of using a new schema to read an old schema format data.
In addition, I would like to ask, if you want to do "State Schema
Evolution" ? Flink currently supports avro+pojo's schema evolution[1], and
you don't need to do this manually.

[1]
https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/serialization/schema_evolution/#supported-data-types-for-schema-evolution

Best,
Guowei


On Fri, Jun 25, 2021 at 3:04 AM Sandeep khanzode 
wrote:

> Hello,
>
> 1.] Can someone please share a working example of how to read
> ValueState and MapState from a checkpoint and update it? I
> tried to assemble a working Java example but there are bit and pieces of
> info around.
>
> 2.] I am using Avro 1.7.7 with Flink for state entities since versions
> belong Avro 1.7.7 add a serialVersionUid and then I cannot replace the
> class with a new Avro schema seamlessly. If I update the Avro schema, and
> the Avro Maven plugin runs, a new class with a new serialVersionUid is
> created and that cannot be replaced in the state with the Java exception
> stating that local and state copies are different.  Any example would be
> really appreciated.
>
> Thanks,
> Sandip

Flink State Processor API Example - Java

[Sandeep khanzode]

Hello,

1.] Can someone please share a working example of how to read ValueState 
and MapState from a checkpoint and update it? I tried to assemble a 
working Java example but there are bit and pieces of info around. 

2.] I am using Avro 1.7.7 with Flink for state entities since versions belong 
Avro 1.7.7 add a serialVersionUid and then I cannot replace the class with a 
new Avro schema seamlessly. If I update the Avro schema, and the Avro Maven 
plugin runs, a new class with a new serialVersionUid is created and that cannot 
be replaced in the state with the Java exception stating that local and state 
copies are different.  Any example would be really appreciated.

Thanks,
Sandip