Great news! Thanks On Thu, Apr 25, 2019, 2:59 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> wrote:
> Hi Cliff, > > Thanks for bringing this up again. > > I think it would make sense to at least move this forward be only > exclusively checking the schema for user keys in MapState, and allow value > schema evolution. > I'll comment on the JIRA about this, and also make it a blocker for 1.9.0 > to make sure it will be resolved by then. > > Concerning your question on GenericRecord: > The actual schema resolution is still performed using the Avro schema, so > you will still bump into the same issue. > > Best, > Gordon > > On Wed, Apr 24, 2019 at 7:45 PM Cliff Resnick <cre...@gmail.com> wrote: > >> Hi Gordon, >> >> I noticed there has been no movement on this issue and I'm wondering if I >> can find some way to work around this. >> My MapState value is a case class container of Avro-generated >> SpecificRecords. If one SpecificRecord changes I am stuck. >> >> From the issue It seems like the blocker is around evolving the MapState >> key type. That may be a nasty problem, but my key type is stable and will >> never change. What do you think the level of difficulty would be to add >> support for evolving only the value? >> >> Also, if I use GenericRecord instead of SpecificRecord will the need for >> schema evolution still be triggered? Or does it actually go down to the >> avro schema rather than just the class serialVersionUID? >> >> >> >> >> >> >> On Mon, Mar 18, 2019 at 1:10 AM Tzu-Li (Gordon) Tai <tzuli...@apache.org> >> wrote: >> >>> Hi Cliff, >>> >>> Thanks for bringing this up! >>> AFAIK, this wouldn't be a long-term blocker. I've just opened a JIRA to >>> track this [1]. >>> >>> As explained in the JIRA ticket, the main reason this is disallowed in >>> the initial support for state schema evolution was due to how migration was >>> implemented in the RocksDB state backend. >>> Technically speaking, enabling this in the future is definitely possible. >>> >>> Cheers, >>> Gordon >>> >>> [1] https://issues.apache.org/jira/browse/FLINK-11947 >>> >>> On Mon, Mar 18, 2019 at 11:20 AM Cliff Resnick <cre...@gmail.com> wrote: >>> >>>> After trying out state migration in 1.8 rc2 I ran into this hard stop >>>> below. The comment does not give an indication why rocksdb map state cannot >>>> be migrated, and I'm wondering what the status is, since we do need this >>>> functionality and would like to know if this is a long-term blocker or not. >>>> Anyone know? >>>> >>>> >>>> https://github.com/apache/flink/blob/953a5ffcbdae4115f7d525f310723cf8770779df/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L542 >>>> >>>
