The proposal reads quite reasonable! I do not have additional comments as long as the change can insure backward compatibility. And many thanks to Dawid for catching this!
Best Yuan On Thu, Jan 19, 2023 at 6:03 PM Piotr Nowojski <pnowoj...@apache.org> wrote: > Hi, > > The idea sounds like a nice improvement to complete the feature. I don't > have any comments on top of what has been written already above. > > Bets, > Piotrek > > czw., 19 sty 2023 o 09:57 Etienne Chauchot <echauc...@apache.org> > napisał(a): > > > Hi, > > > > In the future we could add new compression algorithms by simply > > extending /StreamCompressionDecorator/. For now there is only 2 > > extensions: /UncompressedStreamCompressionDecorator/ and > > /SnappyStreamCompressionDecorator/. > > > > But I agree, I'd stick to /SnappyStreamCompressionDecorator /which is in > > use for keyed state compression. > > > > I'll add a comment on the FLIP saying that. > > > > Best > > > > Etienne > > > > Le 18/01/2023 à 16:36, Dawid Wysakowicz a écrit : > > > For the compression formats I'd stick with what is supported for the > > > keyed state, which is either enable or disable compression. The format > > > used for keyed state backend compression is snappy. > > > > > > Best, > > > > > > Dawid > > > > > > On 18/01/2023 15:33, ConradJam wrote: > > >> Thanks for driver this flip,I think have some question about this flip > > >> > > >> - Can we select the compression format, for example？ such as > > >> snappy or > > >> lz0 > > >> - If compression format is supported, I think the metadata will > > >> probably > > >> have to include which compression format > > >> > > >> and I agree idea with Dawid > > >> > > >> Dawid Wysakowicz <dwysakow...@apache.org> 于2023年1月18日周三 20:55写道： > > >> > > >> > > >>> It makes sense from my side. > > >>> > > >>> Could you, just for completeness, extend it with the info what will > be > > >>> the compression unit? If understand it correctly you envision each > > >>> state > > >>> to be compressed separately. > > >>> > > >>> Best, > > >>> > > >>> Dawid > > >>> > > >>> On 17/01/2023 15:47, Etienne Chauchot wrote: > > >>>> Hi everyone, > > >>>> > > >>>> I just published a new FLIP to introduce operator state compression > > >>>> (1). This feature is pretty straightforward to implement, so a PR is > > >>>> already under review (2) but it appeared during the review that to > be > > >>>> able to insure backward compatibility of the operator state > snapshots, > > >>>> we needed to modify the snapshot format hence the FLIP. > > >>>> > > >>>> Please tell me if you have any thoughts about it. > > >>>> > > >>>> > > >>>> [1] > > >>>> > > >>> > > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-290+Operator+state+compression > > >>> > > >>>> > > >>>> [2] https://github.com/apache/flink/pull/21636 > > >>>> > > >>>> > > >>>> Best, > > >>>> > > >>>> Etienne > > >>>> > > >> >
