Re: Update Netty version

[Alexey Demin]

Hi

I found ticket in jira: https://issues.apache.org/jira/browse/FLINK-3952

For update to latest 4.0 I can create separate ticket and pull request in
nearest 2 day. (I test in my env and all work correctly)

Update on 4.1 not so easy:
1) we use tv.cntt:netty-router:jar:1.10 for rest endpoints
2) netty-router mostly stoped development and have support only netty 4.0
3) latest versions say then they support 4.1 netty, but from my opinion
they under heavy refactoring and I can't recommend replace netty-router
1.10 on this version

we have 2 ways to resolve this problem:
1) remove dependency on netty-router
2) rebuild and repack netty-router 1.10 with new version on netty 4.1 (as
PoC I already did this work, and all work correctly, only small changes
required)

We already have custom akka build and build also netty-router under
discussion.

Thanks
Alexey Diomin



2017-06-08 0:13 GMT+04:00 Greg Hogan :

> Hi Alexey,
>
> Are you looking to create pull requests for upgrading Netty 4.0 and/or 4.1?
>
> Greg
>
> On Thu, May 18, 2017 at 4:41 AM, Alexey Demin  wrote:
>
> > Hi
> >
> > Problem not directly in flink, but it you use flink with beam then in
> > classpath you have original netty 4.0.27 from flink and netty 4.1.x from
> > beam (grpc use netty 4.1.x for communication).
> >
> > Other interest (specific for me now): netty have custom wrapper for
> openssl
> > library which have more productivity versus default jdk version, when you
> > have enabled ssl cluster communication it's will be very usefull give
> users
> > select implementation of ssl (default,openssl,boringssl).
> >
> > But netty have a lot of fixes for openssl/boringssl in latest versions
> and
> > more preferable do update netty as first step and enable select sslengine
> > as second step, not all in one step.
> >
> > > However, if we do the change at the beginning of the next release
> cycle,
> > we
> > > might have enough exposure time to verify whether things work or not.
> >
> > We just start 1.4 iteration and have time for testing.
> >
> > Thank,
> > Alexey Demin
> >
> >
> > 2017-05-18 11:48 GMT+04:00 Till Rohrmann :
> >
> > > Hi Alexey,
> > >
> > > thanks for looking into it. Are we currently facing any problems with
> > Netty
> > > 4.0.27 (bugs or performance)? I agree that in general we should try to
> > use
> > > the latest bug fix release. However, in the past we have seen that they
> > > might entail some slight behaviour changes which breaks things on the
> > Flink
> > > side. Since Netty is quite crucial for Flink, I would be extra careful
> > here
> > > when bumping versions, especially if there is no strong need for it.
> > >
> > > However, if we do the change at the beginning of the next release
> cycle,
> > we
> > > might have enough exposure time to verify whether things work or not.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, May 18, 2017 at 8:51 AM, Alexey Demin 
> > wrote:
> > >
> > > > Hi
> > > >
> > > > For now we use very old netty version.
> > > >
> > > > Netty 4.0.27.Final released on 02-Apr-15
> > > >
> > > > 
> > > >
> > > > If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> > > > custom
> > > > LengthFieldBasedCopyFrameDecoder which extend original
> > > > LengthFieldBasedFrameDecoder and override extractFrame for keep
> current
> > > > behavior.
> > > >
> > > > With this small changes we can update for last 4.0.x.
> > > >
> > > > For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> > > > KvStateServer.
> > > > Can we keep use original LengthFieldBasedFrameDecoder or must also
> > change
> > > > on LengthFieldBasedCopyFrameDecoder?
> > > >
> > > > If we want we can migrate on 4.1.
> > > > I already did tests and all work correctly, small changes in
> > > > NettyBufferPool.java and ChunkedByteBuf.java is required (implement
> new
> > > > method which added to interface).
> > > >
> > > >
> > > > Thanks
> > > > Alexey Diomin
> > > >
> > >
> >
>

Re: Update Netty version

[Greg Hogan]

Hi Alexey,

Are you looking to create pull requests for upgrading Netty 4.0 and/or 4.1?

Greg

On Thu, May 18, 2017 at 4:41 AM, Alexey Demin  wrote:

> Hi
>
> Problem not directly in flink, but it you use flink with beam then in
> classpath you have original netty 4.0.27 from flink and netty 4.1.x from
> beam (grpc use netty 4.1.x for communication).
>
> Other interest (specific for me now): netty have custom wrapper for openssl
> library which have more productivity versus default jdk version, when you
> have enabled ssl cluster communication it's will be very usefull give users
> select implementation of ssl (default,openssl,boringssl).
>
> But netty have a lot of fixes for openssl/boringssl in latest versions and
> more preferable do update netty as first step and enable select sslengine
> as second step, not all in one step.
>
> > However, if we do the change at the beginning of the next release cycle,
> we
> > might have enough exposure time to verify whether things work or not.
>
> We just start 1.4 iteration and have time for testing.
>
> Thank,
> Alexey Demin
>
>
> 2017-05-18 11:48 GMT+04:00 Till Rohrmann :
>
> > Hi Alexey,
> >
> > thanks for looking into it. Are we currently facing any problems with
> Netty
> > 4.0.27 (bugs or performance)? I agree that in general we should try to
> use
> > the latest bug fix release. However, in the past we have seen that they
> > might entail some slight behaviour changes which breaks things on the
> Flink
> > side. Since Netty is quite crucial for Flink, I would be extra careful
> here
> > when bumping versions, especially if there is no strong need for it.
> >
> > However, if we do the change at the beginning of the next release cycle,
> we
> > might have enough exposure time to verify whether things work or not.
> >
> > Cheers,
> > Till
> >
> > On Thu, May 18, 2017 at 8:51 AM, Alexey Demin 
> wrote:
> >
> > > Hi
> > >
> > > For now we use very old netty version.
> > >
> > > Netty 4.0.27.Final released on 02-Apr-15
> > >
> > > 
> > >
> > > If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> > > custom
> > > LengthFieldBasedCopyFrameDecoder which extend original
> > > LengthFieldBasedFrameDecoder and override extractFrame for keep current
> > > behavior.
> > >
> > > With this small changes we can update for last 4.0.x.
> > >
> > > For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> > > KvStateServer.
> > > Can we keep use original LengthFieldBasedFrameDecoder or must also
> change
> > > on LengthFieldBasedCopyFrameDecoder?
> > >
> > > If we want we can migrate on 4.1.
> > > I already did tests and all work correctly, small changes in
> > > NettyBufferPool.java and ChunkedByteBuf.java is required (implement new
> > > method which added to interface).
> > >
> > >
> > > Thanks
> > > Alexey Diomin
> > >
> >
>

Re: Update Netty version

[Alexey Demin]

Hi

Problem not directly in flink, but it you use flink with beam then in
classpath you have original netty 4.0.27 from flink and netty 4.1.x from
beam (grpc use netty 4.1.x for communication).

Other interest (specific for me now): netty have custom wrapper for openssl
library which have more productivity versus default jdk version, when you
have enabled ssl cluster communication it's will be very usefull give users
select implementation of ssl (default,openssl,boringssl).

But netty have a lot of fixes for openssl/boringssl in latest versions and
more preferable do update netty as first step and enable select sslengine
as second step, not all in one step.

> However, if we do the change at the beginning of the next release cycle,
we
> might have enough exposure time to verify whether things work or not.

We just start 1.4 iteration and have time for testing.

Thank,
Alexey Demin


2017-05-18 11:48 GMT+04:00 Till Rohrmann :

> Hi Alexey,
>
> thanks for looking into it. Are we currently facing any problems with Netty
> 4.0.27 (bugs or performance)? I agree that in general we should try to use
> the latest bug fix release. However, in the past we have seen that they
> might entail some slight behaviour changes which breaks things on the Flink
> side. Since Netty is quite crucial for Flink, I would be extra careful here
> when bumping versions, especially if there is no strong need for it.
>
> However, if we do the change at the beginning of the next release cycle, we
> might have enough exposure time to verify whether things work or not.
>
> Cheers,
> Till
>
> On Thu, May 18, 2017 at 8:51 AM, Alexey Demin  wrote:
>
> > Hi
> >
> > For now we use very old netty version.
> >
> > Netty 4.0.27.Final released on 02-Apr-15
> >
> > 
> >
> > If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> > custom
> > LengthFieldBasedCopyFrameDecoder which extend original
> > LengthFieldBasedFrameDecoder and override extractFrame for keep current
> > behavior.
> >
> > With this small changes we can update for last 4.0.x.
> >
> > For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> > KvStateServer.
> > Can we keep use original LengthFieldBasedFrameDecoder or must also change
> > on LengthFieldBasedCopyFrameDecoder?
> >
> > If we want we can migrate on 4.1.
> > I already did tests and all work correctly, small changes in
> > NettyBufferPool.java and ChunkedByteBuf.java is required (implement new
> > method which added to interface).
> >
> >
> > Thanks
> > Alexey Diomin
> >
>

Re: Update Netty version

[Till Rohrmann]

Hi Alexey,

thanks for looking into it. Are we currently facing any problems with Netty
4.0.27 (bugs or performance)? I agree that in general we should try to use
the latest bug fix release. However, in the past we have seen that they
might entail some slight behaviour changes which breaks things on the Flink
side. Since Netty is quite crucial for Flink, I would be extra careful here
when bumping versions, especially if there is no strong need for it.

However, if we do the change at the beginning of the next release cycle, we
might have enough exposure time to verify whether things work or not.

Cheers,
Till

On Thu, May 18, 2017 at 8:51 AM, Alexey Demin  wrote:

> Hi
>
> For now we use very old netty version.
>
> Netty 4.0.27.Final released on 02-Apr-15
>
> 
>
> If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> custom
> LengthFieldBasedCopyFrameDecoder which extend original
> LengthFieldBasedFrameDecoder and override extractFrame for keep current
> behavior.
>
> With this small changes we can update for last 4.0.x.
>
> For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> KvStateServer.
> Can we keep use original LengthFieldBasedFrameDecoder or must also change
> on LengthFieldBasedCopyFrameDecoder?
>
> If we want we can migrate on 4.1.
> I already did tests and all work correctly, small changes in
> NettyBufferPool.java and ChunkedByteBuf.java is required (implement new
> method which added to interface).
>
>
> Thanks
> Alexey Diomin
>