> Flink is using pekko now, and a hard reboot may not be ok in some cases, and because Pekko ships a Netty3 with many CVEs, they may decide to drop pekko and come out with a dedicated rpc implementation based on Netty4 too.
Wasn't aware of this (I thought Flink used the new remoting protocol). I think this is a strong enough reason by itself to upgrade Netty4, but ofcourse deprecated classic remoting. On Mon, Sep 11, 2023 at 3:22 PM kerr <[email protected]> wrote: > Netty 4 is much better than Netty 3 and we can schedule tests to run for it > and make sure the tests all passed for servertimes. > Otherwise, we will have a CVEs old netty 3 in classpath. > > Flink is using pekko now, and a hard reboot may not be ok in some cases, > and because Pekko ships a Netty3 with many CVEs, they may decide to drop > pekko and come out with a dedicated rpc implementation based on Netty4 too. > But after we migrate to Netty 4, they can defer that, I will ask around the > PMC of Flink at here to let them taking a look. > > 何品 > > > Matthew de Detrich <[email protected]> 于2023年9月11日周一 > 19:35写道： > > > > that this is a bug, not a feature > > > > Also meant to say that this is a feature, not a bug > > > > On Mon, Sep 11, 2023 at 1:34 PM Matthew de Detrich < > > [email protected]> wrote: > > > > > While I know that people have reservations for keeping the current > > > classical transport, > > > I just want to relay what I have written at > > > https://github.com/apache/incubator-pekko/pull/643 > > > i.e. > > > > > > - Upgrading the classical transport from Netty 3 to Netty 4 was in > > context > > > quite trivial > > > - Dropping classical transport means that we are technically breaking > > > semver (we can only do this in Pekko 2.0.x) > > > - Not upgrading to Netty3 means that we are shipping code with known > > CVE's > > > > > > For these reasons I personally would prefer to merge the PR and just > get > > > it done with. Even > > > with the PR merged, if we change our minds later nothing is stopping us > > > from dropping it > > > later (or even reverting the PR when testing is done but I doubt that > > will > > > occur). > > > > > > There is of course a risk in that updating to Netty 4 might create some > > > regressions > > > and hence pekko users would complain but I would then argue that this > is > > a > > > bug, > > > not a feature because we then get actual feedback as to how many people > > > are using > > > the classical transport which allows us to make more comprehensive > > > decisions in > > > the future. > > > > > > On Fri, Sep 8, 2023 at 5:56 PM kerr <[email protected]> wrote: > > > > > >> And I gathered this result on reddit > > >> [image: image.png] > > >> 何品 > > >> > > >> > > >> kerr <[email protected]> 于2023年9月8日周五 23:51写道： > > >> > > >>> I just read > > >>> > > >>> To switch a full cluster restart is required and any overrides for > > >>> classic remoting need to be ported to Artery configuration. Artery > has > > a > > >>> completely different protocol, which means that a rolling update is > not > > >>> supported. > > >>> So for Flink use case is that ok? > > >>> 何品 > > >>> > > >>> > > >>> kerr <[email protected]> 于2023年9月8日周五 17:10写道： > > >>> > > >>>> I did a poll on reddit and get the result of > > >>>> [image: image.png] > > >>>> 何品 > > >>>> > > >>>> > > >>>> kerr <[email protected]> 于2023年8月6日周日 00:33写道： > > >>>> > > >>>>> > Is it really worth spending any time to support classic remoting? > > >>>>> Users > > >>>>> would be better served to switch to Artery. > > >>>>> > > >>>>> I'm one of the user and the China Scala User group may not talk > much > > >>>>> on web but most of us is using the netty transport, even Lightbend > > say the > > >>>>> `Artery is the best`. > > >>>>> > > >>>>> Another online product with large scale nodes(here) is running with > > >>>>> Akka 2.4.x with the classic transport, and Flink is using the > > netty.tpc > > >>>>> transport at least for now too. > > >>>>> > > >>>>> I think the community can choose when they migrate the nodes to > > artery > > >>>>> is essential, especially when otherwise your cluster need a hard > > reboot. > > >>>>> > > >>>>> AFAIK, the Artery transport is not wire compatible with the > > >>>>> classic one. > > >>>>> > > >>>>> I suggest: > > >>>>> 1. Start to send PRs to open sources projects like Flink with the > > help > > >>>>> for migrating to the artery transport. > > >>>>> 2. Upgrade to the new version of Netty for bugfix and > > >>>>> better performance. > > >>>>> 3. Suggest all the new community move to the Artery transport. > > >>>>> > > >>>>> Even Akka was removed it in Akka 2.8.x not Akka 2.7.x. > > >>>>> > > >>>>> Anyway, different company has different requirements, that just my > 2 > > >>>>> cents. > > >>>>> > > >>>>> 何品 > > >>>>> > > >>>>> > > >>>>> PJ Fanning <[email protected]> 于2023年8月2日周三 17:29写道： > > >>>>> > > >>>>>> This will end up in a vote. > > >>>>>> > > >>>>>> For me, we shipped 1.0.x. Users who don't want big changes can use > > >>>>>> that version. We can agree to support 1.0.x with critical fixes. > > >>>>>> > > >>>>>> We can now start improving the code for the improvement release > > >>>>>> (v1.1.x or v2.0.x). Improvements include getting rid of deprecated > > >>>>>> code and most of us appear to believe that classic remoting is > long > > >>>>>> deprecated. > > >>>>>> > > >>>>>> > > >>>>>> On Wed, 2 Aug 2023 at 09:51, Matthew de Detrich > > >>>>>> <[email protected]> wrote: > > >>>>>> > > > >>>>>> > > Is it really worth spending any time to support classic > > remoting? > > >>>>>> Users > > >>>>>> > would be better served to switch to Artery. > > >>>>>> > > > >>>>>> > That depends on what you mean by "support". If agreed on, I > would > > >>>>>> > propose marking classic remoting as "to be removed" which would > > >>>>>> > also mean that the code wouldn't be touched (i.e. no feature > > >>>>>> > changes and no bug fixes unless extremely critical). > > >>>>>> > > > >>>>>> > I am in no way suggesting that we support remoting in a > > traditional > > >>>>>> sense > > >>>>>> > and again the only reason why I am even contemplating this is > > >>>>>> because > > >>>>>> > of the CVE's. > > >>>>>> > > > >>>>>> > > I don't think it is likely that a significant portion of Akka > > 2.4 > > >>>>>> users > > >>>>>> > would update to Pekko any time soon if the couldn't update to > Akka > > >>>>>> 2.5/2.6. > > >>>>>> > > > >>>>>> > I also had this impression but it's already come up a few times > > that > > >>>>>> > people still use Akka 2.4.x. They likely haven't bothered > updating > > >>>>>> > because they didn't see a need to (quite famously Akka is quite > > >>>>>> stable > > >>>>>> > and there are cases of companies using it for years without > > needing > > >>>>>> to > > >>>>>> > update, as stated by Lightbend's CEO) but with the license > change > > >>>>>> > it created a catalyst/trigger for some users to consider moving > to > > >>>>>> Pekko. > > >>>>>> > > > >>>>>> > On this note I would also be wary in making assumptions on what > > >>>>>> versions > > >>>>>> > of Akka people happen to be using. As a corollary in the > > discussion > > >>>>>> on > > >>>>>> > dropping JDK 8, when we suggested that we quite quickly got > > >>>>>> feedback that > > >>>>>> > there are people still using JDK 8. And while I know all of the > > >>>>>> arguments > > >>>>>> > that you really should not be using JDK 8 can be carried over to > > >>>>>> Akka. > > >>>>>> > > > >>>>>> > Admittedly it's quite hard to get good feedback on what versions > > >>>>>> people are > > >>>>>> > using, just saying we should be a bit more careful in making > these > > >>>>>> > assumptions. > > >>>>>> > After all, people are still using JDK 8 ;) > > >>>>>> > > > >>>>>> > > > >>>>>> > On Wed, Aug 2, 2023 at 10:19 AM Nicolas Vollmar < > > [email protected]> > > >>>>>> wrote: > > >>>>>> > > > >>>>>> > > Akka 2.4 has been EoL since end of 2017. > > >>>>>> > > I don't think it is likely that a significant portion of Akka > > 2.4 > > >>>>>> users > > >>>>>> > > would update to Pekko any time soon if the couldn't update to > > >>>>>> Akka 2.5/2.6. > > >>>>>> > > > > >>>>>> > > Is it really worth spending any time to support classic > > remoting? > > >>>>>> Users > > >>>>>> > > would be better served to switch to Artery. > > >>>>>> > > > > >>>>>> > > On Wed, 2 Aug 2023 at 10:04, Matthew de Detrich > > >>>>>> > > <[email protected]> wrote: > > >>>>>> > > > > >>>>>> > > > > If you manage to move to Pekko and were still using > classic > > >>>>>> remoting, > > >>>>>> > > > it's likely you are still on Akka 2.4 or 2.5. If you manage > > the > > >>>>>> update > > >>>>>> > > > to Pekko, going to Artery TCP is a small step. > > >>>>>> > > > > > >>>>>> > > > One of the reasons why I was saying this is that I was under > > the > > >>>>>> > > impression > > >>>>>> > > > that there are a non-trivial amount of users still on Akka > 2.4 > > >>>>>> (not sure > > >>>>>> > > > why > > >>>>>> > > > this is the case but kerr was telling me this). > > >>>>>> > > > > > >>>>>> > > > > If people are interested in keeping it around and also opt > > in > > >>>>>> to > > >>>>>> > > > maintaining it (which in the first place means, making the > > >>>>>> tests for > > >>>>>> > > > it work *now*), than fine, but otherwise, being able to use > it > > >>>>>> on > > >>>>>> > > > 1.0.x is already a big benefit. > > >>>>>> > > > > > >>>>>> > > > Since classic remoting is currently using Netty 3 which has > > >>>>>> CVE's > > >>>>>> > > > I don't think it's wise to encourage people that still > happen > > >>>>>> to be > > >>>>>> > > > using classic remoting to stay on 1.0.x. If it wasn't for > > final > > >>>>>> > > > version of netty 3 having CVE's then I would have no qualms > > >>>>>> > > > for dropping it in the 1.1.x series (in fact if there was a > > >>>>>> > > > hypothetical netty 3 without CVE's we would have already > > >>>>>> > > > updated to it in the 1.0.x series and dropped classic > remoting > > >>>>>> in > > >>>>>> > > > 1.1.x without a heartbeat). > > >>>>>> > > > > > >>>>>> > > > I would propose that if we do accept keeping classic > remoting > > >>>>>> > > > and updating to Netty 4, we would mark the feature to be > > >>>>>> > > > dropped in 1.2.x and aside from that Netty 4 upgrade that > > >>>>>> > > > part of code would be untouched unless there are critical > > >>>>>> > > > bugs/regressions. > > >>>>>> > > > > > >>>>>> > > > On Tue, Aug 1, 2023 at 1:38 PM Johannes Rudolph < > > >>>>>> > > > [email protected]> > > >>>>>> > > > wrote: > > >>>>>> > > > > > >>>>>> > > > > We definitely should remove it for 1.1.x. There's no > > >>>>>> technical reason > > >>>>>> > > > > to keep it because the newer artery TCP transport just > > >>>>>> supersedes it. > > >>>>>> > > > > As discussed before, the problem with keeping old features > > >>>>>> around is > > >>>>>> > > > > "death by a thousand cuts". The classic remoting backend > is > > >>>>>> the prime > > >>>>>> > > > > example of it because it is basically unmaintained for > years > > >>>>>> and > > >>>>>> > > > > there's basically no expert knowledge available about how > to > > >>>>>> maintain > > >>>>>> > > > > it or how to fix the existing bugs. Testing remoting > > backends > > >>>>>> has > > >>>>>> > > > > shown to be very complicated and often unreliable and > being > > >>>>>> able to > > >>>>>> > > > > remove this particular component from the test matrix will > > be > > >>>>>> a huge > > >>>>>> > > > > improvement. > > >>>>>> > > > > > > >>>>>> > > > > On Mon, Jul 31, 2023 at 11:28 PM Matthew de Detrich > > >>>>>> > > > > <[email protected]> wrote: > > >>>>>> > > > > > I would prefer adding in netty 4 support for the classic > > >>>>>> transport > > >>>>>> > > > > > mechanism because it gives > > >>>>>> > > > > > people an upgrade path while allowing them to resolve > the > > >>>>>> CVE issues. > > >>>>>> > > > If > > >>>>>> > > > > > Pekko 1.1.x didn't > > >>>>>> > > > > > support the classic transport mechanism then it forces > > >>>>>> Pekko users to > > >>>>>> > > > be > > >>>>>> > > > > > stuck on 1.0.x > > >>>>>> > > > > > with no easy upgrade path. > > >>>>>> > > > > > > >>>>>> > > > > We are not responsible for giving people an easy upgrade > > path > > >>>>>> when > > >>>>>> > > > > they were stuck on features like classic remoting which > has > > >>>>>> basically > > >>>>>> > > > > been outdated for years. It is very important that, going > > >>>>>> forward, > > >>>>>> > > > > that we focus our attention on the most important features > > >>>>>> and make > > >>>>>> > > > > sure to not get stuck in maintaining old parts that we > > cannot > > >>>>>> (or > > >>>>>> > > > > don't want to) care for. > > >>>>>> > > > > > > >>>>>> > > > > If people are interested in keeping it around and also opt > > in > > >>>>>> to > > >>>>>> > > > > maintaining it (which in the first place means, making the > > >>>>>> tests for > > >>>>>> > > > > it work *now*), than fine, but otherwise, being able to > use > > >>>>>> it on > > >>>>>> > > > > 1.0.x is already a big benefit. > > >>>>>> > > > > > > >>>>>> > > > > Apart from that the update path is pretty clear: > > >>>>>> > > > > * move to Pekko 1.0.x > > >>>>>> > > > > * move to Arterty TCP > > >>>>>> > > > > * move to Pekko 1.1.x > > >>>>>> > > > > > > >>>>>> > > > > If you manage to move to Pekko and were still using > classic > > >>>>>> remoting, > > >>>>>> > > > > it's likely you are still on Akka 2.4 or 2.5. If you > manage > > >>>>>> the update > > >>>>>> > > > > to Pekko, going to Artery TCP is a small step. > > >>>>>> > > > > > > >>>>>> > > > > Johannes > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > --------------------------------------------------------------------- > > >>>>>> > > > > To unsubscribe, e-mail: [email protected] > > >>>>>> > > > > For additional commands, e-mail: > [email protected] > > >>>>>> > > > > > > >>>>>> > > > > > > >>>>>> > > > > > >>>>>> > > > -- > > >>>>>> > > > > > >>>>>> > > > Matthew de Detrich > > >>>>>> > > > > > >>>>>> > > > *Aiven Deutschland GmbH* > > >>>>>> > > > > > >>>>>> > > > Immanuelkirchstraße 26, 10405 Berlin > > >>>>>> > > > > > >>>>>> > > > Amtsgericht Charlottenburg, HRB 209739 B > > >>>>>> > > > > > >>>>>> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >>>>>> > > > > > >>>>>> > > > *m:* +491603708037 > > >>>>>> > > > > > >>>>>> > > > *w:* aiven.io *e:* [email protected] > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > >>>>>> > > > >>>>>> > -- > > >>>>>> > > > >>>>>> > Matthew de Detrich > > >>>>>> > > > >>>>>> > *Aiven Deutschland GmbH* > > >>>>>> > > > >>>>>> > Immanuelkirchstraße 26, 10405 Berlin > > >>>>>> > > > >>>>>> > Amtsgericht Charlottenburg, HRB 209739 B > > >>>>>> > > > >>>>>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >>>>>> > > > >>>>>> > *m:* +491603708037 > > >>>>>> > > > >>>>>> > *w:* aiven.io *e:* [email protected] > > >>>>>> > > >>>>>> > > --------------------------------------------------------------------- > > >>>>>> To unsubscribe, e-mail: [email protected] > > >>>>>> For additional commands, e-mail: [email protected] > > >>>>>> > > >>>>>> > > > > > > -- > > > > > > Matthew de Detrich > > > > > > *Aiven Deutschland GmbH* > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > *m:* +491603708037 > > > > > > *w:* aiven.io *e:* [email protected] > > > > > > > > > -- > > > > Matthew de Detrich > > > > *Aiven Deutschland GmbH* > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > *m:* +491603708037 > > > > *w:* aiven.io *e:* [email protected] > > > -- Matthew de Detrich *Aiven Deutschland GmbH* Immanuelkirchstraße 26, 10405 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen *m:* +491603708037 *w:* aiven.io *e:* [email protected]
