Can we get a link to the Flink discussion? On Mon, 11 Sept 2023 at 15:25, Matthew de Detrich <[email protected]> wrote: > > > Flink is using pekko now, and a hard reboot may not be ok in some cases, > and because Pekko ships a Netty3 with many CVEs, they may decide to drop > pekko and come out with a dedicated rpc implementation based on Netty4 too. > > Wasn't aware of this (I thought Flink used the new remoting protocol). > I think this is a strong enough reason by itself to upgrade Netty4, but > ofcourse > deprecated classic remoting. > > On Mon, Sep 11, 2023 at 3:22 PM kerr <[email protected]> wrote: > > > Netty 4 is much better than Netty 3 and we can schedule tests to run for it > > and make sure the tests all passed for servertimes. > > Otherwise, we will have a CVEs old netty 3 in classpath. > > > > Flink is using pekko now, and a hard reboot may not be ok in some cases, > > and because Pekko ships a Netty3 with many CVEs, they may decide to drop > > pekko and come out with a dedicated rpc implementation based on Netty4 too. > > But after we migrate to Netty 4, they can defer that, I will ask around the > > PMC of Flink at here to let them taking a look. > > > > 何品 > > > > > > Matthew de Detrich <[email protected]> 于2023年9月11日周一 > > 19:35写道： > > > > > > that this is a bug, not a feature > > > > > > Also meant to say that this is a feature, not a bug > > > > > > On Mon, Sep 11, 2023 at 1:34 PM Matthew de Detrich < > > > [email protected]> wrote: > > > > > > > While I know that people have reservations for keeping the current > > > > classical transport, > > > > I just want to relay what I have written at > > > > https://github.com/apache/incubator-pekko/pull/643 > > > > i.e. > > > > > > > > - Upgrading the classical transport from Netty 3 to Netty 4 was in > > > context > > > > quite trivial > > > > - Dropping classical transport means that we are technically breaking > > > > semver (we can only do this in Pekko 2.0.x) > > > > - Not upgrading to Netty3 means that we are shipping code with known > > > CVE's > > > > > > > > For these reasons I personally would prefer to merge the PR and just > > get > > > > it done with. Even > > > > with the PR merged, if we change our minds later nothing is stopping us > > > > from dropping it > > > > later (or even reverting the PR when testing is done but I doubt that > > > will > > > > occur). > > > > > > > > There is of course a risk in that updating to Netty 4 might create some > > > > regressions > > > > and hence pekko users would complain but I would then argue that this > > is > > > a > > > > bug, > > > > not a feature because we then get actual feedback as to how many people > > > > are using > > > > the classical transport which allows us to make more comprehensive > > > > decisions in > > > > the future. > > > > > > > > On Fri, Sep 8, 2023 at 5:56 PM kerr <[email protected]> wrote: > > > > > > > >> And I gathered this result on reddit > > > >> [image: image.png] > > > >> 何品 > > > >> > > > >> > > > >> kerr <[email protected]> 于2023年9月8日周五 23:51写道： > > > >> > > > >>> I just read > > > >>> > > > >>> To switch a full cluster restart is required and any overrides for > > > >>> classic remoting need to be ported to Artery configuration. Artery > > has > > > a > > > >>> completely different protocol, which means that a rolling update is > > not > > > >>> supported. > > > >>> So for Flink use case is that ok? > > > >>> 何品 > > > >>> > > > >>> > > > >>> kerr <[email protected]> 于2023年9月8日周五 17:10写道： > > > >>> > > > >>>> I did a poll on reddit and get the result of > > > >>>> [image: image.png] > > > >>>> 何品 > > > >>>> > > > >>>> > > > >>>> kerr <[email protected]> 于2023年8月6日周日 00:33写道： > > > >>>> > > > >>>>> > Is it really worth spending any time to support classic remoting? > > > >>>>> Users > > > >>>>> would be better served to switch to Artery. > > > >>>>> > > > >>>>> I'm one of the user and the China Scala User group may not talk > > much > > > >>>>> on web but most of us is using the netty transport, even Lightbend > > > say the > > > >>>>> `Artery is the best`. > > > >>>>> > > > >>>>> Another online product with large scale nodes(here) is running with > > > >>>>> Akka 2.4.x with the classic transport, and Flink is using the > > > netty.tpc > > > >>>>> transport at least for now too. > > > >>>>> > > > >>>>> I think the community can choose when they migrate the nodes to > > > artery > > > >>>>> is essential, especially when otherwise your cluster need a hard > > > reboot. > > > >>>>> > > > >>>>> AFAIK, the Artery transport is not wire compatible with the > > > >>>>> classic one. > > > >>>>> > > > >>>>> I suggest: > > > >>>>> 1. Start to send PRs to open sources projects like Flink with the > > > help > > > >>>>> for migrating to the artery transport. > > > >>>>> 2. Upgrade to the new version of Netty for bugfix and > > > >>>>> better performance. > > > >>>>> 3. Suggest all the new community move to the Artery transport. > > > >>>>> > > > >>>>> Even Akka was removed it in Akka 2.8.x not Akka 2.7.x. > > > >>>>> > > > >>>>> Anyway, different company has different requirements, that just my > > 2 > > > >>>>> cents. > > > >>>>> > > > >>>>> 何品 > > > >>>>> > > > >>>>> > > > >>>>> PJ Fanning <[email protected]> 于2023年8月2日周三 17:29写道： > > > >>>>> > > > >>>>>> This will end up in a vote. > > > >>>>>> > > > >>>>>> For me, we shipped 1.0.x. Users who don't want big changes can use > > > >>>>>> that version. We can agree to support 1.0.x with critical fixes. > > > >>>>>> > > > >>>>>> We can now start improving the code for the improvement release > > > >>>>>> (v1.1.x or v2.0.x). Improvements include getting rid of deprecated > > > >>>>>> code and most of us appear to believe that classic remoting is > > long > > > >>>>>> deprecated. > > > >>>>>> > > > >>>>>> > > > >>>>>> On Wed, 2 Aug 2023 at 09:51, Matthew de Detrich > > > >>>>>> <[email protected]> wrote: > > > >>>>>> > > > > >>>>>> > > Is it really worth spending any time to support classic > > > remoting? > > > >>>>>> Users > > > >>>>>> > would be better served to switch to Artery. > > > >>>>>> > > > > >>>>>> > That depends on what you mean by "support". If agreed on, I > > would > > > >>>>>> > propose marking classic remoting as "to be removed" which would > > > >>>>>> > also mean that the code wouldn't be touched (i.e. no feature > > > >>>>>> > changes and no bug fixes unless extremely critical). > > > >>>>>> > > > > >>>>>> > I am in no way suggesting that we support remoting in a > > > traditional > > > >>>>>> sense > > > >>>>>> > and again the only reason why I am even contemplating this is > > > >>>>>> because > > > >>>>>> > of the CVE's. > > > >>>>>> > > > > >>>>>> > > I don't think it is likely that a significant portion of Akka > > > 2.4 > > > >>>>>> users > > > >>>>>> > would update to Pekko any time soon if the couldn't update to > > Akka > > > >>>>>> 2.5/2.6. > > > >>>>>> > > > > >>>>>> > I also had this impression but it's already come up a few times > > > that > > > >>>>>> > people still use Akka 2.4.x. They likely haven't bothered > > updating > > > >>>>>> > because they didn't see a need to (quite famously Akka is quite > > > >>>>>> stable > > > >>>>>> > and there are cases of companies using it for years without > > > needing > > > >>>>>> to > > > >>>>>> > update, as stated by Lightbend's CEO) but with the license > > change > > > >>>>>> > it created a catalyst/trigger for some users to consider moving > > to > > > >>>>>> Pekko. > > > >>>>>> > > > > >>>>>> > On this note I would also be wary in making assumptions on what > > > >>>>>> versions > > > >>>>>> > of Akka people happen to be using. As a corollary in the > > > discussion > > > >>>>>> on > > > >>>>>> > dropping JDK 8, when we suggested that we quite quickly got > > > >>>>>> feedback that > > > >>>>>> > there are people still using JDK 8. And while I know all of the > > > >>>>>> arguments > > > >>>>>> > that you really should not be using JDK 8 can be carried over to > > > >>>>>> Akka. > > > >>>>>> > > > > >>>>>> > Admittedly it's quite hard to get good feedback on what versions > > > >>>>>> people are > > > >>>>>> > using, just saying we should be a bit more careful in making > > these > > > >>>>>> > assumptions. > > > >>>>>> > After all, people are still using JDK 8 ;) > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > On Wed, Aug 2, 2023 at 10:19 AM Nicolas Vollmar < > > > [email protected]> > > > >>>>>> wrote: > > > >>>>>> > > > > >>>>>> > > Akka 2.4 has been EoL since end of 2017. > > > >>>>>> > > I don't think it is likely that a significant portion of Akka > > > 2.4 > > > >>>>>> users > > > >>>>>> > > would update to Pekko any time soon if the couldn't update to > > > >>>>>> Akka 2.5/2.6. > > > >>>>>> > > > > > >>>>>> > > Is it really worth spending any time to support classic > > > remoting? > > > >>>>>> Users > > > >>>>>> > > would be better served to switch to Artery. > > > >>>>>> > > > > > >>>>>> > > On Wed, 2 Aug 2023 at 10:04, Matthew de Detrich > > > >>>>>> > > <[email protected]> wrote: > > > >>>>>> > > > > > >>>>>> > > > > If you manage to move to Pekko and were still using > > classic > > > >>>>>> remoting, > > > >>>>>> > > > it's likely you are still on Akka 2.4 or 2.5. If you manage > > > the > > > >>>>>> update > > > >>>>>> > > > to Pekko, going to Artery TCP is a small step. > > > >>>>>> > > > > > > >>>>>> > > > One of the reasons why I was saying this is that I was under > > > the > > > >>>>>> > > impression > > > >>>>>> > > > that there are a non-trivial amount of users still on Akka > > 2.4 > > > >>>>>> (not sure > > > >>>>>> > > > why > > > >>>>>> > > > this is the case but kerr was telling me this). > > > >>>>>> > > > > > > >>>>>> > > > > If people are interested in keeping it around and also opt > > > in > > > >>>>>> to > > > >>>>>> > > > maintaining it (which in the first place means, making the > > > >>>>>> tests for > > > >>>>>> > > > it work *now*), than fine, but otherwise, being able to use > > it > > > >>>>>> on > > > >>>>>> > > > 1.0.x is already a big benefit. > > > >>>>>> > > > > > > >>>>>> > > > Since classic remoting is currently using Netty 3 which has > > > >>>>>> CVE's > > > >>>>>> > > > I don't think it's wise to encourage people that still > > happen > > > >>>>>> to be > > > >>>>>> > > > using classic remoting to stay on 1.0.x. If it wasn't for > > > final > > > >>>>>> > > > version of netty 3 having CVE's then I would have no qualms > > > >>>>>> > > > for dropping it in the 1.1.x series (in fact if there was a > > > >>>>>> > > > hypothetical netty 3 without CVE's we would have already > > > >>>>>> > > > updated to it in the 1.0.x series and dropped classic > > remoting > > > >>>>>> in > > > >>>>>> > > > 1.1.x without a heartbeat). > > > >>>>>> > > > > > > >>>>>> > > > I would propose that if we do accept keeping classic > > remoting > > > >>>>>> > > > and updating to Netty 4, we would mark the feature to be > > > >>>>>> > > > dropped in 1.2.x and aside from that Netty 4 upgrade that > > > >>>>>> > > > part of code would be untouched unless there are critical > > > >>>>>> > > > bugs/regressions. > > > >>>>>> > > > > > > >>>>>> > > > On Tue, Aug 1, 2023 at 1:38 PM Johannes Rudolph < > > > >>>>>> > > > [email protected]> > > > >>>>>> > > > wrote: > > > >>>>>> > > > > > > >>>>>> > > > > We definitely should remove it for 1.1.x. There's no > > > >>>>>> technical reason > > > >>>>>> > > > > to keep it because the newer artery TCP transport just > > > >>>>>> supersedes it. > > > >>>>>> > > > > As discussed before, the problem with keeping old features > > > >>>>>> around is > > > >>>>>> > > > > "death by a thousand cuts". The classic remoting backend > > is > > > >>>>>> the prime > > > >>>>>> > > > > example of it because it is basically unmaintained for > > years > > > >>>>>> and > > > >>>>>> > > > > there's basically no expert knowledge available about how > > to > > > >>>>>> maintain > > > >>>>>> > > > > it or how to fix the existing bugs. Testing remoting > > > backends > > > >>>>>> has > > > >>>>>> > > > > shown to be very complicated and often unreliable and > > being > > > >>>>>> able to > > > >>>>>> > > > > remove this particular component from the test matrix will > > > be > > > >>>>>> a huge > > > >>>>>> > > > > improvement. > > > >>>>>> > > > > > > > >>>>>> > > > > On Mon, Jul 31, 2023 at 11:28 PM Matthew de Detrich > > > >>>>>> > > > > <[email protected]> wrote: > > > >>>>>> > > > > > I would prefer adding in netty 4 support for the classic > > > >>>>>> transport > > > >>>>>> > > > > > mechanism because it gives > > > >>>>>> > > > > > people an upgrade path while allowing them to resolve > > the > > > >>>>>> CVE issues. > > > >>>>>> > > > If > > > >>>>>> > > > > > Pekko 1.1.x didn't > > > >>>>>> > > > > > support the classic transport mechanism then it forces > > > >>>>>> Pekko users to > > > >>>>>> > > > be > > > >>>>>> > > > > > stuck on 1.0.x > > > >>>>>> > > > > > with no easy upgrade path. > > > >>>>>> > > > > > > > >>>>>> > > > > We are not responsible for giving people an easy upgrade > > > path > > > >>>>>> when > > > >>>>>> > > > > they were stuck on features like classic remoting which > > has > > > >>>>>> basically > > > >>>>>> > > > > been outdated for years. It is very important that, going > > > >>>>>> forward, > > > >>>>>> > > > > that we focus our attention on the most important features > > > >>>>>> and make > > > >>>>>> > > > > sure to not get stuck in maintaining old parts that we > > > cannot > > > >>>>>> (or > > > >>>>>> > > > > don't want to) care for. > > > >>>>>> > > > > > > > >>>>>> > > > > If people are interested in keeping it around and also opt > > > in > > > >>>>>> to > > > >>>>>> > > > > maintaining it (which in the first place means, making the > > > >>>>>> tests for > > > >>>>>> > > > > it work *now*), than fine, but otherwise, being able to > > use > > > >>>>>> it on > > > >>>>>> > > > > 1.0.x is already a big benefit. > > > >>>>>> > > > > > > > >>>>>> > > > > Apart from that the update path is pretty clear: > > > >>>>>> > > > > * move to Pekko 1.0.x > > > >>>>>> > > > > * move to Arterty TCP > > > >>>>>> > > > > * move to Pekko 1.1.x > > > >>>>>> > > > > > > > >>>>>> > > > > If you manage to move to Pekko and were still using > > classic > > > >>>>>> remoting, > > > >>>>>> > > > > it's likely you are still on Akka 2.4 or 2.5. If you > > manage > > > >>>>>> the update > > > >>>>>> > > > > to Pekko, going to Artery TCP is a small step. > > > >>>>>> > > > > > > > >>>>>> > > > > Johannes > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > --------------------------------------------------------------------- > > > >>>>>> > > > > To unsubscribe, e-mail: [email protected] > > > >>>>>> > > > > For additional commands, e-mail: > > [email protected] > > > >>>>>> > > > > > > > >>>>>> > > > > > > > >>>>>> > > > > > > >>>>>> > > > -- > > > >>>>>> > > > > > > >>>>>> > > > Matthew de Detrich > > > >>>>>> > > > > > > >>>>>> > > > *Aiven Deutschland GmbH* > > > >>>>>> > > > > > > >>>>>> > > > Immanuelkirchstraße 26, 10405 Berlin > > > >>>>>> > > > > > > >>>>>> > > > Amtsgericht Charlottenburg, HRB 209739 B > > > >>>>>> > > > > > > >>>>>> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > >>>>>> > > > > > > >>>>>> > > > *m:* +491603708037 > > > >>>>>> > > > > > > >>>>>> > > > *w:* aiven.io *e:* [email protected] > > > >>>>>> > > > > > > >>>>>> > > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> > -- > > > >>>>>> > > > > >>>>>> > Matthew de Detrich > > > >>>>>> > > > > >>>>>> > *Aiven Deutschland GmbH* > > > >>>>>> > > > > >>>>>> > Immanuelkirchstraße 26, 10405 Berlin > > > >>>>>> > > > > >>>>>> > Amtsgericht Charlottenburg, HRB 209739 B > > > >>>>>> > > > > >>>>>> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > >>>>>> > > > > >>>>>> > *m:* +491603708037 > > > >>>>>> > > > > >>>>>> > *w:* aiven.io *e:* [email protected] > > > >>>>>> > > > >>>>>> > > > --------------------------------------------------------------------- > > > >>>>>> To unsubscribe, e-mail: [email protected] > > > >>>>>> For additional commands, e-mail: [email protected] > > > >>>>>> > > > >>>>>> > > > > > > > > -- > > > > > > > > Matthew de Detrich > > > > > > > > *Aiven Deutschland GmbH* > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > *m:* +491603708037 > > > > > > > > *w:* aiven.io *e:* [email protected] > > > > > > > > > > > > > -- > > > > > > Matthew de Detrich > > > > > > *Aiven Deutschland GmbH* > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > *m:* +491603708037 > > > > > > *w:* aiven.io *e:* [email protected] > > > > > > > > -- > > Matthew de Detrich > > *Aiven Deutschland GmbH* > > Immanuelkirchstraße 26, 10405 Berlin > > Amtsgericht Charlottenburg, HRB 209739 B > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > *m:* +491603708037 > > *w:* aiven.io *e:* [email protected]
--------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
