> On 30 Jun 2022, at 07:53, Moritz Mack <[email protected]> wrote: > > Thanks so much Luke! > Just to confirm, given beam-runners-spark has a much longer history, it looks > like the numbers are swapped.
Yes, I guess it’ a typo there. Spark3 runner artifacts got to be distributed starting from Beam 2.29.0 > Looks like this – unfortunately - sheds a different light on this thread… > thoughts? As I said privately, I believe that the deprecation of Spark2 runner should finally accelerate the process of migration to Spark3. Officially we still keep it for the next 3 Beam releases but we can extend this period if needed. — Alexey > > On 29.06.22, 17:21, "Luke Cwik" <[email protected] <mailto:[email protected]>> > wrote: > > Here are the download stats for the last month listed by version and count. > beam-runners-spark 2.29.0 16 2.30.0 22 2.31.0 22 2.32.0 16 2.33.0 18 2.34.0 > 17 2.35.0 47 2.36.0 160 2.37.0 17 2.38.0 114 2.39.0 23 All 472 > beam-runners-spark-3 > ZjQcmQRYFpfptBannerStart > This Message Is From an External Sender > This message came from outside your organization. > Exercise caution when opening attachments or clicking any links. > ZjQcmQRYFpfptBannerEnd > Here are the download stats for the last month listed by version and count. > > beam-runners-spark > 2.29.0 16 > 2.30.0 22 > 2.31.0 22 > 2.32.0 16 > 2.33.0 18 > 2.34.0 17 > 2.35.0 47 > 2.36.0 160 > 2.37.0 17 > 2.38.0 114 > 2.39.0 23 > All 472 > > beam-runners-spark-3 > 0.1.0-incubating 24 > 0.2.0-incubating 41 > 0.3.0-incubating 44 > 0.4.0 36 > 0.5.0 26 > 0.6.0 48 > 2.0.0 400 > 2.1.0 30 > 2.2.0 34 > 2.3.0 25 > 2.4.0 32 > 2.5.0 77 > 2.6.0 23 > 2.7.0 25 > 2.8.0 27 > 2.9.0 26 > 2.10.0 27 > 2.11.0 21 > 2.12.0 321 > 2.13.0 162 > 2.14.0 21 > 2.15.0 32 > 2.16.0 35 > 2.17.0 18 > 2.18.0 67 > 2.19.0 29 > 2.20.0 22 > 2.21.0 23 > 2.22.0 47 > 2.23.0 19 > 2.24.0 63 > 2.25.0 17 > 2.26.0 23 > 2.27.0 18 > 2.28.0 81 > 2.29.0 268 > 2.30.0 26 > 2.31.0 24 > 2.32.0 69 > 2.33.0 403 > 2.34.0 352 > 2.35.0 1543 > 2.36.0 50 > 2.37.0 19 > 2.38.0 420 > 2.39.0 86 > All 5224 > > On Wed, Jun 29, 2022 at 1:24 AM Moritz Mack <[email protected] > <mailto:[email protected]>> wrote: > Who could help pulling the latest Maven download stats for beam-runners-spark > and beam-runners-spark-3 for the last few Beam releases? > > Thanks so much! > / Moritz > > On 01.04.22, 16:54, "Moritz Mack" <[email protected] > <mailto:[email protected]>> wrote: > > I just started looking into the Spark runner code a bit to helpfully help > supporting it. Besides having to maintain (test!) twice the number of > artifacts, there’s also a significant negative impact on developer ergonomics > / productivity supporting > I just started looking into the Spark runner code a bit to helpfully help > supporting it. > Besides having to maintain (test!) twice the number of artifacts, there’s > also a significant negative impact on developer ergonomics / productivity > supporting multiple major versions (separate modules to deal with breaking > changes and all the trouble that comes with that). > > Thanks, Alexey, for opening the discussion. Certainly a big +1 from my side. > > / Moritz > > > > From: Alexey Romanenko <[email protected] > <mailto:[email protected]>> > Date: Thursday, 31. March 2022 at 18:51 > To: dev <[email protected] <mailto:[email protected]>> > Subject: Re: [PROPOSAL] Stop Spark2 support in Spark Runner > > !-------------------------------------------------------------------| > This Message Is From an External Sender > This message came from outside your organization. > Exercise caution when opening attachments or clicking any > links. > |-------------------------------------------------------------------! > > > > On 31 Mar 2022, at 18:02, Robert Bradshaw <[email protected] > > <mailto:[email protected]>> wrote: > > > > Generally makes sense to me, though I'm curious what the maintenance > > burden is *high or low) in keeping it around. > > Well, we need to provide two versions of spark runner artifacts, job-servers > and docker images, to test them separately (different Jenkins jobs). We also > have two different code paths for the cases where API is not compatible > between Spark2 and Spark3. > > > We should probably > > deprecate it for a period of time before removing support. > > Agree and I’d suggest even ask users on user@/twitter before. > > > Actually, I see some problem with naming. By default, we used to call “Spark > runner” as a runner that works with Spark2 (for example, the artifacts > [1][2]). When Spark3 support was added, all its Beam artifacts and related > names reflect its version [3][4]. So, it’s not clear how it will be better to > deal with this, especially, taking into account, that new Spark version (4, > 5, etc) will be available sooner or later. Perhaps, to avoid a confusion in > the future, we need to follow the same naming pattern. > > — > Alexey > > [1] > https://urldefense.com/v3/__https://search.maven.org/artifact/org.apache.beam/beam-runners-spark__;!!CiXD_PY!URNp5UzJrCpB9s1jH33QcFeeNp5f3S7yzes0A03mrqRxCP9P3ZJZy1_2l3mF5QcCHmGhxZl0fWKf9ZbVIoOAIg$ > > <https://urldefense.com/v3/__https:/search.maven.org/artifact/org.apache.beam/beam-runners-spark__;!!CiXD_PY!URNp5UzJrCpB9s1jH33QcFeeNp5f3S7yzes0A03mrqRxCP9P3ZJZy1_2l3mF5QcCHmGhxZl0fWKf9ZbVIoOAIg$> > > [2] > https://urldefense.com/v3/__https://search.maven.org/artifact/org.apache.beam/beam-runners-spark-job-server__;!!CiXD_PY!URNp5UzJrCpB9s1jH33QcFeeNp5f3S7yzes0A03mrqRxCP9P3ZJZy1_2l3mF5QcCHmGhxZl0fWKf9ZZb2YMCkw$ > > <https://urldefense.com/v3/__https:/search.maven.org/artifact/org.apache.beam/beam-runners-spark-job-server__;!!CiXD_PY!URNp5UzJrCpB9s1jH33QcFeeNp5f3S7yzes0A03mrqRxCP9P3ZJZy1_2l3mF5QcCHmGhxZl0fWKf9ZZb2YMCkw$> > > [3] > https://urldefense.com/v3/__https://search.maven.org/artifact/org.apache.beam/beam-runners-spark-3__;!!CiXD_PY!URNp5UzJrCpB9s1jH33QcFeeNp5f3S7yzes0A03mrqRxCP9P3ZJZy1_2l3mF5QcCHmGhxZl0fWKf9ZZYlOwKfg$ > > <https://urldefense.com/v3/__https:/search.maven.org/artifact/org.apache.beam/beam-runners-spark-3__;!!CiXD_PY!URNp5UzJrCpB9s1jH33QcFeeNp5f3S7yzes0A03mrqRxCP9P3ZJZy1_2l3mF5QcCHmGhxZl0fWKf9ZZYlOwKfg$> > > [4] > https://urldefense.com/v3/__https://search.maven.org/artifact/org.apache.beam/beam-runners-spark-3-job-server__;!!CiXD_PY!URNp5UzJrCpB9s1jH33QcFeeNp5f3S7yzes0A03mrqRxCP9P3ZJZy1_2l3mF5QcCHmGhxZl0fWKf9ZaC2NvaLw$ > > <https://urldefense.com/v3/__https:/search.maven.org/artifact/org.apache.beam/beam-runners-spark-3-job-server__;!!CiXD_PY!URNp5UzJrCpB9s1jH33QcFeeNp5f3S7yzes0A03mrqRxCP9P3ZJZy1_2l3mF5QcCHmGhxZl0fWKf9ZaC2NvaLw$> > > > > > > On Thu, Mar 31, 2022 at 8:52 AM Alexey Romanenko > > <[email protected] <mailto:[email protected]>> wrote: > >> > >> Hi everyone, > >> > >> For the moment, Beam Spark Runner supports two versions of Spark - 2.x and > >> 3.x. > >> > >> Taking into account the several things that: > >> - almost all cloud providers already mostly moved to Spark 3.x as a main > >> supported version; > >> - the latest Spark 2.x release (Spark 2.4.8, maintenance release) was done > >> almost a year ago; > >> - Spark 3 is considered as a mainstream Spark version for development and > >> bug fixing; > >> - better to avoid the burden of maintenance (there are some > >> incompatibilities between Spark 2 and 3) of two versions; > >> > >> I’d suggest to stop support Spark 2 for the Spark Runner in the one of the > >> next Beam releases. > >> > >> What are your thoughts on this? Are there any principal objections or > >> reasons for not doing this that I probably missed? > >> > >> — > >> Alexey > >> > >> > > As a recipient of an email from Talend, your contact personal data will be on > our systems. Please see our privacy notice. <https://www.talend.com/privacy/> > > > As a recipient of an email from Talend, your contact personal data will be on > our systems. Please see our privacy notice. <https://www.talend.com/privacy/> > > > As a recipient of an email from Talend, your contact personal data will be on > our systems. Please see our privacy notice. <https://www.talend.com/privacy/>
