> On 31 Mar 2022, at 18:02, Robert Bradshaw <[email protected]> wrote: > > Generally makes sense to me, though I'm curious what the maintenance > burden is *high or low) in keeping it around.
Well, we need to provide two versions of spark runner artifacts, job-servers and docker images, to test them separately (different Jenkins jobs). We also have two different code paths for the cases where API is not compatible between Spark2 and Spark3. > We should probably > deprecate it for a period of time before removing support. Agree and I’d suggest even ask users on user@/twitter before. Actually, I see some problem with naming. By default, we used to call “Spark runner” as a runner that works with Spark2 (for example, the artifacts [1][2]). When Spark3 support was added, all its Beam artifacts and related names reflect its version [3][4]. So, it’s not clear how it will be better to deal with this, especially, taking into account, that new Spark version (4, 5, etc) will be available sooner or later. Perhaps, to avoid a confusion in the future, we need to follow the same naming pattern. — Alexey [1] https://search.maven.org/artifact/org.apache.beam/beam-runners-spark [2] https://search.maven.org/artifact/org.apache.beam/beam-runners-spark-job-server [3] https://search.maven.org/artifact/org.apache.beam/beam-runners-spark-3 [4] https://search.maven.org/artifact/org.apache.beam/beam-runners-spark-3-job-server > > On Thu, Mar 31, 2022 at 8:52 AM Alexey Romanenko > <[email protected]> wrote: >> >> Hi everyone, >> >> For the moment, Beam Spark Runner supports two versions of Spark - 2.x and >> 3.x. >> >> Taking into account the several things that: >> - almost all cloud providers already mostly moved to Spark 3.x as a main >> supported version; >> - the latest Spark 2.x release (Spark 2.4.8, maintenance release) was done >> almost a year ago; >> - Spark 3 is considered as a mainstream Spark version for development and >> bug fixing; >> - better to avoid the burden of maintenance (there are some >> incompatibilities between Spark 2 and 3) of two versions; >> >> I’d suggest to stop support Spark 2 for the Spark Runner in the one of the >> next Beam releases. >> >> What are your thoughts on this? Are there any principal objections or >> reasons for not doing this that I probably missed? >> >> — >> Alexey >> >>
