I already added Spark 3.5.0 version to Beam Spark version tests [1] and I didn’t notice any regression.
The next Beam release (2.53.0) should be available in a couple on months, depending on release preparation process. — Alexey [1] https://github.com/apache/beam/pull/29327 > On 9 Nov 2023, at 06:37, Giridhar Addepalli <giridhar1...@gmail.com> wrote: > > Thank you Alexey for sharing the details. > > Can you please let us know if you are planning to add Spark 3.5.0 > compatibility test as part of Beam 2.53.0 or not. > If so, approximately what is the timeline we are looking at for Beam 2.53.0 > release. > https://github.com/apache/beam/milestone/17 > > Thanks, > Giridhar. > > On Tue, Nov 7, 2023 at 6:24 PM Alexey Romanenko <aromanenko....@gmail.com > <mailto:aromanenko....@gmail.com>> wrote: >> Hi Giridhar, >> >>> On 4 Nov 2023, at 08:04, Giridhar Addepalli <giridhar1...@gmail.com >>> <mailto:giridhar1...@gmail.com>> wrote: >>> >>> Thank you Alexey for the response. >>> >>> We are using Beam 2.41.0 with Spark 3.3.0 cluster. >>> We did not run into any issues. >>> is it because in Beam 2.41.0, compatibility tests were run against spark >>> 3.3.0 ? >>> https://github.com/apache/beam/blob/release-2.41.0/runners/spark/3/build.gradle >> >> Correct. >> There are some incompatibilities between Spark 3.1/3.2/3.3 versions and we >> fixed this for Spark runner in Beam 2.41 to make it possible to compile and >> run with different Spark versions. That was a goal of these compatibility >> tests. >> >> <22157.png> >> Fixes #22156: Fix Spark3 runner to compile against Spark 3.2/3.3 by mosche · >> Pull Request #22157 · apache/beam >> github.com >> <https://github.com/apache/beam/pull/22157>Fixes #22156: Fix Spark3 runner >> to compile against Spark 3.2/3.3 by mosche · Pull Request #22157 · >> apache/beam <https://github.com/apache/beam/pull/22157> >> github.com <https://github.com/apache/beam/pull/22157> >> >>> If so, since compatibility tests were not run against Spark 3.5.0 even in >>> latest release of Beam 2.52.0, is it not advised to use Beam 2.52.0 with >>> Spark 3.5.0 cluster ? >> >> I’d say, for now it's up to user to test and run it since it was not tested >> on Beam CI. >> I’m going to add this version for future testing. >> >> — >> Alexey >> >>> >>> Thanks, >>> Giridhar. >>> >>> On 2023/11/03 13:05:45 Alexey Romanenko wrote: >>> > AFAICT, the latest tested (compatibility tests) version for now is 3.4.1 >>> > [1] We may try to add 3.5.x version there. >>> > >>> > I believe that ValidateRunners tests are run only against default Spark >>> > 3.2.2 version. >>> > >>> > — >>> > Alexey >>> > >>> > [1] >>> > https://github.com/apache/beam/blob/2aaf09c0eb6928390d861ba228447338b8ca92d3/runners/spark/3/build.gradle#L36 >>> > >>> > >>> > > On 3 Nov 2023, at 05:06, Sri Ganesh Venkataraman <sr...@gmail.com >>> > > <mailto:sr...@gmail.com>> wrote: >>> > > >>> > > Does Apache Beam version (2.41.0) or latest (2.51.0) support Spark 3.5 >>> > > environment for spark runner ? >>> > > >>> > > Apache Beam - Spark Runner Documentation states - >>> > > The Spark runner currently supports Spark’s 3.2.x branch >>> > > >>> > > Thanks >>> > > Sri Ganesh V >>> > >>> > >>
