Thank you Alexey for the response.
We are using Beam 2.41.0 with Spark 3.3.0 cluster. We did not run into any issues. is it because in Beam 2.41.0, compatibility tests were run against spark 3.3.0 ? https://github.com/apache/beam/blob/release-2.41.0/runners/spark/3/build.gradle If so, since compatibility tests were not run against Spark 3.5.0 even in latest release of Beam 2.52.0, is it not advised to use Beam 2.52.0 with Spark 3.5.0 cluster ? Thanks, Giridhar. On 2023/11/03 13:05:45 Alexey Romanenko wrote: > AFAICT, the latest tested (compatibility tests) version for now is 3.4.1 [1] We may try to add 3.5.x version there. > > I believe that ValidateRunners tests are run only against default Spark 3.2.2 version. > > — > Alexey > > [1] https://github.com/apache/beam/blob/2aaf09c0eb6928390d861ba228447338b8ca92d3/runners/spark/3/build.gradle#L36 > > > > On 3 Nov 2023, at 05:06, Sri Ganesh Venkataraman <[email protected]> wrote: > > > > Does Apache Beam version (2.41.0) or latest (2.51.0) support Spark 3.5 environment for spark runner ? > > > > Apache Beam - Spark Runner Documentation states - > > The Spark runner currently supports Spark’s 3.2.x branch > > > > Thanks > > Sri Ganesh V > >
