So, it turns out this was a Spark version mismatch between the Beam JobServer and the Spark platform.
I'm running both Beam and Spark on Docker; the Spark image [0] provided version 3.3.0 and Scala version 2.12, but I used apache/beam_spark_job_server:2.41.0 [1] which provides Spark 2.4.x libraries, Scala version 2.11. Instead, I needed to use the apache/beam_spark3_job_server:2.41.0 image [2], which provides Spark 3.3.x. [0] https://hub.docker.com/r/apache/spark/tags [1] https://hub.docker.com/r/apache/beam_spark_job_server/tags [2] https://hub.docker.com/r/apache/beam_spark3_job_server/tags On 2022/08/25 17:34:17 Elliot Metsger wrote: > [Apologies for cross posting] > > Howdy folks, super-new to Beam, and attempting to get a simple example > working with Go, using the portable runner and Spark. There seems to be an > incompatibility between Java components, and I’m not quite sure where the > disconnect is, but at the root it seems to be an incompatibility with > object serializations. > > When I submit the job via the go sdk, it errors out on the Spark side with: > [8:59 AM] 22/08/25 12:45:59 ERROR TransportRequestHandler: Error while > invoking RpcHandler#receive() for one-way message. > java.io.InvalidClassException: > org.apache.spark.deploy.ApplicationDescription; local class incompatible: > stream classdesc serialVersionUID = 6543101073799644159, local class > serialVersionUID = 1574364215946805297 > I’m using apache/beam_spark_job_server:2.41.0 and apache/spark:latest. > (docker-compose[0], hello world wordcount example pipeline[1]). > > Any ideas on where to look? It looks like the Beam JobService is using > Java 8 (?) and Spark is using Java 11. I’ve tried downgrading Spark from > 3.3.0 to 3.1.3 (the earliest version for which Docker images are > available), and downgrading to Beam 2.40.0 with no luck. > > This simple repo[2] should demonstrate the issue. Any pointers would be > appreciated! > > [0]: https://github.com/emetsger/beam-test/blob/develop/docker-compose.yml > [1]: > https://github.com/emetsger/beam-test/blob/develop/debugging_wordcount.go > [2]: https://github.com/emetsger/beam-test >
