Howdy folks, super-new to Beam, and attempting to get a simple example working with Go, using the portable runner and Spark. There seems to be an incompatibility between Java components, and I’m not quite sure where the disconnect is, but at the root it seems to be an incompatibility with object serializations.
When I submit the job via the go sdk, it errors out on the Spark side with: [8:59 AM] 22/08/25 12:45:59 ERROR TransportRequestHandler: Error while invoking RpcHandler#receive() for one-way message. java.io.InvalidClassException: org.apache.spark.deploy.ApplicationDescription; local class incompatible: stream classdesc serialVersionUID = 6543101073799644159, local class serialVersionUID = 1574364215946805297 I’m using apache/beam_spark_job_server:2.41.0 and apache/spark:latest. (docker-compose[0], hello world wordcount example pipeline[1]). Any ideas on where to look? It looks like the Beam JobService is using Java 8 (?) and Spark is using Java 11. I’ve tried downgrading Spark from 3.3.0 to 3.1.3 (the earliest version for which Docker images are available), and downgrading to Beam 2.40.0 with no luck. This simple repo[2] should demonstrate the issue. Any pointers would be appreciated! [0]: https://github.com/emetsger/beam-test/blob/develop/docker-compose.yml [1]: https://github.com/emetsger/beam-test/blob/develop/debugging_wordcount.go [2]: https://github.com/emetsger/beam-test