Sure. I starred your repository. On Sat, Aug 21, 2021 at 11:27 AM cw <second_co...@yahoo.com> wrote:
> Hello Yu, > i done lot of testing, it only work for spark 2+, not 3. if you need a > working example on kubernetes, > https://github.com/cometta/python-apache-beam-spark , feel free to > improve the code, if you would like to contribute. help me *star if if it > is useful for you. thank you > > On Monday, August 16, 2021, 12:37:46 AM GMT+8, Yu Watanabe < > yu.w.ten...@gmail.com> wrote: > > > Hello . > > I would like to ask question for spark runner. > > Using spark downloaded from below link, > > > https://www.apache.org/dyn/closer.lua/spark/spark-3.1.2/spark-3.1.2-bin-hadoop3.2.tgz > > I get below error when submitting a pipeline. > Full error is on > https://gist.github.com/yuwtennis/7b0c1dc0dcf98297af1e3179852ca693. > > > ------------------------------------------------------------------------------------------------------------------ > 21/08/16 01:10:26 WARN TransportChannelHandler: Exception in connection > from /192.168.11.2:35601 > java.io.InvalidClassException: > scala.collection.mutable.WrappedArray$ofRef; local class incompatible: > stream classdesc serialVersionUID = 3456489343829468865, local class > serialVersionUID = 1028182004549731694 > at > java.base/java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:689) > ... > > ------------------------------------------------------------------------------------------------------------------ > > SDK Harness and Job service are deployed as below. > > 1. SDK Harness > > sudo docker run --net=host apache/beam_spark3_job_server:2.31.0 > --spark-master-url=spark://localhost:7077 --clean-artifacts-per-job true > > 2. Job service > > sudo docker run --net=host apache/beam_python3.8_sdk:2.31.0 --worker_pool > > * apache/beam_spark_job_server:2.31.0 for spark 2.4.8 > > 3. SDK client code > > https://gist.github.com/yuwtennis/2e4c13c79f71e8f713e947955115b3e2 > > Spark 2.4.8 succeeded without any errors using above components. > > > https://archive.apache.org/dist/spark/spark-2.4.8/spark-2.4.8-bin-hadoop2.7.tgz > > Would there be any setting which you need to be aware of for spark 3.1.2 ? > > Thanks, > Yu Watanabe > > -- > Yu Watanabe > > linkedin: www.linkedin.com/in/yuwatanabe1/ > twitter: twitter.com/yuwtennis > > -- Yu Watanabe linkedin: www.linkedin.com/in/yuwatanabe1/ twitter: twitter.com/yuwtennis
