Keep in mind that those instructions about spark-submit are meant only to apply to the Java-only runner. For Python, running spark-submit in this manner is not going to work.
See https://issues.apache.org/jira/browse/BEAM-8970 On Thu, May 14, 2020 at 2:55 PM Heejong Lee <[email protected]> wrote: > How did you start spark job server and what version of Apache Beam SDK did > you use? > > There were some protocol changes recently so if both versions are not > matched you may see gRPC errors. If you used the gradle command on the > latest head for starting spark job server, I would recommend checking out > the same version of the source with the SDK version you installed and > trying again. > > On Wed, May 13, 2020 at 2:51 PM Naveen M <[email protected]> wrote: > >> Hi, >> >> I am trying to run sample WordCount beam job with PortableRunner by >> following the documentation here, >> >> https://beam.apache.org/documentation/runners/spark/ >> >> I want to run this as a spark-submit command with YARN resource manager. >> >> Can you please let me know what is missing here? Thanks your help. >> >> >> I tried the below commands and giving some weird errors, >> >> >> >> >> spark-submit --master yarn --deploy-mode client --driver-memory 2g >> --executor-memory 1g --executor-cores 1 WordCount.py --input "XXXX" >> --output "XXXX" --runner PortableRunner --job_endpoint localhost:8099 >> >> >> >> >> >> >> File >> "/usr/local/lib64/python3.7/site-packages/apache_beam/pipeline.py", line >> 503, in __exit__ >> >> self.run().wait_until_finish() >> >> File >> "/usr/local/lib64/python3.7/site-packages/apache_beam/pipeline.py", line >> 483, in run >> >> self._options).run(False) >> >> File >> "/usr/local/lib64/python3.7/site-packages/apache_beam/pipeline.py", line >> 496, in run >> >> return self.runner.run_pipeline(self, self._options) >> >> File >> "/usr/local/lib64/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", >> line 384, in run_pipeline >> >> job_service_plan.submit(proto_pipeline) >> >> File >> "/usr/local/lib64/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", >> line 105, in submit >> >> prepare_response.staging_session_token) >> >> File >> "/usr/local/lib64/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", >> line 190, in stage >> >> staging_location='') >> >> File >> "/usr/local/lib64/python3.7/site-packages/apache_beam/runners/portability/stager.py", >> line 229, in stage_job_resources >> >> self.stage_artifact(pickled_session_file, staged_path) >> >> File >> "/usr/local/lib64/python3.7/site-packages/apache_beam/runners/portability/portable_stager.py", >> line 98, in stage_artifact >> >> self._artifact_staging_stub.PutArtifact(artifact_request_generator()) >> >> File "/usr/local/lib64/python3.7/site-packages/grpc/_channel.py", line >> 1011, in __call__ >> >> return _end_unary_response_blocking(state, call, False, None) >> >> File "/usr/local/lib64/python3.7/site-packages/grpc/_channel.py", line >> 729, in _end_unary_response_blocking >> >> raise _InactiveRpcError(state) >> >> grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that >> terminated with: >> >> status = StatusCode.UNIMPLEMENTED >> >> details = "Method not found: >> org.apache.beam.model.job_management.v1.ArtifactStagingService/PutArtifact" >> >> debug_error_string = >> "{"created":"@1589406258.175447016","description":"Error received from peer >> ipv6:[::1]:8098","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Method >> not found: >> org.apache.beam.model.job_management.v1.ArtifactStagingService/PutArtifact","grpc_status":12}" >> >
