Re: How to use the loopback?
The StopWorker "errors" are harmless and there's an easy patch for them: https://github.com/apache/beam/pull/9600 (I already included this in my reply on the other thread, but putting it down here for the record) Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com On Mon, Sep 16, 2019 at 11:25 PM Benjamin Tan wrote: > Some updates! > > I built and install the beam 2.16.0 python library for both Python 2 and > Python 3. > > Python 3 leaves me with the same error. > > However, apparently Python 2 works. I can see the generated output. > However, I get the following error: > > Any ideas? I'm not sure why this is so hard. > > /home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/__init__.py:84: > UserWarning: You are using Apache Beam with Python 2. New releases of > Apache Beam will soon support Python 3 only. > 'You are using Apache Beam with Python 2. ' > ERROR:grpc._server:Exception calling application: u'1-1' > Traceback (most recent call last): > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", > line 434, in _call_behavior > response_or_iterator = behavior(argument, context) > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", > line 126, in StopWorker > worker_process = > self._worker_processes.pop(stop_worker_request.worker_id) > KeyError: u'1-1' > ERROR:grpc._server:Exception calling application: u'2-1' > Traceback (most recent call last): > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", > line 434, in _call_behavior > response_or_iterator = behavior(argument, context) > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", > line 126, in StopWorker > worker_process = > self._worker_processes.pop(stop_worker_request.worker_id) > KeyError: u'2-1' > ERROR:grpc._server:Exception calling application: u'4-1' > Traceback (most recent call last): > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", > line 434, in _call_behavior > response_or_iterator = behavior(argument, context) > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", > line 126, in StopWorker > worker_process = > self._worker_processes.pop(stop_worker_request.worker_id) > KeyError: u'4-1' > ERROR:grpc._server:Exception calling application: u'3-1' > Traceback (most recent call last): > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", > line 434, in _call_behavior > response_or_iterator = behavior(argument, context) > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", > line 126, in StopWorker > worker_process = > self._worker_processes.pop(stop_worker_request.worker_id) > KeyError: u'3-1' > ERROR:grpc._server:Exception calling application: u'6-1' > Traceback (most recent call last): > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", > line 434, in _call_behavior > response_or_iterator = behavior(argument, context) > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", > line 126, in StopWorker > worker_process = > self._worker_processes.pop(stop_worker_request.worker_id) > KeyError: u'6-1' > ERROR:grpc._server:Exception calling application: u'5-1' > Traceback (most recent call last): > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", > line 434, in _call_behavior > response_or_iterator = behavior(argument, context) > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", > line 126, in StopWorker > worker_process = > self._worker_processes.pop(stop_worker_request.worker_id) > KeyError: u'5-1' > ERROR:grpc._server:Exception calling application: u'7-1' > Traceback (most recent call last): > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", > line 434, in _call_behavior > response_or_iterator = behavior(argument, context) > File > "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", > line 126, in StopWorker > worker_process = > self._worker_processes.pop(stop_worker_request.worker_id) > KeyError: u'7-1' > ERROR:grpc._server:Exception calling application: u'8-1' > Traceback (most recent call last): > File >
Re: How to use the loopback?
Some updates! I built and install the beam 2.16.0 python library for both Python 2 and Python 3. Python 3 leaves me with the same error. However, apparently Python 2 works. I can see the generated output. However, I get the following error: Any ideas? I'm not sure why this is so hard. /home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/__init__.py:84: UserWarning: You are using Apache Beam with Python 2. New releases of Apache Beam will soon support Python 3 only. 'You are using Apache Beam with Python 2. ' ERROR:grpc._server:Exception calling application: u'1-1' Traceback (most recent call last): File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", line 126, in StopWorker worker_process = self._worker_processes.pop(stop_worker_request.worker_id) KeyError: u'1-1' ERROR:grpc._server:Exception calling application: u'2-1' Traceback (most recent call last): File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", line 126, in StopWorker worker_process = self._worker_processes.pop(stop_worker_request.worker_id) KeyError: u'2-1' ERROR:grpc._server:Exception calling application: u'4-1' Traceback (most recent call last): File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", line 126, in StopWorker worker_process = self._worker_processes.pop(stop_worker_request.worker_id) KeyError: u'4-1' ERROR:grpc._server:Exception calling application: u'3-1' Traceback (most recent call last): File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", line 126, in StopWorker worker_process = self._worker_processes.pop(stop_worker_request.worker_id) KeyError: u'3-1' ERROR:grpc._server:Exception calling application: u'6-1' Traceback (most recent call last): File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", line 126, in StopWorker worker_process = self._worker_processes.pop(stop_worker_request.worker_id) KeyError: u'6-1' ERROR:grpc._server:Exception calling application: u'5-1' Traceback (most recent call last): File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", line 126, in StopWorker worker_process = self._worker_processes.pop(stop_worker_request.worker_id) KeyError: u'5-1' ERROR:grpc._server:Exception calling application: u'7-1' Traceback (most recent call last): File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", line 126, in StopWorker worker_process = self._worker_processes.pop(stop_worker_request.worker_id) KeyError: u'7-1' ERROR:grpc._server:Exception calling application: u'8-1' Traceback (most recent call last): File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/grpc/_server.py", line 434, in _call_behavior response_or_iterator = behavior(argument, context) File "/home/benjamintan/miniconda3/envs/apbeam-playground/lib/python2.7/site-packages/apache_beam/runners/worker/worker_pool_main.py", line 126, in StopWorker worker_process = self._worker_processes.pop(stop_worker_request.worker_id) KeyError: u'8-1' ERROR:grpc._server:Exception calling application: u'10-1' Traceback (most recent call last):
Re: How to use the loopback?
Here you go! builder@dnnserver2:~/beam (release-2.16.0) $ ./gradlew :runners:spark:job-server:runShadow -PsparkMasterUrl=spark://dnnserver2:7077 Configuration on demand is an incubating feature. > Task :runners:spark:job-server:runShadow Listening for transport dt_socket at address: 5005 log4j:WARN No appenders could be found for logger (org.apache.beam.vendor.grpc.v1p21p0.io.netty.util.internal.logging.InternalLoggerFactory). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 19/09/17 12:57:06 INFO SparkContext: Running Spark version 2.4.4 19/09/17 12:57:06 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 19/09/17 12:57:06 INFO SparkContext: Submitted application: BeamApp-builder-0917045705-aa3391a3_96f998c3-3c0d-42f1-885c-f1e2538310bc 19/09/17 12:57:06 INFO SecurityManager: Changing view acls to: builder 19/09/17 12:57:06 INFO SecurityManager: Changing modify acls to: builder 19/09/17 12:57:06 INFO SecurityManager: Changing view acls groups to: 19/09/17 12:57:06 INFO SecurityManager: Changing modify acls groups to: 19/09/17 12:57:06 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(builder); groups with view permissions: Set(); users with modify permissions: Set(builder); groups with modify permissions: Set() 19/09/17 12:57:07 INFO Utils: Successfully started service 'sparkDriver' on port 36069. 19/09/17 12:57:07 INFO SparkEnv: Registering MapOutputTracker 19/09/17 12:57:07 INFO SparkEnv: Registering BlockManagerMaster 19/09/17 12:57:07 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 19/09/17 12:57:07 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 19/09/17 12:57:07 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-92f6079e-4a85-4b09-b48b-5d58ddf304a6 19/09/17 12:57:07 INFO MemoryStore: MemoryStore started with capacity 1949.1 MB 19/09/17 12:57:07 INFO SparkEnv: Registering OutputCommitCoordinator 19/09/17 12:57:07 INFO Utils: Successfully started service 'SparkUI' on port 4040. 19/09/17 12:57:07 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://dnnserver2:4040 19/09/17 12:57:07 INFO SparkContext: Added JAR /home/builder/beam/runners/spark/job-server/build/install/job-server-shadow/lib/beam-runners-spark-job-server-2.16.0-SNAPSHOT.jar at spark://dnnserver2:36069/jars/beam-runners-spark-job-server-2.16.0-SNAPSHOT.jar with timestamp 1568696227623 19/09/17 12:57:07 INFO StandaloneAppClient$ClientEndpoint: Connecting to master spark://dnnserver2:7077... 19/09/17 12:57:07 INFO TransportClientFactory: Successfully created connection to dnnserver2/10.64.1.208:7077 after 40 ms (0 ms spent in bootstraps) 19/09/17 12:57:07 INFO StandaloneSchedulerBackend: Connected to Spark cluster with app ID app-20190917125707-0066 19/09/17 12:57:07 INFO StandaloneAppClient$ClientEndpoint: Executor added: app-20190917125707-0066/0 on worker-20190916143324-10.64.1.208-41823 (10.64.1.208:41823) with 12 core(s) 19/09/17 12:57:07 INFO StandaloneSchedulerBackend: Granted executor ID app-20190917125707-0066/0 on hostPort 10.64.1.208:41823 with 12 core(s), 1024.0 MB RAM 19/09/17 12:57:07 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37069. 19/09/17 12:57:07 INFO NettyBlockTransferService: Server created on dnnserver2:37069 19/09/17 12:57:07 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy 19/09/17 12:57:07 INFO StandaloneAppClient$ClientEndpoint: Executor updated: app-20190917125707-0066/0 is now RUNNING 19/09/17 12:57:07 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, dnnserver2, 37069, None) 19/09/17 12:57:07 INFO BlockManagerMasterEndpoint: Registering block manager dnnserver2:37069 with 1949.1 MB RAM, BlockManagerId(driver, dnnserver2, 37069, None) 19/09/17 12:57:07 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, dnnserver2, 37069, None) 19/09/17 12:57:07 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, dnnserver2, 37069, None) 19/09/17 12:57:07 INFO StandaloneSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0 19/09/17 12:57:07 INFO SparkPipelineRunner: Running job BeamApp-builder-0917045705-aa3391a3_96f998c3-3c0d-42f1-885c-f1e2538310bc on Spark master spark://dnnserver2:7077 19/09/17 12:57:07 INFO AggregatorsAccumulator: Instantiated aggregators accumulator: 19/09/17 12:57:08 INFO MetricsAccumulator: Instantiated metrics accumulator: MetricQueryResults() 19/09/17 12:57:08 WARN GroupNonMergingWindowsFunctions: Either coder
Re: How to use the loopback?
Could you share more of the stack trace? Kyle Weaver | Software Engineer | github.com/ibzib | kcwea...@google.com On Mon, Sep 16, 2019 at 7:49 PM Benjamin Tan wrote: > I'm trying to use the loopback via the environment_type option: > > options = PipelineOptions(["--runner=PortableRunner", > > "--environment_config=-apachebeam/python3.7_sdk ", >"--environment_type=LOOPBACK", > > "--job_endpoint=dnnserver2:8099"]) > > Previouly, I've done: > > ./gradlew -p sdks/python/container buildAll > > And ran the Spark job server: > > ./gradlew :runners:spark:job-server:runShadow > -PsparkMasterUrl=spark://dnnserver2:7077 > > However, I get a pretty cryptic error message: > > org.apache.beam.vendor.guava.v26_0_jre.com.google.common.util.concurrent.UncheckedExecutionException: > org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: > UNIMPLEMENTED: Method not found! > > Any ideas? > > >
