Hello . >From this line in the log, may I confirm whether you set up harness container ?
I have executed your code in below environment and it worked. Python version is newer though. OS: Fedora 31 BEAM: 2.19.0 PYTHON: 3.7.6 Compared to my log , I realized below line does not exist in your log. -------------------------------------------------------------------------------------------- 357 [grpc-default-executor-1] INFO org.apache.beam.runners.fnexecution.control.FnApiControlClientPoolService - Beam Fn Control client connected with id 1-1 358 [grpc-default-executor-1] INFO org.apache.beam.runners.fnexecution.data.GrpcDataService - Beam Fn Data client connected. -------------------------------------------------------------------------------------------- One thing I realized was in LOOPBACK mode for harness container , job server uses hostname for when communicating with flink. _______________________________________________________ Caused by: java.net.UnknownHostException: fedora-desktop: Name or service not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323) at java.net.InetAddress.getLocalHost(InetAddress.java:1500) ... 34 more _______________________________________________________ Perhaps temporarily adding your hostname to "hosts file" by resolving with 127.0.0.1 and see if it works ? Thanks, Yu Watanabe On Mon, Feb 10, 2020 at 3:29 AM Xander Song <iamuuriw...@gmail.com> wrote: > > Hi Jincheng, > > Thanks for your help. Yes, I am using Mac. Your suggestion allowed me to > submit the job on port 8099. However, I am now encountering a different error > message. > > ERROR:root:java.net.ConnectException: Connection refused > > Traceback (most recent call last): > > File "test_beam_local_flink.py", line 18, in <module> > > | apache_beam.Map(lambda x: print(x)) > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py", > line 481, in __exit__ > > self.run().wait_until_finish() > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py", > line 461, in run > > self._options).run(False) > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py", > line 474, in run > > return self.runner.run_pipeline(self, self._options) > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 334, in run_pipeline > > result.wait_until_finish() > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 455, in wait_until_finish > > self._job_id, self._state, self._last_error_message())) > > RuntimeError: Pipeline > BeamApp-root-0209182021-fc6ec677_c351e1bb-d428-4475-9910-4544d8ec1de4 failed > in state FAILED: java.net.ConnectException: Connection refused > > > I've attached a text file containing the output from the terminal running the > Docker container in case that is informative (it is quite lengthy). Any > suggestions are appreciated. > > Best, > Xander > > On Sun, Feb 9, 2020 at 2:44 AM jincheng sun <sunjincheng...@gmail.com> wrote: >> >> >> Hi Xander, >> >> Are you using Mac? The option --net=host doesn't work on Mac[1]. Could you >> try to see if the command >> `docker run -p 8099:8099 -p 8098:8098 -p 8097:8097 >> apachebeam/flink1.9_job_server:latest` works? >> >> [1] https://forums.docker.com/t/should-docker-run-net-host-work/14215/28 >> >> Best, >> Jincheng >> ------------- >> Twitter: https://twitter.com/sunjincheng121 >> ------------- >> >> >> Xander Song <iamuuriw...@gmail.com> 于2020年2月9日周日 上午11:52写道: >>> >>> Do you have any suggestions for addressing this issue? I am unsure of what >>> to try next. >>> >>> On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka <goe...@google.com> wrote: >>>> >>>> That seems to be a problem. >>>> >>>> When I try the command, I get >>>> >>>> $ telnet localhost 8099 >>>> Trying ::1... >>>> Connected to localhost. >>>> Escape character is '^]'. >>>> �^CConnection closed by foreign host. >>>> >>>> On Fri, Feb 7, 2020 at 5:34 PM Xander Song <iamuuriw...@gmail.com> wrote: >>>>> >>>>> Thanks for the response. After entering telnet localhost 8099, I receive >>>>> >>>>> Trying ::1... >>>>> >>>>> telnet: connect to address ::1: Connection refused >>>>> >>>>> Trying 127.0.0.1... >>>>> >>>>> telnet: connect to address 127.0.0.1: Connection refused >>>>> >>>>> telnet: Unable to connect to remote host >>>>> >>>>> >>>>> >>>>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka <goe...@google.com> wrote: >>>>>> >>>>>> Seems that pipeline submission from sdk is not able to reach the job >>>>>> server which was started in docker. >>>>>> >>>>>> Can you try running "telnet localhost 8099" to make sure that pipeline >>>>>> submission can reach the job server. >>>>>> >>>>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song <iamuuriw...@gmail.com> wrote: >>>>>>> >>>>>>> I am having difficulty following the Python guide for running Beam on >>>>>>> Flink. I created a virtual environment with Apache Beam installed, then >>>>>>> I started up the JobService Docker container with >>>>>>> >>>>>>> docker run --net=host apachebeam/flink1.9_job_server:latest >>>>>>> >>>>>>> >>>>>>> I receive the following message confirming that the container is >>>>>>> running. >>>>>>> >>>>>>> >>>>>>> [main] INFO >>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - >>>>>>> ArtifactStagingService started on localhost:8098 >>>>>>> >>>>>>> [main] INFO >>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - >>>>>>> Java ExpansionService started on localhost:8097 >>>>>>> >>>>>>> [main] INFO >>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - >>>>>>> JobService started on localhost:8099 >>>>>>> >>>>>>> >>>>>>> In another terminal, I execute a Beam script called >>>>>>> test_beam_local_flink.py based on the example. >>>>>>> >>>>>>> >>>>>>> from __future__ import print_function >>>>>>> >>>>>>> import apache_beam >>>>>>> from apache_beam.options.pipeline_options import PipelineOptions >>>>>>> >>>>>>> data = [1,2,3] >>>>>>> >>>>>>> options = PipelineOptions([ >>>>>>> "--runner=PortableRunner", >>>>>>> "--job_endpoint=localhost:8099", >>>>>>> "--environment_type=LOOPBACK" >>>>>>> ]) >>>>>>> >>>>>>> with apache_beam.Pipeline(options=options) as p: >>>>>>> video_collection = ( >>>>>>> p | apache_beam.Create(data) >>>>>>> | apache_beam.Map(lambda x: x + 1) >>>>>>> | apache_beam.Map(lambda x: print(x)) >>>>>>> ) >>>>>>> >>>>>>> print('Done') >>>>>>> >>>>>>> After a wait, I get the following traceback. >>>>>>> >>>>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84: >>>>>>> UserWarning: You are using Apache Beam with Python 2. New releases of >>>>>>> Apache Beam will soon support Python 3 only. >>>>>>> >>>>>>> 'You are using Apache Beam with Python 2. ' >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> >>>>>>> File "test_beam_local_flink.py", line 18, in <module> >>>>>>> >>>>>>> | apache_beam.Map(lambda x: print(x)) >>>>>>> >>>>>>> File >>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", >>>>>>> line 481, in __exit__ >>>>>>> >>>>>>> self.run().wait_until_finish() >>>>>>> >>>>>>> File >>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", >>>>>>> line 461, in run >>>>>>> >>>>>>> self._options).run(False) >>>>>>> >>>>>>> File >>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py", >>>>>>> line 474, in run >>>>>>> >>>>>>> return self.runner.run_pipeline(self, self._options) >>>>>>> >>>>>>> File >>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", >>>>>>> line 220, in run_pipeline >>>>>>> >>>>>>> job_service = self.create_job_service(options) >>>>>>> >>>>>>> File >>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py", >>>>>>> line 136, in create_job_service >>>>>>> >>>>>>> return server.start() >>>>>>> >>>>>>> File >>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py", >>>>>>> line 59, in start >>>>>>> >>>>>>> grpc.channel_ready_future(channel).result(timeout=self._timeout) >>>>>>> >>>>>>> File >>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", >>>>>>> line 140, in result >>>>>>> >>>>>>> self._block(timeout) >>>>>>> >>>>>>> File >>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py", >>>>>>> line 86, in _block >>>>>>> >>>>>>> raise grpc.FutureTimeoutError() >>>>>>> >>>>>>> grpc.FutureTimeoutError >>>>>>> >>>>>>> >>>>>>> >>>>>>> Any help is greatly appreciated. -- Yu Watanabe linkedin: www.linkedin.com/in/yuwatanabe1/ twitter: twitter.com/yuwtennis