Re: Running Beam on Flink
>>>>> Trying ::1...
>>>>>
>>>>> telnet: connect to address ::1: Connection refused
>>>>>
>>>>> Trying 127.0.0.1...
>>>>>
>>>>> telnet: connect to address 127.0.0.1: Connection refused
>>>>>
>>>>> telnet: Unable to connect to remote host
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka wrote:
>>>>>>
>>>>>> Seems that pipeline submission from sdk is not able to reach the job
>>>>>> server which was started in docker.
>>>>>>
>>>>>> Can you try running "telnet localhost 8099" to make sure that pipeline
>>>>>> submission can reach the job server.
>>>>>>
>>>>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song wrote:
>>>>>>>
>>>>>>> I am having difficulty following the Python guide for running Beam on
>>>>>>> Flink. I created a virtual environment with Apache Beam installed, then
>>>>>>> I started up the JobService Docker container with
>>>>>>>
>>>>>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>>>>>
>>>>>>>
>>>>>>> I receive the following message confirming that the container is
>>>>>>> running.
>>>>>>>
>>>>>>>
>>>>>>> [main] INFO
>>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>>> ArtifactStagingService started on localhost:8098
>>>>>>>
>>>>>>> [main] INFO
>>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>>> Java ExpansionService started on localhost:8097
>>>>>>>
>>>>>>> [main] INFO
>>>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>>>> JobService started on localhost:8099
>>>>>>>
>>>>>>>
>>>>>>> In another terminal, I execute a Beam script called
>>>>>>> test_beam_local_flink.py based on the example.
>>>>>>>
>>>>>>>
>>>>>>> from __future__ import print_function
>>>>>>>
>>>>>>> import apache_beam
>>>>>>> from apache_beam.options.pipeline_options import PipelineOptions
>>>>>>>
>>>>>>> data = [1,2,3]
>>>>>>>
>>>>>>> options = PipelineOptions([
>>>>>>> "--runner=PortableRunner",
>>>>>>> "--job_endpoint=localhost:8099",
>>>>>>> "--environment_type=LOOPBACK"
>>>>>>> ])
>>>>>>>
>>>>>>> with apache_beam.Pipeline(options=options) as p:
>>>>>>> video_collection = (
>>>>>>> p | apache_beam.Create(data)
>>>>>>> | apache_beam.Map(lambda x: x + 1)
>>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>>> )
>>>>>>>
>>>>>>> print('Done')
>>>>>>>
>>>>>>> After a wait, I get the following traceback.
>>>>>>>
>>>>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84:
>>>>>>> UserWarning: You are using Apache Beam with Python 2. New releases of
>>>>>>> Apache Beam will soon support Python 3 only.
>>>>>>>
>>>>>>> 'You are using Apache Beam with Python 2. '
>>>>>>>
>>>>>>> Traceback (most recent call last):
>>>>>>>
>>>>>>> File "test_beam_local_flink.py", line 18, in
>>>>>>>
>>>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>>>>>> line 481, in __exit__
>>>>>>>
>>>>>>> self.run().wait_until_finish()
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>>>>>> line 461, in run
>>>>>>>
>>>>>>> self._options).run(False)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>>>>>> line 474, in run
>>>>>>>
>>>>>>> return self.runner.run_pipeline(self, self._options)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>>>>>>> line 220, in run_pipeline
>>>>>>>
>>>>>>> job_service = self.create_job_service(options)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>>>>>>> line 136, in create_job_service
>>>>>>>
>>>>>>> return server.start()
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py",
>>>>>>> line 59, in start
>>>>>>>
>>>>>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>>>>>>> line 140, in result
>>>>>>>
>>>>>>> self._block(timeout)
>>>>>>>
>>>>>>> File
>>>>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>>>>>>> line 86, in _block
>>>>>>>
>>>>>>> raise grpc.FutureTimeoutError()
>>>>>>>
>>>>>>> grpc.FutureTimeoutError
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Any help is greatly appreciated.
--
Yu Watanabe
linkedin: www.linkedin.com/in/yuwatanabe1/
twitter: twitter.com/yuwtennis
Re: Running Beam on Flink
Hi Xander, It will start an external Python worker pool service at the client side when submitting the job if the environment type is configured as "LOOPBACK". The operators(which run in docker) of the job will try to connect to the external Python worker pool service to start the Python SDK harness. The reason of the latest failure is because the operators which run in the docker could not connect to the service which is outside of the docker container(as a result it could not connect to the external Python worker pool service). I'm sorry that I have also no idea about the best practise for how to make it works if the job service runs in a docker container in Mac. Before knowing of the best practise, I guess you could start up the job service directly on the bare metal. Best, Jincheng - Twitter: https://twitter.com/sunjincheng121 - Xander Song 于2020年2月10日周一 上午2:29写道： > Hi Jincheng, > > Thanks for your help. Yes, I am using Mac. Your suggestion allowed me to > submit the job on port 8099. However, I am now encountering a different > error message. > > ERROR:root:java.net.ConnectException: Connection refused > > Traceback (most recent call last): > > File "test_beam_local_flink.py", line 18, in > > | apache_beam.Map(lambda x: print(x)) > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py", > line 481, in __exit__ > > self.run().wait_until_finish() > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py", > line 461, in run > > self._options).run(False) > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/pipeline.py", > line 474, in run > > return self.runner.run_pipeline(self, self._options) > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 334, in run_pipeline > > result.wait_until_finish() > > File > "/Users/xander/Projects/flink-test/env/lib/python3.7/site-packages/apache_beam/runners/portability/portable_runner.py", > line 455, in wait_until_finish > > self._job_id, self._state, self._last_error_message())) > > RuntimeError: Pipeline > BeamApp-root-0209182021-fc6ec677_c351e1bb-d428-4475-9910-4544d8ec1de4 > failed in state FAILED: java.net.ConnectException: Connection refused > > I've attached a text file containing the output from the terminal running > the Docker container in case that is informative (it is quite lengthy). Any > suggestions are appreciated. > > Best, > Xander > > On Sun, Feb 9, 2020 at 2:44 AM jincheng sun > wrote: > >> >> Hi Xander, >> >> Are you using Mac? The option --net=host doesn't work on Mac[1]. Could >> you try to see if the command >> `docker run -p 8099:8099 -p 8098:8098 -p 8097:8097 >> apachebeam/flink1.9_job_server:latest` works? >> >> [1] https://forums.docker.com/t/should-docker-run-net-host-work/14215/28 >> >> Best, >> Jincheng >> - >> Twitter: https://twitter.com/sunjincheng121 >> - >> >> >> Xander Song 于2020年2月9日周日 上午11:52写道： >> >>> Do you have any suggestions for addressing this issue? I am unsure of >>> what to try next. >>> >>> On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka wrote: >>> >>>> That seems to be a problem. >>>> >>>> When I try the command, I get >>>> >>>> $ telnet localhost 8099 >>>> Trying ::1... >>>> Connected to localhost. >>>> Escape character is '^]'. >>>> �^CConnection closed by foreign host. >>>> >>>> On Fri, Feb 7, 2020 at 5:34 PM Xander Song >>>> wrote: >>>> >>>>> Thanks for the response. After entering telnet localhost 8099, I >>>>> receive >>>>> >>>>> Trying ::1... >>>>> >>>>> telnet: connect to address ::1: Connection refused >>>>> >>>>> Trying 127.0.0.1... >>>>> >>>>> telnet: connect to address 127.0.0.1: Connection refused >>>>> >>>>> telnet: Unable to connect to remote host >>>>> >>>>> >>>>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka >>>>> wrote: >>>>> >>>>>> Seems that pipeline submission from sdk is not able to reach the job >>>>>> server which was started in docker.
Re: Running Beam on Flink
Hi Xander,
Are you using Mac? The option --net=host doesn't work on Mac[1]. Could you
try to see if the command
`docker run -p 8099:8099 -p 8098:8098 -p 8097:8097
apachebeam/flink1.9_job_server:latest` works?
[1] https://forums.docker.com/t/should-docker-run-net-host-work/14215/28
Best,
Jincheng
-
Twitter: https://twitter.com/sunjincheng121
-
Xander Song 于2020年2月9日周日 上午11:52写道：
> Do you have any suggestions for addressing this issue? I am unsure of what
> to try next.
>
> On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka wrote:
>
>> That seems to be a problem.
>>
>> When I try the command, I get
>>
>> $ telnet localhost 8099
>> Trying ::1...
>> Connected to localhost.
>> Escape character is '^]'.
>> �^CConnection closed by foreign host.
>>
>> On Fri, Feb 7, 2020 at 5:34 PM Xander Song wrote:
>>
>>> Thanks for the response. After entering telnet localhost 8099, I receive
>>>
>>> Trying ::1...
>>>
>>> telnet: connect to address ::1: Connection refused
>>>
>>> Trying 127.0.0.1...
>>>
>>> telnet: connect to address 127.0.0.1: Connection refused
>>>
>>> telnet: Unable to connect to remote host
>>>
>>>
>>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka wrote:
>>>
>>>> Seems that pipeline submission from sdk is not able to reach the job
>>>> server which was started in docker.
>>>>
>>>> Can you try running "telnet localhost 8099" to make sure that pipeline
>>>> submission can reach the job server.
>>>>
>>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song
>>>> wrote:
>>>>
>>>>> I am having difficulty following the Python guide for running Beam on
>>>>> Flink <https://beam.apache.org/documentation/runners/flink/>. I
>>>>> created a virtual environment with Apache Beam installed, then I started
>>>>> up
>>>>> the JobService Docker container with
>>>>>
>>>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>>>
>>>>>
>>>>> I receive the following message confirming that the container is
>>>>> running.
>>>>>
>>>>>
>>>>> [main] INFO
>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>> ArtifactStagingService started on localhost:8098
>>>>>
>>>>> [main] INFO
>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>>>>> ExpansionService started on localhost:8097
>>>>>
>>>>> [main] INFO
>>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>>> JobService started on localhost:8099
>>>>>
>>>>>
>>>>> In another terminal, I execute a Beam script called
>>>>> test_beam_local_flink.py based on the example.
>>>>>
>>>>>
>>>>> from __future__ import print_function
>>>>> import apache_beamfrom apache_beam.options.pipeline_options import
>>>>> PipelineOptions
>>>>>
>>>>> data = [1,2,3]
>>>>>
>>>>> options = PipelineOptions([
>>>>> "--runner=PortableRunner",
>>>>> "--job_endpoint=localhost:8099",
>>>>> "--environment_type=LOOPBACK"])
>>>>> with apache_beam.Pipeline(options=options) as p:
>>>>> video_collection = (
>>>>> p | apache_beam.Create(data)
>>>>> | apache_beam.Map(lambda x: x + 1)
>>>>> | apache_beam.Map(lambda x: print(x))
>>>>> )
>>>>> print('Done')
>>>>>
>>>>> After a wait, I get the following traceback.
>>>>>
>>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84:
>>>>> UserWarning: You are using Apache Beam with Python 2. New releases of
>>>>> Apache Beam will soon support Python 3 only.
>>>>>
>>>>> 'You are using Apache Beam with Python 2. '
>>>>>
>>>>> Traceback (most recent call last):
>>>>>
>>>>> File "test_beam_local_flink.py", line 18, in
>>>>>
>>>>> | apache_beam.Map(lambda x: print(x))
>>>>>
&g
Re: Running Beam on Flink
Do you have any suggestions for addressing this issue? I am unsure of what
to try next.
On Fri, Feb 7, 2020 at 5:55 PM Ankur Goenka wrote:
> That seems to be a problem.
>
> When I try the command, I get
>
> $ telnet localhost 8099
> Trying ::1...
> Connected to localhost.
> Escape character is '^]'.
> �^CConnection closed by foreign host.
>
> On Fri, Feb 7, 2020 at 5:34 PM Xander Song wrote:
>
>> Thanks for the response. After entering telnet localhost 8099, I receive
>>
>> Trying ::1...
>>
>> telnet: connect to address ::1: Connection refused
>>
>> Trying 127.0.0.1...
>>
>> telnet: connect to address 127.0.0.1: Connection refused
>>
>> telnet: Unable to connect to remote host
>>
>>
>> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka wrote:
>>
>>> Seems that pipeline submission from sdk is not able to reach the job
>>> server which was started in docker.
>>>
>>> Can you try running "telnet localhost 8099" to make sure that pipeline
>>> submission can reach the job server.
>>>
>>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song
>>> wrote:
>>>
>>>> I am having difficulty following the Python guide for running Beam on
>>>> Flink <https://beam.apache.org/documentation/runners/flink/>. I
>>>> created a virtual environment with Apache Beam installed, then I started up
>>>> the JobService Docker container with
>>>>
>>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>>
>>>>
>>>> I receive the following message confirming that the container is
>>>> running.
>>>>
>>>>
>>>> [main] INFO
>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>> ArtifactStagingService started on localhost:8098
>>>>
>>>> [main] INFO
>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>>>> ExpansionService started on localhost:8097
>>>>
>>>> [main] INFO
>>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>>> JobService started on localhost:8099
>>>>
>>>>
>>>> In another terminal, I execute a Beam script called
>>>> test_beam_local_flink.py based on the example.
>>>>
>>>>
>>>> from __future__ import print_function
>>>> import apache_beamfrom apache_beam.options.pipeline_options import
>>>> PipelineOptions
>>>>
>>>> data = [1,2,3]
>>>>
>>>> options = PipelineOptions([
>>>> "--runner=PortableRunner",
>>>> "--job_endpoint=localhost:8099",
>>>> "--environment_type=LOOPBACK"])
>>>> with apache_beam.Pipeline(options=options) as p:
>>>> video_collection = (
>>>> p | apache_beam.Create(data)
>>>> | apache_beam.Map(lambda x: x + 1)
>>>> | apache_beam.Map(lambda x: print(x))
>>>> )
>>>> print('Done')
>>>>
>>>> After a wait, I get the following traceback.
>>>>
>>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84:
>>>> UserWarning: You are using Apache Beam with Python 2. New releases of
>>>> Apache Beam will soon support Python 3 only.
>>>>
>>>> 'You are using Apache Beam with Python 2. '
>>>>
>>>> Traceback (most recent call last):
>>>>
>>>> File "test_beam_local_flink.py", line 18, in
>>>>
>>>> | apache_beam.Map(lambda x: print(x))
>>>>
>>>> File
>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>>> line 481, in __exit__
>>>>
>>>> self.run().wait_until_finish()
>>>>
>>>> File
>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>>> line 461, in run
>>>>
>>>> self._options).run(False)
>>>>
>>>> File
>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>>> line 474, in run
>>>>
>>>> return self.runner.run_pipeline(self, self._options)
>>>>
>>>> File
>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>>>> line 220, in run_pipeline
>>>>
>>>> job_service = self.create_job_service(options)
>>>>
>>>> File
>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>>>> line 136, in create_job_service
>>>>
>>>> return server.start()
>>>>
>>>> File
>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py",
>>>> line 59, in start
>>>>
>>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>>
>>>> File
>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>>>> line 140, in result
>>>>
>>>> self._block(timeout)
>>>>
>>>> File
>>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>>>> line 86, in _block
>>>>
>>>> raise grpc.FutureTimeoutError()
>>>>
>>>> grpc.FutureTimeoutError
>>>>
>>>>
>>>>
>>>> Any help is greatly appreciated.
>>>>
>>>>
Re: Running Beam on Flink
That seems to be a problem.
When I try the command, I get
$ telnet localhost 8099
Trying ::1...
Connected to localhost.
Escape character is '^]'.
�^CConnection closed by foreign host.
On Fri, Feb 7, 2020 at 5:34 PM Xander Song wrote:
> Thanks for the response. After entering telnet localhost 8099, I receive
>
> Trying ::1...
>
> telnet: connect to address ::1: Connection refused
>
> Trying 127.0.0.1...
>
> telnet: connect to address 127.0.0.1: Connection refused
>
> telnet: Unable to connect to remote host
>
>
> On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka wrote:
>
>> Seems that pipeline submission from sdk is not able to reach the job
>> server which was started in docker.
>>
>> Can you try running "telnet localhost 8099" to make sure that pipeline
>> submission can reach the job server.
>>
>> On Thu, Feb 6, 2020 at 8:16 PM Xander Song wrote:
>>
>>> I am having difficulty following the Python guide for running Beam on
>>> Flink <https://beam.apache.org/documentation/runners/flink/>. I created
>>> a virtual environment with Apache Beam installed, then I started up the
>>> JobService Docker container with
>>>
>>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>>
>>>
>>> I receive the following message confirming that the container is running.
>>>
>>>
>>> [main] INFO
>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>> ArtifactStagingService started on localhost:8098
>>>
>>> [main] INFO
>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>>> ExpansionService started on localhost:8097
>>>
>>> [main] INFO
>>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>>> JobService started on localhost:8099
>>>
>>>
>>> In another terminal, I execute a Beam script called
>>> test_beam_local_flink.py based on the example.
>>>
>>>
>>> from __future__ import print_function
>>> import apache_beamfrom apache_beam.options.pipeline_options import
>>> PipelineOptions
>>>
>>> data = [1,2,3]
>>>
>>> options = PipelineOptions([
>>> "--runner=PortableRunner",
>>> "--job_endpoint=localhost:8099",
>>> "--environment_type=LOOPBACK"])
>>> with apache_beam.Pipeline(options=options) as p:
>>> video_collection = (
>>> p | apache_beam.Create(data)
>>> | apache_beam.Map(lambda x: x + 1)
>>> | apache_beam.Map(lambda x: print(x))
>>> )
>>> print('Done')
>>>
>>> After a wait, I get the following traceback.
>>>
>>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84:
>>> UserWarning: You are using Apache Beam with Python 2. New releases of
>>> Apache Beam will soon support Python 3 only.
>>>
>>> 'You are using Apache Beam with Python 2. '
>>>
>>> Traceback (most recent call last):
>>>
>>> File "test_beam_local_flink.py", line 18, in
>>>
>>> | apache_beam.Map(lambda x: print(x))
>>>
>>> File
>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>> line 481, in __exit__
>>>
>>> self.run().wait_until_finish()
>>>
>>> File
>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>> line 461, in run
>>>
>>> self._options).run(False)
>>>
>>> File
>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>>> line 474, in run
>>>
>>> return self.runner.run_pipeline(self, self._options)
>>>
>>> File
>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>>> line 220, in run_pipeline
>>>
>>> job_service = self.create_job_service(options)
>>>
>>> File
>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>>> line 136, in create_job_service
>>>
>>> return server.start()
>>>
>>> File
>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py",
>>> line 59, in start
>>>
>>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>>
>>> File
>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>>> line 140, in result
>>>
>>> self._block(timeout)
>>>
>>> File
>>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>>> line 86, in _block
>>>
>>> raise grpc.FutureTimeoutError()
>>>
>>> grpc.FutureTimeoutError
>>>
>>>
>>>
>>> Any help is greatly appreciated.
>>>
>>>
Re: Running Beam on Flink
Thanks for the response. After entering telnet localhost 8099, I receive
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
telnet: connect to address 127.0.0.1: Connection refused
telnet: Unable to connect to remote host
On Fri, Feb 7, 2020 at 11:41 AM Ankur Goenka wrote:
> Seems that pipeline submission from sdk is not able to reach the job
> server which was started in docker.
>
> Can you try running "telnet localhost 8099" to make sure that pipeline
> submission can reach the job server.
>
> On Thu, Feb 6, 2020 at 8:16 PM Xander Song wrote:
>
>> I am having difficulty following the Python guide for running Beam on
>> Flink <https://beam.apache.org/documentation/runners/flink/>. I created
>> a virtual environment with Apache Beam installed, then I started up the
>> JobService Docker container with
>>
>> docker run --net=host apachebeam/flink1.9_job_server:latest
>>
>>
>> I receive the following message confirming that the container is running.
>>
>>
>> [main] INFO
>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>> ArtifactStagingService started on localhost:8098
>>
>> [main] INFO
>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
>> ExpansionService started on localhost:8097
>>
>> [main] INFO
>> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
>> JobService started on localhost:8099
>>
>>
>> In another terminal, I execute a Beam script called
>> test_beam_local_flink.py based on the example.
>>
>>
>> from __future__ import print_function
>> import apache_beamfrom apache_beam.options.pipeline_options import
>> PipelineOptions
>>
>> data = [1,2,3]
>>
>> options = PipelineOptions([
>> "--runner=PortableRunner",
>> "--job_endpoint=localhost:8099",
>> "--environment_type=LOOPBACK"])
>> with apache_beam.Pipeline(options=options) as p:
>> video_collection = (
>> p | apache_beam.Create(data)
>> | apache_beam.Map(lambda x: x + 1)
>> | apache_beam.Map(lambda x: print(x))
>> )
>> print('Done')
>>
>> After a wait, I get the following traceback.
>>
>> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84:
>> UserWarning: You are using Apache Beam with Python 2. New releases of
>> Apache Beam will soon support Python 3 only.
>>
>> 'You are using Apache Beam with Python 2. '
>>
>> Traceback (most recent call last):
>>
>> File "test_beam_local_flink.py", line 18, in
>>
>> | apache_beam.Map(lambda x: print(x))
>>
>> File
>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>> line 481, in __exit__
>>
>> self.run().wait_until_finish()
>>
>> File
>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>> line 461, in run
>>
>> self._options).run(False)
>>
>> File
>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
>> line 474, in run
>>
>> return self.runner.run_pipeline(self, self._options)
>>
>> File
>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>> line 220, in run_pipeline
>>
>> job_service = self.create_job_service(options)
>>
>> File
>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
>> line 136, in create_job_service
>>
>> return server.start()
>>
>> File
>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py",
>> line 59, in start
>>
>> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>>
>> File
>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>> line 140, in result
>>
>> self._block(timeout)
>>
>> File
>> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
>> line 86, in _block
>>
>> raise grpc.FutureTimeoutError()
>>
>> grpc.FutureTimeoutError
>>
>>
>>
>> Any help is greatly appreciated.
>>
>>
Re: Running Beam on Flink
Seems that pipeline submission from sdk is not able to reach the job server
which was started in docker.
Can you try running "telnet localhost 8099" to make sure that pipeline
submission can reach the job server.
On Thu, Feb 6, 2020 at 8:16 PM Xander Song wrote:
> I am having difficulty following the Python guide for running Beam on
> Flink <https://beam.apache.org/documentation/runners/flink/>. I created a
> virtual environment with Apache Beam installed, then I started up the
> JobService Docker container with
>
> docker run --net=host apachebeam/flink1.9_job_server:latest
>
>
> I receive the following message confirming that the container is running.
>
>
> [main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> ArtifactStagingService started on localhost:8098
>
> [main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
> ExpansionService started on localhost:8097
>
> [main] INFO
> org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
> JobService started on localhost:8099
>
>
> In another terminal, I execute a Beam script called
> test_beam_local_flink.py based on the example.
>
>
> from __future__ import print_function
> import apache_beamfrom apache_beam.options.pipeline_options import
> PipelineOptions
>
> data = [1,2,3]
>
> options = PipelineOptions([
> "--runner=PortableRunner",
> "--job_endpoint=localhost:8099",
> "--environment_type=LOOPBACK"])
> with apache_beam.Pipeline(options=options) as p:
> video_collection = (
> p | apache_beam.Create(data)
> | apache_beam.Map(lambda x: x + 1)
> | apache_beam.Map(lambda x: print(x))
> )
> print('Done')
>
> After a wait, I get the following traceback.
>
> /Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84:
> UserWarning: You are using Apache Beam with Python 2. New releases of Apache
> Beam will soon support Python 3 only.
>
> 'You are using Apache Beam with Python 2. '
>
> Traceback (most recent call last):
>
> File "test_beam_local_flink.py", line 18, in
>
> | apache_beam.Map(lambda x: print(x))
>
> File
> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 481, in __exit__
>
> self.run().wait_until_finish()
>
> File
> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 461, in run
>
> self._options).run(False)
>
> File
> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 474, in run
>
> return self.runner.run_pipeline(self, self._options)
>
> File
> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 220, in run_pipeline
>
> job_service = self.create_job_service(options)
>
> File
> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
> line 136, in create_job_service
>
> return server.start()
>
> File
> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py",
> line 59, in start
>
> grpc.channel_ready_future(channel).result(timeout=self._timeout)
>
> File
> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
> line 140, in result
>
> self._block(timeout)
>
> File
> "/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
> line 86, in _block
>
> raise grpc.FutureTimeoutError()
>
> grpc.FutureTimeoutError
>
>
>
> Any help is greatly appreciated.
>
>
Running Beam on Flink
I am having difficulty following the Python guide for running Beam on Flink
<https://beam.apache.org/documentation/runners/flink/>. I created a virtual
environment with Apache Beam installed, then I started up the JobService
Docker container with
docker run --net=host apachebeam/flink1.9_job_server:latest
I receive the following message confirming that the container is running.
[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
ArtifactStagingService started on localhost:8098
[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java
ExpansionService started on localhost:8097
[main] INFO
org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver -
JobService started on localhost:8099
In another terminal, I execute a Beam script called
test_beam_local_flink.py based on the example.
from __future__ import print_function
import apache_beamfrom apache_beam.options.pipeline_options import
PipelineOptions
data = [1,2,3]
options = PipelineOptions([
"--runner=PortableRunner",
"--job_endpoint=localhost:8099",
"--environment_type=LOOPBACK"])
with apache_beam.Pipeline(options=options) as p:
video_collection = (
p | apache_beam.Create(data)
| apache_beam.Map(lambda x: x + 1)
| apache_beam.Map(lambda x: print(x))
)
print('Done')
After a wait, I get the following traceback.
/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/__init__.py:84:
UserWarning: You are using Apache Beam with Python 2. New releases of
Apache Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
Traceback (most recent call last):
File "test_beam_local_flink.py", line 18, in
| apache_beam.Map(lambda x: print(x))
File
"/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 481, in __exit__
self.run().wait_until_finish()
File
"/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 461, in run
self._options).run(False)
File
"/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/pipeline.py",
line 474, in run
return self.runner.run_pipeline(self, self._options)
File
"/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 220, in run_pipeline
job_service = self.create_job_service(options)
File
"/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/portable_runner.py",
line 136, in create_job_service
return server.start()
File
"/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/apache_beam/runners/portability/job_server.py",
line 59, in start
grpc.channel_ready_future(channel).result(timeout=self._timeout)
File
"/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
line 140, in result
self._block(timeout)
File
"/Users/xander/Projects/flink-test/env2/lib/python2.7/site-packages/grpc/_utilities.py",
line 86, in _block
raise grpc.FutureTimeoutError()
grpc.FutureTimeoutError
Any help is greatly appreciated.
