Hi Jan,
Thanks for the reply, I was able to submit the job to flink but it's
failing due to an OOM issue so I am moving to the GKE. I got the flink UI
there but submitted a job not appearing on flink UI. I am using the same
script which I shared with you.. Do I need to make some changes for Google
Kubernetes Environment?
On Tue, 31 Jan 2023 at 20:20, Jan Lukavský <je...@seznam.cz> wrote:
> The script looks good to me, did you run the SDK harness? External
> environment needs the SDK harness to be run externally, see [1]. Generally,
> the best option is DOCKER, but that usually does not work in k8s. For this,
> you might try PROCESS environment and build your own docker image for
> flink, which will contain the Beam harness, e.g. [2]. You will need to pass
> the environment config using --environment_config={"command":
> "/opt/apache/beam/boot"}.
>
> From the screenshot it seems, that the Flink UI is accessible, so this is
> the only option that comes to my mind. Did you check logs of the Flink
> jobmanager pod?
>
> Jan
>
> [1] https://beam.apache.org/documentation/runtime/sdk-harness-config/
>
> [2]
> https://github.com/PacktPublishing/Building-Big-Data-Pipelines-with-Apache-Beam/blob/main/env/docker/flink/Dockerfile
> On 1/31/23 13:33, P Singh wrote:
>
> HI Jan,
>
> Thanks for your reply, please find attached script, I am newbie with flink
> and minikube though i am trying to connect them by script from local
> machine as suggested by flink kubernetes documents link
> <https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/resource-providers/standalone/kubernetes/>
>
> I have changed the log level to ERROR but didn't find much... Can you
> please help me out how to run the script from inside the pod.
>
> On Tue, 31 Jan 2023 at 15:40, Jan Lukavský <je...@seznam.cz> wrote:
>
>> Hi,
>>
>> can you please share the also the script itself? I'd say that the problem
>> is that the flink jobmanager is not accessible through localhost:8081,
>> because it runs inside the minikube. You need to expose it outside of the
>> minikube via [1], or run the script from pod inside the minikube and access
>> job manager via flink-jobmanager:8081. I'm surprised that the log didn't
>> make this more obvious, though. Is it possible that you changed the default
>> log level to ERROR? Can you try DEBUG or similar?
>>
>> Jan
>>
>> [1] https://minikube.sigs.k8s.io/docs/handbook/accessing/
>> On 1/30/23 18:36, P Singh wrote:
>>
>> Hi Jan,
>>
>> Yeah I am using minikube and beam image with python 3.10.
>>
>> Please find the attached screenshots.
>>
>>
>>
>> On Mon, 30 Jan 2023 at 21:22, Jan Lukavský <je...@seznam.cz> wrote:
>>
>>> Hi,
>>>
>>> can you please share the command-line and complete output of the script?
>>> Are you using minikube? Can you share list of your running pods?
>>>
>>> Jan
>>>
>>> On 1/30/23 14:25, P Singh wrote:
>>> > Hi Team,
>>> >
>>> > I am trying to run beam job on top of flink on my local machine
>>> > (kubernetes).
>>> >
>>> > I have flink 1.14 and beam 2.43 images both running but when i submit
>>> > the job it's not reaching to the flink cluster and getting failed with
>>> > below error.
>>> >
>>> > ERROR:apache_beam.utils.subprocess_server:Starting job service with
>>> > ['java', '-jar',
>>> >
>>> '/Users/spsingh/.apache_beam/cache/jars/beam-runners-flink-1.14-job-server-2.43.0.jar',
>>>
>>> > '--flink-master', 'http://localhost:8081', '--artifacts-dir',
>>> >
>>> '/var/folders/n3/dqblsr792yj4kfs7xlfmdj540000gr/T/beam-tempvphhje07/artifacts6kjt60ch',
>>>
>>> > '--job-port', '57882', '--artifact-port', '0', '--expansion-port', '0']
>>> > ERROR:apache_beam.utils.subprocess_server:Error bringing up service
>>> > Traceback (most recent call last):
>>> > File
>>> >
>>> "/Users/flink_deploy/flink_env/lib/python3.10/site-packages/apache_beam/utils/subprocess_server.py",
>>>
>>> > line 88, in start
>>> > raise RuntimeError(
>>> > RuntimeError: Service failed to start up with error 1
>>> >
>>> > Any help would be appreciated.
>>>
>>
