Re: Flink Kubernetes Operator Application mode multiple jobs

[Raihan Sunny]

Got it. Thanks for the clarification guys.
________________________________
From: Guozhen Yang <abandoncl...@gmail.com>
Sent: Wednesday, May 8, 2024 9:38 AM
To: user@flink.apache.org <user@flink.apache.org>
Subject: RE: Flink Kubernetes Operator Application mode multiple jobs

Hi Raihan,

We have encountered the same issue though we are using Flink Kubernetes 
Operator 1.6.

Biao Geng's explanation is correct. We also have a detailed briefing in this 
jira ticket<https://issues.apache.org/jira/browse/FLINK-32423> but it seems 
that submitting multiple jobs is not allowed by design.

So we made a patch to solve this issue in our custom flink distribution. 
Solving this is simple.

All we need is make one line change in ApplicationReconciler 
class<https://github.com/apache/flink-kubernetes-operator/blob/77908c34dc6b35305c28f6e745e0547858509c6d/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/reconciler/deployment/ApplicationReconciler.java#L174>
 like below.

change from
        setJobIdIfNecessary(spec, relatedResource, deployConfig);
to

        if (HighAvailabilityMode.isHighAvailabilityModeActivated(deployConfig)) 
{
            setJobIdIfNecessary(spec, relatedResource, deployConfig);
        }

After this modification, everything works perfectly fine as we expected in our 
environment.

On 2024/05/07 05:48:02 Raihan Sunny wrote:
> Hi everyone,
>
> I need some help with deploying multiple jobs from a single main function in 
> Application mode using Flink Kubernetes Operator. As per the documentation 
> [1] it should be possible to use multiple "executeAsync()" to deploy multiple 
> jobs from the same file. This indeed is the case when running the job locally 
> using the CLI with something like `/bin/flink run -pym main -pyfs /project/` 
> (I'm using PyFlink btw) and I can see multiple jobs running in the UI. 
> However, when I try to deploy the same job using Flink Kubernetes Operator, 
> it seems that only the first job gets submitted. The second job is never 
> submitted although the code leading up to "executeAsync()" does get executed.
>
> This is a minimal representation of the deployment manifest that I tried to 
> run:
>
> apiVersion: flink.apache.org/v1beta1<http://flink.apache.org/v1beta1>
> kind: FlinkDeployment
> metadata:
>     name: flink-job
> spec:
>     image: flinkimage
>     imagePullPolicy: IfNotPresent
>     flinkVersion: v1_17
>     flinkConfiguration:
>         taskmanager.numberOfTaskSlots: "1"
>         state.savepoints.dir: hdfs://...
>         state.checkpoints.dir: hdfs://...
>     serviceAccount: flink
>     jobManager:
>         resource:
>             memory: "1024m"
>             cpu: 0.5
>     taskManager:
>         resource:
>             memory: "1024m"
>             cpu: 0.5
>     job:
>         jarURI: local:///opt/flink/opt/flink-python_2.12-1.17.0.jar
>         entryClass: "org.apache.flink.client.python.PythonDriver"
>         args: ["python", "-pym", "main", "-pyfs", "/project/"]
>         parallelism: 1
>         upgradeMode: savepoint
>         state: running
>
> Any help would be greatly appreciated. I'm using Flink v1.17 and Flink 
> Kubernetes Operator v1.7.0.
>
> [1]https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/overview/#application-mode<https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/deployment/overview/#application-mode),>
>
>
> Thanks,
> Sunny
>