Hi Experts,
I recently tried to run yarn-application mode on my yarn cluster, and I had
a problem related to configuring `execution.target`.
After reading the source code and doing some experiments, I found that
there should be some room of improvement for `FlinkYarnSessionCli` or
`AbstractYarnCli`.
My experiments are:
1. setting `execution.target: yarn-application` in flink-conf.yaml and
run `flink run-application -t yarn-application`: run job successfully.
1. `FlinkYarnSessionCli` is not active
2. `GenericCLI` is active
2. setting `execution.target: yarn-per-job` in flink-conf.yaml and
run `flink
run-application -t yarn-application`: run job failed
1. failed due to `ClusterDeploymentException` [1]
2. `FlinkYarnSessionCli` is active
3. setting `execution.target: yarn-application` in flink-conf.yaml and
run `flink run -t yarn-per-job`: run job successfully.
1. `FlinkYarnSessionCli` is not active
2. `GenericCLI` is active
4. setting `execution.target: yarn-per-job` in flink-conf.yaml and
run `flink
run -t yarn-per-job`: run job successfully.
1. `FlinkYarnSessionCli` is active
>From `AbstractYarnCli#isActive` [2] and `FlinkYarnSessionCli#isActive` [3],
`FlinkYarnSessionCli` will be active when `execution.target` is specified
with `yarn-per-job` or `yarn-session`.
According to the flink official document [4], I thought the 2nd experiment
should also work well, but it didn't.
> The --target will overwrite the execution.target
> <https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/config.html#execution-target>
> specified
> in the config/flink-config.yaml.
The root cause is that `FlinkYarnSessionCli` only overwrite the
`execution.target` with `yarn-session` or `yarn-per-job` [5], but no
`yarn-application`.
So, my question is
1. should we use `FlinkYarnSessionCli` in case 2?
2. if we should, how we can improve `FlinkYarnSessionCli` so that we can
overwrite `execution.target` via `--target`?
and one more improvement, the config description for `execution.target` [6]
should include `yarn-application` as well.
[1]
https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L439-L447
[2]
https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/AbstractYarnCli.java#L54-L66
[3]
https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L373-L377
[4]
https://ci.apache.org/projects/flink/flink-docs-stable/deployment/cli.html#selecting-deployment-targets
[5]
https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L397-L413
[6]
https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptions.java#L41-L46
best regards,
