Hi, Tony. What is the version of your flink-dist. AFAIK, this issue should be addressed in FLINK-15852[1]. Could you give the client log of case 2(set the log level to DEBUG would be better).
[1] https://issues.apache.org/jira/browse/FLINK-15852 Best, Yangze Guo On Sun, Apr 25, 2021 at 11:33 AM Tony Wei <tony19920...@gmail.com> wrote: > > Hi Experts, > > I recently tried to run yarn-application mode on my yarn cluster, and I had a > problem related to configuring `execution.target`. > After reading the source code and doing some experiments, I found that there > should be some room of improvement for `FlinkYarnSessionCli` or > `AbstractYarnCli`. > > My experiments are: > > setting `execution.target: yarn-application` in flink-conf.yaml and run > `flink run-application -t yarn-application`: run job successfully. > > `FlinkYarnSessionCli` is not active > `GenericCLI` is active > > setting `execution.target: yarn-per-job` in flink-conf.yaml and run `flink > run-application -t yarn-application`: run job failed > > failed due to `ClusterDeploymentException` [1] > `FlinkYarnSessionCli` is active > > setting `execution.target: yarn-application` in flink-conf.yaml and run > `flink run -t yarn-per-job`: run job successfully. > > `FlinkYarnSessionCli` is not active > `GenericCLI` is active > > setting `execution.target: yarn-per-job` in flink-conf.yaml and run `flink > run -t yarn-per-job`: run job successfully. > > `FlinkYarnSessionCli` is active > > From `AbstractYarnCli#isActive` [2] and `FlinkYarnSessionCli#isActive` [3], > `FlinkYarnSessionCli` will be active when `execution.target` is specified > with `yarn-per-job` or `yarn-session`. > > According to the flink official document [4], I thought the 2nd experiment > should also work well, but it didn't. >> >> The --target will overwrite the execution.target specified in the >> config/flink-config.yaml. > > > The root cause is that `FlinkYarnSessionCli` only overwrite the > `execution.target` with `yarn-session` or `yarn-per-job` [5], but no > `yarn-application`. > So, my question is > > should we use `FlinkYarnSessionCli` in case 2? > if we should, how we can improve `FlinkYarnSessionCli` so that we can > overwrite `execution.target` via `--target`? > > and one more improvement, the config description for `execution.target` [6] > should include `yarn-application` as well. > > [1] > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L439-L447 > [2] > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/AbstractYarnCli.java#L54-L66 > [3] > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L373-L377 > [4] > https://ci.apache.org/projects/flink/flink-docs-stable/deployment/cli.html#selecting-deployment-targets > [5] > https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L397-L413 > [6] > https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/configuration/DeploymentOptions.java#L41-L46 > > best regards, >
