[ https://issues.apache.org/jira/browse/FLINK-16305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17046975#comment-17046975 ]
Daniel Laszlo Magyar commented on FLINK-16305: ---------------------------------------------- Thank you guys for your insights, much appreciated. I don't have the fix yet and wasn't aware of the possibility of leveraging {{ExecutorCLI}}, I'm definitely going to test it, thanks [~fly_in_gis] for the pointer! Either way, I would like to leave FlinkYarnSessionCli in a bit better shape and at the very least check if an executor is configured already explicitly and not overwrite it to yarn-session. > FlinkYarnSessionClI ignores target executor and uses yarn-session if YARN > properties file is present > ---------------------------------------------------------------------------------------------------- > > Key: FLINK-16305 > URL: https://issues.apache.org/jira/browse/FLINK-16305 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission, Command Line Client > Affects Versions: 1.10.0 > Reporter: Daniel Laszlo Magyar > Assignee: Daniel Laszlo Magyar > Priority: Major > > The presence of the hidden YARN property file (which contains the last > started YARN session’s application id), causes the cli to ignore the > {{execution.target}} property set in the {{conf/flink-conf.yaml}} > configuration file, which leads to unexpected behaviour at the time of job > submission via cli, e.g. when using {{flink run}} or SQL client. > The code that ignores the execution target if the YARN application id is set > in the hidden property file is at > [https://github.com/apache/flink/blob/release-1.10/flink-yarn/src/main/java/org/apache/flink/yarn/cli/FlinkYarnSessionCli.java#L337-L351]. > > Reproduction steps: > • start flink yarn session via {{./bin/yarn-session.sh -d}}, this writes the > application id to {{/tmp/.yarn-properties-root}} > • set {{execution.target: yarn-per-job}} in > {{/etc/flink/conf/flink-conf.yaml}} > • enable debug logging > • run a flink job e.g. {{flink run -d -p 2 examples/streaming/WordCount.jar > --input README.txt}} > • the logs below show that even though the {{execution.target}} property is > read properly, {{FlinkYarnSessionCli}} is chosen and the execution.target is > reset to yarn-session > {code:java} > 20/02/26 12:14:24 INFO configuration.GlobalConfiguration: Loading > configuration property: execution.target, yarn-per-job > ... > 20/02/26 12:14:24 INFO cli.FlinkYarnSessionCli: Found Yarn properties file > under /tmp/.yarn-properties-root. > 20/02/26 12:14:24 DEBUG fs.FileSystem: Loading extension file systems via > services > 20/02/26 12:14:24 DEBUG cli.CliFrontend: Custom commandlines: > [org.apache.flink.yarn.cli.FlinkYarnSessionCli@43df23d3, > org.apache.flink.client.cli.ExecutorCLI@6d60fe40, > org.apache.flink.client.cli.DefaultCLI@792b749c] > 20/02/26 12:14:24 DEBUG cli.CliFrontend: Checking custom commandline > org.apache.flink.yarn.cli.FlinkYarnSessionCli@43df23d3, isActive: true > ... > 20/02/26 12:14:25 DEBUG cli.CliFrontend: Effective executor configuration: > {...execution.target=yarn-session, ....} > 20/02/26 12:14:25 INFO client.ClientUtils: Starting program (detached: true) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)