Re: flink命令行参数不生效问题
我知道的比较相关的 FLIP 和 JIRA 有下面两个: 1. https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission FLIP-73: Introducing Executors for job submission - Apache Flink - The Apache Software Foundation<https://cwiki.apache.org/confluence/display/FLINK/FLIP-73%3A+Introducing+Executors+for+job+submission> Status. Current state: Accepted. Discussion thread: https://lists.apache.org/thread.html/dc3a541709f96906b43df4155373af1cd09e08c3f105b0bd0ba3fca2@%3Cdev.flink.apache ... cwiki.apache.org 2. https://issues.apache.org/jira/browse/FLINK-15179 特别是这个 JIRA 的评论区可以看一下 Best, Zhanghao Chen From: QQ <1139872...@qq.com.INVALID> Sent: Tuesday, April 12, 2022 9:19 To: user-zh@flink.apache.org Subject: Re: flink命令行参数不生效问题 非常感谢解答。关于这套新的统一的cli命令的说明或者FLIP在哪里? > 2022年4月11日 下午11:49,Zhanghao Chen 写道: > > 你好,-m 配合 -yxx 的参数是早期 Flink on YARN 的 cli 参数用法,后来社区开始推进一套新的统一的 cli 命令,使用 -t > 指定部署形式,并将原先的 cli options 统一动态参数化,比如原先的 -yxx 命令都能从 > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn > 找到替代的动态参数。 > Configuration | Apache > Flink<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn> > Key Default Type Description; restart-strategy.fixed-delay.attempts: 1: > Integer: The number of times that Flink retries the execution before the job > is declared as failed if restart-strategy has been set to fixed-delay.: > restart-strategy.fixed-delay.delay > nightlies.apache.org > > > Best, > Zhanghao Chen > > From: gangzi <1139872...@qq.com.INVALID> > Sent: Monday, April 11, 2022 19:55 > To: user-zh > Subject: flink命令行参数不生效问题 > > 我用命令提交作业:flink run -t yarn-per-job -ynm SocketWordCount -yqu root.root > -d -n SocketWindowWordCount.jar --hostname 10.199.0.97 --port > 9878。结果作业提交成功之后发现 -ynm和-yqu不生效。后来通过查看源码发现是因为如果指定了 > -t,那么-y开头的所有参数都不生效了,因为-y系列参数是在FlinkYarnSessionCli中解析的,而源码中:public > CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) { > LOG.debug("Custom commandlines: {}", customCommandLines); > for (CustomCommandLine cli : customCommandLines) { > LOG.debug( > "Checking custom > commandline {}, isActive: {}", cli, cli.isActive(commandLine)); > if (cli.isActive(commandLine)) { > return cli; > } > } > throw new IllegalStateException("No valid command-line found."); > } > 这段代码返回的是GenericCLI。导致后面的: > final Configuration effectiveConfiguration = > getEffectiveConfiguration(activeCommandLine, > commandLine, programOptions, jobJars); > 这行代码返回的命令行参数配置只包含了GenericCli中定义的参数。想请教一下,-t和-m设置参数时有啥区别?如何解决上述参数不生效的问题?这是一个bug么?
Re: flink命令行参数不生效问题
非常感谢解答。关于这套新的统一的cli命令的说明或者FLIP在哪里? > 2022年4月11日 下午11:49,Zhanghao Chen 写道: > > 你好,-m 配合 -yxx 的参数是早期 Flink on YARN 的 cli 参数用法,后来社区开始推进一套新的统一的 cli 命令,使用 -t > 指定部署形式,并将原先的 cli options 统一动态参数化,比如原先的 -yxx 命令都能从 > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn > 找到替代的动态参数。 > Configuration | Apache > Flink<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn> > Key Default Type Description; restart-strategy.fixed-delay.attempts: 1: > Integer: The number of times that Flink retries the execution before the job > is declared as failed if restart-strategy has been set to fixed-delay.: > restart-strategy.fixed-delay.delay > nightlies.apache.org > > > Best, > Zhanghao Chen > > From: gangzi <1139872...@qq.com.INVALID> > Sent: Monday, April 11, 2022 19:55 > To: user-zh > Subject: flink命令行参数不生效问题 > > 我用命令提交作业:flink run -t yarn-per-job -ynm SocketWordCount -yqu root.root > -d -n SocketWindowWordCount.jar --hostname 10.199.0.97 --port > 9878。结果作业提交成功之后发现 -ynm和-yqu不生效。后来通过查看源码发现是因为如果指定了 > -t,那么-y开头的所有参数都不生效了,因为-y系列参数是在FlinkYarnSessionCli中解析的,而源码中:public > CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) { > LOG.debug("Custom commandlines: {}", customCommandLines); > for (CustomCommandLine cli : customCommandLines) { > LOG.debug( > "Checking custom > commandline {}, isActive: {}", cli, cli.isActive(commandLine)); > if (cli.isActive(commandLine)) { > return cli; > } > } > throw new IllegalStateException("No valid command-line found."); > } > 这段代码返回的是GenericCLI。导致后面的: > final Configuration effectiveConfiguration = > getEffectiveConfiguration(activeCommandLine, > commandLine, programOptions, jobJars); > 这行代码返回的命令行参数配置只包含了GenericCli中定义的参数。想请教一下,-t和-m设置参数时有啥区别?如何解决上述参数不生效的问题?这是一个bug么?
Re: flink命令行参数不生效问题
你好,-m 配合 -yxx 的参数是早期 Flink on YARN 的 cli 参数用法,后来社区开始推进一套新的统一的 cli 命令,使用 -t 指定部署形式,并将原先的 cli options 统一动态参数化,比如原先的 -yxx 命令都能从 https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn 找到替代的动态参数。 Configuration | Apache Flink<https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/config/#yarn> Key Default Type Description; restart-strategy.fixed-delay.attempts: 1: Integer: The number of times that Flink retries the execution before the job is declared as failed if restart-strategy has been set to fixed-delay.: restart-strategy.fixed-delay.delay nightlies.apache.org Best, Zhanghao Chen From: gangzi <1139872...@qq.com.INVALID> Sent: Monday, April 11, 2022 19:55 To: user-zh Subject: flink命令行参数不生效问题 我用命令提交作业:flink run -t yarn-per-job -ynm SocketWordCount -yqu root.root -d -n SocketWindowWordCount.jar --hostname 10.199.0.97 --port 9878。结果作业提交成功之后发现 -ynm和-yqu不生效。后来通过查看源码发现是因为如果指定了 -t,那么-y开头的所有参数都不生效了,因为-y系列参数是在FlinkYarnSessionCli中解析的,而源码中:public CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) { LOG.debug("Custom commandlines: {}", customCommandLines); for (CustomCommandLine cli : customCommandLines) { LOG.debug( "Checking custom commandline {}, isActive: {}", cli, cli.isActive(commandLine)); if (cli.isActive(commandLine)) { return cli; } } throw new IllegalStateException("No valid command-line found."); } 这段代码返回的是GenericCLI。导致后面的: final Configuration effectiveConfiguration = getEffectiveConfiguration(activeCommandLine, commandLine, programOptions, jobJars); 这行代码返回的命令行参数配置只包含了GenericCli中定义的参数。想请教一下,-t和-m设置参数时有啥区别?如何解决上述参数不生效的问题?这是一个bug么?
flink命令行参数不生效问题
我用命令提交作业:flink run -t yarn-per-job -ynm SocketWordCount -yqu root.root -d -n SocketWindowWordCount.jar --hostname 10.199.0.97 --port 9878。结果作业提交成功之后发现 -ynm和-yqu不生效。后来通过查看源码发现是因为如果指定了 -t,那么-y开头的所有参数都不生效了,因为-y系列参数是在FlinkYarnSessionCli中解析的,而源码中:public CustomCommandLine validateAndGetActiveCommandLine(CommandLine commandLine) { LOG.debug("Custom commandlines: {}", customCommandLines); for (CustomCommandLine cli : customCommandLines) { LOG.debug( "Checking custom commandline {}, isActive: {}", cli, cli.isActive(commandLine)); if (cli.isActive(commandLine)) { return cli; } } throw new IllegalStateException("No valid command-line found."); } 这段代码返回的是GenericCLI。导致后面的: final Configuration effectiveConfiguration = getEffectiveConfiguration(activeCommandLine, commandLine, programOptions, jobJars); 这行代码返回的命令行参数配置只包含了GenericCli中定义的参数。想请教一下,-t和-m设置参数时有啥区别?如何解决上述参数不生效的问题?这是一个bug么?