Re: Flink 1.8 run参数不一样

2019-07-24 文章 王佩 
问题解决了，非常感谢！

解决流程：

1、确实在log/下找到了Could not load CLI class
org.apache.flink.yarn.cli.FlinkYarnSessionCli.异常

2、设置 export HADOOP_CONF_DIR=`hadoop classpath`

3、重新运行 bin/flink run --help ，出现了`Options for yarn-cluster mode` 选项

感谢大佬！❤❤❤

Zili Chen  于2019年7月24日周三 上午9:51写道：

> 你好，可以查看下 log/ 目录下的相关日志有没有这样一段
>
> 2019-07-24 09:34:36,507 WARN  org.apache.flink.client.cli.CliFrontend
> - Could not load CLI class
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.
>
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/yarn/exceptions/YarnException
>
> at java.lang.Class.forName0(Native Method)
>
> at java.lang.Class.forName(Class.java:264)
>
> at
>
> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1187)
>
> at
>
> org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1147)
>
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1072)
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.yarn.exceptions.YarnException
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
>
> ... 5 more
>
>
> 如果有的话，现在 Flink 把核心包和 hadoop 的 pre-bundled 包分开 release，需要你单独下载 pre-bundled 的
> hadoop 然后放到 lib/ 文件夹里。
>
>
> 具体地，请仔细阅读下载页面[1] "Apache Flink 1.8.1" 字样上面的文字内容。
>
> Best,
> tison.
>
> [1] https://flink.apache.org/downloads.html
>
>
> 王佩  于2019年7月24日周三 上午9:30写道：
>
> > 之前下载的Flink 1.8，运行bin/flink run --help，会有 yarn-cluster 的一些参数，如下:
> > Options for yarn-cluster mode:
> >  -d,--detachedIf present, runs the job in
> > detached
> >   mode
> >  -m,--jobmanager Address of the JobManager
> > (master) to
> >   which to connect. Use this flag
> > to
> >   connect to a different
> JobManager
> > than
> >   the one specified in the
> >   configuration.
> >  -sae,--shutdownOnAttachedExitIf the job is submitted in
> > attached
> >   mode, perform a best-effort
> > cluster
> >   shutdown when the CLI is
> > terminated
> >   abruptly, e.g., in response to
> a
> > user
> >   interrupt, such as typing Ctrl
> +
> > C.
> >  -yD  use value for given property
> >  -yd,--yarndetached   If present, runs the job in
> > detached
> >   mode (deprecated; use non-YARN
> >   specific option instead)
> >  -yh,--yarnhelp   Help for the Yarn session CLI.
> >  -yid,--yarnapplicationIdAttach to running YARN session
> >  -yj,--yarnjar   Path to Flink jar file
> >  -yjm,--yarnjobManagerMemory Memory for JobManager Container
> > with
> >   optional unit (default: MB)
> >  -yn,--yarncontainer Number of YARN container to
> > allocate
> >   (=Number of Task Managers)
> >  -ynl,--yarnnodeLabelSpecify YARN node label for the
> > YARN
> >   application
> >  -ynm,--yarnname Set a custom name for the
> > application
> >   on YARN
> >  -yq,--yarnquery  Display available YARN
> resources
> >   (memory, cores)
> >  -yqu,--yarnqueueSpecify YARN queue.
> >  -ys,--yarnslots Number of slots per TaskManager
> >  -yst,--yarnstreaming Start Flink in streaming mode
> >  -yt,--yarnship  Ship files in the specified
> > directory
> >   (t for transfer)
> >  -ytm,--yarntaskManagerMemoryMemory per TaskManager
> Container
> > with
> >   optional unit (default: MB)
> >  -yz,--yarnzookeeperNamespaceNamespace to create the
> Zookeeper
> >   sub-paths for high availability
> > mode
> >  -z,--zookeeperNamespace Namespace to create the
> Zookeeper
> >   sub-paths for high availability
> > mode
> >
> >
> > 现在下载的Flink 1.8，运行bin/flink run --help，总共只有如下参数，少了yarn-cluster选项:
> > Action "run" compiles and runs a program.
> >
> >   Syntax: run [OPTIONS]  
> >   "run" action options:
> >  -c,--classClass with the program entry
> > point
> >

Re: Flink 1.8 run参数不一样

2019-07-23 文章 Zili Chen 
你好，可以查看下 log/ 目录下的相关日志有没有这样一段

2019-07-24 09:34:36,507 WARN  org.apache.flink.client.cli.CliFrontend
- Could not load CLI class
org.apache.flink.yarn.cli.FlinkYarnSessionCli.

java.lang.NoClassDefFoundError:
org/apache/hadoop/yarn/exceptions/YarnException

at java.lang.Class.forName0(Native Method)

at java.lang.Class.forName(Class.java:264)

at
org.apache.flink.client.cli.CliFrontend.loadCustomCommandLine(CliFrontend.java:1187)

at
org.apache.flink.client.cli.CliFrontend.loadCustomCommandLines(CliFrontend.java:1147)

at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1072)

Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.yarn.exceptions.YarnException

at java.net.URLClassLoader.findClass(URLClassLoader.java:382)

at java.lang.ClassLoader.loadClass(ClassLoader.java:424)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)

at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

... 5 more


如果有的话，现在 Flink 把核心包和 hadoop 的 pre-bundled 包分开 release，需要你单独下载 pre-bundled 的
hadoop 然后放到 lib/ 文件夹里。


具体地，请仔细阅读下载页面[1] "Apache Flink 1.8.1" 字样上面的文字内容。

Best,
tison.

[1] https://flink.apache.org/downloads.html


王佩  于2019年7月24日周三 上午9:30写道：

> 之前下载的Flink 1.8，运行bin/flink run --help，会有 yarn-cluster 的一些参数，如下:
> Options for yarn-cluster mode:
>  -d,--detachedIf present, runs the job in
> detached
>   mode
>  -m,--jobmanager Address of the JobManager
> (master) to
>   which to connect. Use this flag
> to
>   connect to a different JobManager
> than
>   the one specified in the
>   configuration.
>  -sae,--shutdownOnAttachedExitIf the job is submitted in
> attached
>   mode, perform a best-effort
> cluster
>   shutdown when the CLI is
> terminated
>   abruptly, e.g., in response to a
> user
>   interrupt, such as typing Ctrl +
> C.
>  -yD  use value for given property
>  -yd,--yarndetached   If present, runs the job in
> detached
>   mode (deprecated; use non-YARN
>   specific option instead)
>  -yh,--yarnhelp   Help for the Yarn session CLI.
>  -yid,--yarnapplicationIdAttach to running YARN session
>  -yj,--yarnjar   Path to Flink jar file
>  -yjm,--yarnjobManagerMemory Memory for JobManager Container
> with
>   optional unit (default: MB)
>  -yn,--yarncontainer Number of YARN container to
> allocate
>   (=Number of Task Managers)
>  -ynl,--yarnnodeLabelSpecify YARN node label for the
> YARN
>   application
>  -ynm,--yarnname Set a custom name for the
> application
>   on YARN
>  -yq,--yarnquery  Display available YARN resources
>   (memory, cores)
>  -yqu,--yarnqueueSpecify YARN queue.
>  -ys,--yarnslots Number of slots per TaskManager
>  -yst,--yarnstreaming Start Flink in streaming mode
>  -yt,--yarnship  Ship files in the specified
> directory
>   (t for transfer)
>  -ytm,--yarntaskManagerMemoryMemory per TaskManager Container
> with
>   optional unit (default: MB)
>  -yz,--yarnzookeeperNamespaceNamespace to create the Zookeeper
>   sub-paths for high availability
> mode
>  -z,--zookeeperNamespace Namespace to create the Zookeeper
>   sub-paths for high availability
> mode
>
>
> 现在下载的Flink 1.8，运行bin/flink run --help，总共只有如下参数，少了yarn-cluster选项:
> Action "run" compiles and runs a program.
>
>   Syntax: run [OPTIONS]  
>   "run" action options:
>  -c,--classClass with the program entry
> point
>   ("main" method or "getPlan()"
> method.
>   Only needed if the JAR file does
> not
>   specify the class in its
> manifest.
>  -C,--classpath  Adds a URL to each user code
>   classloader  on all nodes in the
>   cluster. The paths must specify a
>   protocol (e.g. file://)

Flink 1.8 run参数不一样

2019-07-23 文章 王佩 
之前下载的Flink 1.8，运行bin/flink run --help，会有 yarn-cluster 的一些参数，如下:
Options for yarn-cluster mode:
 -d,--detachedIf present, runs the job in
detached
  mode
 -m,--jobmanager Address of the JobManager
(master) to
  which to connect. Use this flag to
  connect to a different JobManager
than
  the one specified in the
  configuration.
 -sae,--shutdownOnAttachedExitIf the job is submitted in
attached
  mode, perform a best-effort
cluster
  shutdown when the CLI is
terminated
  abruptly, e.g., in response to a
user
  interrupt, such as typing Ctrl +
C.
 -yD  use value for given property
 -yd,--yarndetached   If present, runs the job in
detached
  mode (deprecated; use non-YARN
  specific option instead)
 -yh,--yarnhelp   Help for the Yarn session CLI.
 -yid,--yarnapplicationIdAttach to running YARN session
 -yj,--yarnjar   Path to Flink jar file
 -yjm,--yarnjobManagerMemory Memory for JobManager Container
with
  optional unit (default: MB)
 -yn,--yarncontainer Number of YARN container to
allocate
  (=Number of Task Managers)
 -ynl,--yarnnodeLabelSpecify YARN node label for the
YARN
  application
 -ynm,--yarnname Set a custom name for the
application
  on YARN
 -yq,--yarnquery  Display available YARN resources
  (memory, cores)
 -yqu,--yarnqueueSpecify YARN queue.
 -ys,--yarnslots Number of slots per TaskManager
 -yst,--yarnstreaming Start Flink in streaming mode
 -yt,--yarnship  Ship files in the specified
directory
  (t for transfer)
 -ytm,--yarntaskManagerMemoryMemory per TaskManager Container
with
  optional unit (default: MB)
 -yz,--yarnzookeeperNamespaceNamespace to create the Zookeeper
  sub-paths for high availability
mode
 -z,--zookeeperNamespace Namespace to create the Zookeeper
  sub-paths for high availability
mode


现在下载的Flink 1.8，运行bin/flink run --help，总共只有如下参数，少了yarn-cluster选项:
Action "run" compiles and runs a program.

  Syntax: run [OPTIONS]  
  "run" action options:
 -c,--classClass with the program entry point
  ("main" method or "getPlan()"
method.
  Only needed if the JAR file does
not
  specify the class in its manifest.
 -C,--classpath  Adds a URL to each user code
  classloader  on all nodes in the
  cluster. The paths must specify a
  protocol (e.g. file://) and be
  accessible on all nodes (e.g. by
means
  of a NFS share). You can use this
  option multiple times for
specifying
  more than one URL. The protocol
must
  be supported by the {@link
  java.net.URLClassLoader}.
 -d,--detachedIf present, runs the job in
detached
  mode
 -n,--allowNonRestoredState   Allow to skip savepoint state that
  cannot be restored. You need to
allow
  this if you removed an operator
from
  your program that was part of the
  program when the savepoint was
  triggered.
 -p,--parallelismThe parallelism with which to run
the
  program. Optional flag to
override the
  default value specified in the
  configuration.
 -q,--sysoutLogging   If present,