Thanks a lot for posting the full output. It seems that Flink is passing an invalid list of arguments to the JVM. Can you - set the root log level in conf/log4j-yarn-session.properties to DEBUG - then launch the YARN session - share the log file of the yarn session on the mailing list?
I'm particularly interested in the line printed here, as it shows the JVM invocation. https://github.com/apache/flink/blob/release-1.6/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L1630 On Mon, May 11, 2020 at 9:56 PM Jacky D <jacky.du0...@gmail.com> wrote: > Hi，Robert > > Yes , I tried to retrieve more log info from yarn UI , the full logs > showing below , this happens when I try to create a flink yarn session on > emr when set up jitwatch configuration . > > 2020-05-11 19:06:09,552 ERROR > org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error while > running the Flink Yarn session. > java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862) > at > org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) > at > org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:813) > Caused by: org.apache.flink.client.deployment.ClusterDeploymentException: > Couldn't deploy Yarn session cluster > at > org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:429) > at > org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:610) > at > org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:813) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) > ... 2 more > Caused by: > org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException: > The YARN application unexpectedly switched to state FAILED during > deployment. > Diagnostics from YARN: Application application_1584459865196_0165 failed 1 > times (global limit =2; local limit is =1) due to AM Container for > appattempt_1584459865196_0165_000001 exited with exitCode: 1 > Failing this attempt.Diagnostics: Exception from container-launch. > Container id: container_1584459865196_0165_01_000001 > Exit code: 1 > Exception message: Usage: java [-options] class [args...] > (to execute a class) > or java [-options] -jar jarfile [args...] > (to execute a jar file) > where options include: > -d32 use a 32-bit data model if available > -d64 use a 64-bit data model if available > -server to select the "server" VM > The default VM is server, > because you are running on a server-class machine. > > > -cp <class search path of directories and zip/jar files> > -classpath <class search path of directories and zip/jar files> > A : separated list of directories, JAR archives, > and ZIP archives to search for class files. > -D<name>=<value> > set a system property > -verbose:[class|gc|jni] > enable verbose output > -version print product version and exit > -version:<value> > Warning: this feature is deprecated and will be removed > in a future release. > require the specified version to run > -showversion print product version and continue > -jre-restrict-search | -no-jre-restrict-search > Warning: this feature is deprecated and will be removed > in a future release. > include/exclude user private JREs in the version search > -? -help print this help message > -X print help on non-standard options > -ea[:<packagename>...|:<classname>] > -enableassertions[:<packagename>...|:<classname>] > enable assertions with specified granularity > -da[:<packagename>...|:<classname>] > -disableassertions[:<packagename>...|:<classname>] > disable assertions with specified granularity > -esa | -enablesystemassertions > enable system assertions > -dsa | -disablesystemassertions > disable system assertions > -agentlib:<libname>[=<options>] > load native agent library <libname>, e.g. -agentlib:hprof > see also, -agentlib:jdwp=help and -agentlib:hprof=help > -agentpath:<pathname>[=<options>] > load native agent library by full pathname > -javaagent:<jarpath>[=<options>] > load Java programming language agent, see > java.lang.instrument > -splash:<imagepath> > show splash screen with specified image > See http://www.oracle.com/technetwork/java/javase/documentation/index.html > for more details. > > Thanks > Jacky > > Robert Metzger <rmetz...@apache.org> 于2020年5月11日周一 下午3:42写道： > >> Hey Jacky, >> >> The error says "The YARN application unexpectedly switched to state >> FAILED during deployment.". >> Have you tried retrieving the YARN application logs? >> Does the YARN UI / resource manager logs reveal anything on the reason >> for the deployment to fail? >> >> Best, >> Robert >> >> >> On Mon, May 11, 2020 at 9:34 PM Jacky D <jacky.du0...@gmail.com> wrote: >> >>> >>> >>> ---------- Forwarded message --------- >>> 发件人： Jacky D <jacky.du0...@gmail.com> >>> Date: 2020年5月11日周一 下午3:12 >>> Subject: Re: Flink Memory analyze on AWS EMR >>> To: Khachatryan Roman <khachatryan.ro...@gmail.com> >>> >>> >>> Hi, Roman >>> >>> Thanks for quick response , I tried without logFIle option but failed >>> with same error , I'm currently using flink 1.6 >>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >>> so I can only use Jitwatch or JMC . I guess those tools only available on >>> Standalone cluster ? as document mentioned "Each standalone JobManager, >>> TaskManager, HistoryServer, and ZooKeeper daemon redirects stdout and >>> stderr to a file with a .out filename suffix and writes internal >>> logging to a file with a .log suffix. Java options configured by the >>> user in env.java.opts" ? >>> >>> Thanks >>> Jacky >>> >>
