Hi Jacky, Could you search for "Application Master start command:" in the debug log and post the result and a few lines before & after that? This is not included in the clip of attached log file.
Thank you~ Xintong Song On Tue, May 12, 2020 at 5:33 AM Jacky D <jacky.du0...@gmail.com> wrote: > hi, Robert > > Thanks so much for quick reply , I changed the log level to debug and > attach the log file . > > Thanks > Jacky > > Robert Metzger <rmetz...@apache.org> 于2020年5月11日周一 下午4:14写道： > >> Thanks a lot for posting the full output. >> >> It seems that Flink is passing an invalid list of arguments to the JVM. >> Can you >> - set the root log level in conf/log4j-yarn-session.properties to DEBUG >> - then launch the YARN session >> - share the log file of the yarn session on the mailing list? >> >> I'm particularly interested in the line printed here, as it shows the JVM >> invocation. >> >> https://github.com/apache/flink/blob/release-1.6/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L1630 >> >> >> On Mon, May 11, 2020 at 9:56 PM Jacky D <jacky.du0...@gmail.com> wrote: >> >>> Hi，Robert >>> >>> Yes , I tried to retrieve more log info from yarn UI , the full logs >>> showing below , this happens when I try to create a flink yarn session on >>> emr when set up jitwatch configuration . >>> >>> 2020-05-11 19:06:09,552 ERROR >>> org.apache.flink.yarn.cli.FlinkYarnSessionCli - Error while >>> running the Flink Yarn session. >>> java.lang.reflect.UndeclaredThrowableException >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1862) >>> at >>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >>> at >>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:813) >>> Caused by: >>> org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't >>> deploy Yarn session cluster >>> at >>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deploySessionCluster(AbstractYarnClusterDescriptor.java:429) >>> at >>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:610) >>> at >>> org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$2(FlinkYarnSessionCli.java:813) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:422) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) >>> ... 2 more >>> Caused by: >>> org.apache.flink.yarn.AbstractYarnClusterDescriptor$YarnDeploymentException: >>> The YARN application unexpectedly switched to state FAILED during >>> deployment. >>> Diagnostics from YARN: Application application_1584459865196_0165 failed >>> 1 times (global limit =2; local limit is =1) due to AM Container for >>> appattempt_1584459865196_0165_000001 exited with exitCode: 1 >>> Failing this attempt.Diagnostics: Exception from container-launch. >>> Container id: container_1584459865196_0165_01_000001 >>> Exit code: 1 >>> Exception message: Usage: java [-options] class [args...] >>> (to execute a class) >>> or java [-options] -jar jarfile [args...] >>> (to execute a jar file) >>> where options include: >>> -d32 use a 32-bit data model if available >>> -d64 use a 64-bit data model if available >>> -server to select the "server" VM >>> The default VM is server, >>> because you are running on a server-class machine. >>> >>> >>> -cp <class search path of directories and zip/jar files> >>> -classpath <class search path of directories and zip/jar files> >>> A : separated list of directories, JAR archives, >>> and ZIP archives to search for class files. >>> -D<name>=<value> >>> set a system property >>> -verbose:[class|gc|jni] >>> enable verbose output >>> -version print product version and exit >>> -version:<value> >>> Warning: this feature is deprecated and will be removed >>> in a future release. >>> require the specified version to run >>> -showversion print product version and continue >>> -jre-restrict-search | -no-jre-restrict-search >>> Warning: this feature is deprecated and will be removed >>> in a future release. >>> include/exclude user private JREs in the version search >>> -? -help print this help message >>> -X print help on non-standard options >>> -ea[:<packagename>...|:<classname>] >>> -enableassertions[:<packagename>...|:<classname>] >>> enable assertions with specified granularity >>> -da[:<packagename>...|:<classname>] >>> -disableassertions[:<packagename>...|:<classname>] >>> disable assertions with specified granularity >>> -esa | -enablesystemassertions >>> enable system assertions >>> -dsa | -disablesystemassertions >>> disable system assertions >>> -agentlib:<libname>[=<options>] >>> load native agent library <libname>, e.g. >>> -agentlib:hprof >>> see also, -agentlib:jdwp=help and -agentlib:hprof=help >>> -agentpath:<pathname>[=<options>] >>> load native agent library by full pathname >>> -javaagent:<jarpath>[=<options>] >>> load Java programming language agent, see >>> java.lang.instrument >>> -splash:<imagepath> >>> show splash screen with specified image >>> See >>> http://www.oracle.com/technetwork/java/javase/documentation/index.html >>> for more details. >>> >>> Thanks >>> Jacky >>> >>> Robert Metzger <rmetz...@apache.org> 于2020年5月11日周一 下午3:42写道： >>> >>>> Hey Jacky, >>>> >>>> The error says "The YARN application unexpectedly switched to state >>>> FAILED during deployment.". >>>> Have you tried retrieving the YARN application logs? >>>> Does the YARN UI / resource manager logs reveal anything on the reason >>>> for the deployment to fail? >>>> >>>> Best, >>>> Robert >>>> >>>> >>>> On Mon, May 11, 2020 at 9:34 PM Jacky D <jacky.du0...@gmail.com> wrote: >>>> >>>>> >>>>> >>>>> ---------- Forwarded message --------- >>>>> 发件人： Jacky D <jacky.du0...@gmail.com> >>>>> Date: 2020年5月11日周一 下午3:12 >>>>> Subject: Re: Flink Memory analyze on AWS EMR >>>>> To: Khachatryan Roman <khachatryan.ro...@gmail.com> >>>>> >>>>> >>>>> Hi, Roman >>>>> >>>>> Thanks for quick response , I tried without logFIle option but failed >>>>> with same error , I'm currently using flink 1.6 >>>>> https://ci.apache.org/projects/flink/flink-docs-release-1.6/monitoring/application_profiling.html, >>>>> so I can only use Jitwatch or JMC . I guess those tools only available on >>>>> Standalone cluster ? as document mentioned "Each standalone >>>>> JobManager, TaskManager, HistoryServer, and ZooKeeper daemon redirects >>>>> stdout and stderr to a file with a .out filename suffix and writes >>>>> internal logging to a file with a .log suffix. Java options >>>>> configured by the user in env.java.opts" ? >>>>> >>>>> Thanks >>>>> Jacky >>>>> >>>>
