Could you check which xerces version you have on your classpath? Apparently, it cannot read core-default.xml as Ted pointed out. This might be the root cause for the failure.
Cheers, Till On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <garvit...@gmail.com> wrote: > Hi, > > Sorry for the confusion, but the yarn is running on Hadoop version 2.7 > only and hence I am using Flink 1.5 Hadoop 2.7 binary. > > Below are the details provided by Yarn version command : > > Hadoop 2.7.3.2.6.3.0-235 > Subversion g...@github.com:hortonworks/hadoop.git -r > 45bfd33bba8acadfa0e6024c80981c023b28d454 > Compiled by jenkins on 2017-10-30T02:31Z > Compiled with protoc 2.5.0 > From source with checksum cd1a4a466ef450f547c279989f3aa3 > This command was run using > /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar > > Please let me know if you have found the resolution to my issue :) > > Thanks, > > > On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <trohrm...@apache.org> > wrote: > >> Which Hadoop version have you installed? It looks as if Flink has been >> build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If >> you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink >> binaries or the one built for Hadoop 2.6. >> >> Cheers, >> Till >> >> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <garvit...@gmail.com> >> wrote: >> >>> Ok, I have attached the log file. >>> >>> Please check and let me know. >>> >>> Thanks, >>> >>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj201...@gmail.com> wrote: >>> >>>> Hi Gravit, >>>> >>>> I think Till is interested to know about classpath details present at >>>> the start of JM and TM logs e.g. following logs provide classpath details >>>> used by TM in our case. >>>> >>>> 2018-06-17 19:01:30,656 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> -------------------------------------------------------------------------------- >>>> 2018-06-17 19:01:30,658 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Starting >>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @ >>>> 14:54:44 UTC) >>>> 2018-06-17 19:01:30,659 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS >>>> current user: yarn >>>> 2018-06-17 19:01:31,662 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Current >>>> Hadoop/Kerberos user: hadoop >>>> 2018-06-17 19:01:31,663 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM: >>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10 >>>> 2018-06-17 19:01:31,663 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Maximum >>>> heap size: 6647 MiBytes >>>> 2018-06-17 19:01:31,663 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> JAVA_HOME: /usr/lib/jvm/java-openjdk >>>> 2018-06-17 19:01:31,664 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop >>>> version: 2.8.3 >>>> 2018-06-17 19:01:31,664 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM >>>> Options: >>>> 2018-06-17 19:01:31,665 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> -Xms6936m >>>> 2018-06-17 19:01:31,665 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> -Xmx6936m >>>> 2018-06-17 19:01:31,665 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> -XX:MaxDirectMemorySize=4072m >>>> 2018-06-17 19:01:31,665 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log >>>> 2018-06-17 19:01:31,665 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> -Dlogback.configurationFile=file:./logback.xml >>>> 2018-06-17 19:01:31,665 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> -Dlog4j.configuration=file:./log4j.properties >>>> 2018-06-17 19:01:31,665 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Program >>>> Arguments: >>>> 2018-06-17 19:01:31,665 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> --configDir >>>> 2018-06-17 19:01:31,665 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - . >>>> *2018-06-17 19:01:31,666 INFO >>>> org.apache.flink.yarn.YarnTaskExecutorRunner - >>>> Classpath: >>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........* >>>> >>>> -- >>>> Thanks, >>>> Amit >>>> >>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <garvit...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Please refer to my previous mail for complete logs. >>>>> >>>>> Thanks, >>>>> >>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <trohrm...@apache.org> >>>>> wrote: >>>>> >>>>>> Could you also please share the complete log file with us. >>>>>> >>>>>> Cheers, >>>>>> Till >>>>>> >>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yuzhih...@gmail.com> wrote: >>>>>> >>>>>>> The error for core-default.xml is interesting. >>>>>>> >>>>>>> Flink doesn't have this file. Probably it came with Yarn. Please >>>>>>> check the hadoop version Flink was built with versus the hadoop version >>>>>>> in >>>>>>> your cluster. >>>>>>> >>>>>>> Thanks >>>>>>> >>>>>>> -------- Original message -------- >>>>>>> From: Garvit Sharma <garvit...@gmail.com> >>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00) >>>>>>> To: trohrm...@apache.org >>>>>>> Cc: Chesnay Schepler <ches...@apache.org>, user@flink.apache.org >>>>>>> Subject: Re: Exception while submitting jobs through Yarn >>>>>>> >>>>>>> I am not able to figure out, got stuck badly in this since last 1 >>>>>>> week. Any little help would be appreciated. >>>>>>> >>>>>>> >>>>>>> 2018-06-16 19:25:10,523 DEBUG >>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - >>>>>>> Parallelism set: 1 for 8 >>>>>>> >>>>>>> 2018-06-16 19:25:10,578 DEBUG >>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - >>>>>>> Parallelism set: 1 for 1 >>>>>>> >>>>>>> 2018-06-16 19:25:10,588 DEBUG >>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - >>>>>>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8 >>>>>>> >>>>>>> 2018-06-16 19:25:10,591 DEBUG >>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - >>>>>>> Parallelism set: 1 for 5 >>>>>>> >>>>>>> 2018-06-16 19:25:10,597 DEBUG >>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator - >>>>>>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8 >>>>>>> >>>>>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration >>>>>>> - error parsing conf core-default.xml >>>>>>> >>>>>>> javax.xml.parsers.ParserConfigurationException: Feature ' >>>>>>> http://apache.org/xml/features/xinclude' is not recognized. >>>>>>> >>>>>>> at >>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown >>>>>>> Source) >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482) >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444) >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361) >>>>>>> >>>>>>> at org.apache.hadoop.conf.Configuration.get(Configuration.java:1188) >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49) >>>>>>> >>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32) >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495) >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235) >>>>>>> >>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096) >>>>>>> >>>>>>> at java.security.AccessController.doPrivileged(Native Method) >>>>>>> >>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422) >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096) >>>>>>> >>>>>>> 2018-06-16 19:25:10,620 WARN >>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor >>>>>>> - Error while getting queue information from YARN: null >>>>>>> >>>>>>> 2018-06-16 19:25:10,621 DEBUG >>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor - >>>>>>> Error details >>>>>>> >>>>>>> java.lang.ExceptionInInitializerError >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495) >>>>>>> >>>>>>> at >>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235) >>>>>>> >>>>>>> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020) >>>>>>> >>>>>>> at >>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096) >>>>>>> >>>>>> >>>>> >>>>> -- >>>>> >>>>> Garvit Sharma >>>>> github.com/garvitlnmiit/ >>>>> >>>>> No Body is a Scholar by birth, its only hard work and strong >>>>> determination that makes him master. >>>>> >>>> >>>> >>> >>> -- >>> >>> Garvit Sharma >>> github.com/garvitlnmiit/ >>> >>> No Body is a Scholar by birth, its only hard work and strong >>> determination that makes him master. >>> >> > > -- > > Garvit Sharma > github.com/garvitlnmiit/ > > No Body is a Scholar by birth, its only hard work and strong determination > that makes him master. >
