I don't think I can access core-default as it comes with Hadoop jar
On Mon, 18 Jun 2018 at 7:30 PM, Till Rohrmann <trohrm...@apache.org> wrote:
> Hmm, could you check whether core-default.xml contains any suspicious
> entries? Apparently xerces:2.9.1 cannot read it.
>
> On Mon, Jun 18, 2018 at 3:40 PM Garvit Sharma <garvit...@gmail.com> wrote:
>
>> Hi,
>>
>> After putting the following log in my code, I can see that the Xerces
>> version is - Xerces version : Xerces-J 2.9.1
>>
>> log.info("Xerces version : {}", org.apache.xerces.impl.Version.getVersion());
>>
>> Also, following is the response of *$* *locate xerces* command on the
>> server -
>>
>>
>> /usr/hdp/2.6.1.0-129/falcon/client/lib/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/hadoop/client/xercesImpl.jar
>>
>> /usr/hdp/2.6.1.0-129/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/hbase/lib/xercesImpl-2.9.1.jar
>>
>>
>> /usr/hdp/2.6.1.0-129/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/livy/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/livy2/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/oozie/lib/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.1.0-129/oozie/libserver/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.1.0-129/oozie/libtools/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.1.0-129/slider/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/spark2/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.1.0-129/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>
>> /usr/hdp/2.6.3.0-235/falcon/client/lib/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/hadoop/client/xercesImpl.jar
>>
>> /usr/hdp/2.6.3.0-235/hadoop-hdfs/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/hbase/lib/xercesImpl-2.9.1.jar
>>
>>
>> /usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/svr/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/livy/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/livy2/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/oozie/lib/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.3.0-235/oozie/libserver/xercesImpl-2.10.0.jar
>>
>> /usr/hdp/2.6.3.0-235/oozie/libtools/xercesImpl-2.10.0.jar
>>
>>
>> /usr/hdp/2.6.3.0-235/ranger-admin/ews/webapp/WEB-INF/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/slider/lib/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/spark2/jars/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/storm/contrib/storm-autocreds/xercesImpl-2.9.1.jar
>>
>> /usr/hdp/2.6.3.0-235/zookeeper/lib/xercesMinimal-1.9.6.2.jar
>>
>> /usr/hdp/share/hst/hst-common/lib/xercesImpl-2.9.1.jar
>>
>> Now, I can say that the version of xerces are same.
>>
>>
>> So, what is causing this issue if Xerces version is in sync?
>>
>>
>> I am very excited to discover the issue :)
>>
>>
>> Thanks,
>>
>> On Mon, Jun 18, 2018 at 6:27 PM Till Rohrmann <trohrm...@apache.org>
>> wrote:
>>
>>> Could you check which xerces version you have on your classpath?
>>> Apparently, it cannot read core-default.xml as Ted pointed out. This might
>>> be the root cause for the failure.
>>>
>>> Cheers,
>>> Till
>>>
>>> On Mon, Jun 18, 2018 at 1:31 PM Garvit Sharma <garvit...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Sorry for the confusion, but the yarn is running on Hadoop version 2.7
>>>> only and hence I am using Flink 1.5 Hadoop 2.7 binary.
>>>>
>>>> Below are the details provided by Yarn version command :
>>>>
>>>> Hadoop 2.7.3.2.6.3.0-235
>>>> Subversion g...@github.com:hortonworks/hadoop.git -r
>>>> 45bfd33bba8acadfa0e6024c80981c023b28d454
>>>> Compiled by jenkins on 2017-10-30T02:31Z
>>>> Compiled with protoc 2.5.0
>>>> From source with checksum cd1a4a466ef450f547c279989f3aa3
>>>> This command was run using
>>>> /usr/hdp/2.6.3.0-235/hadoop/hadoop-common-2.7.3.2.6.3.0-235.jar
>>>>
>>>> Please let me know if you have found the resolution to my issue :)
>>>>
>>>> Thanks,
>>>>
>>>>
>>>> On Mon, Jun 18, 2018 at 4:50 PM Till Rohrmann <trohrm...@apache.org>
>>>> wrote:
>>>>
>>>>> Which Hadoop version have you installed? It looks as if Flink has been
>>>>> build with Hadoop 2.7 but I see /usr/hdp/2.6.3.0-235 in the class path. If
>>>>> you want to run Flink on Hadoop 2.6, then try to use the Hadoop free Flink
>>>>> binaries or the one built for Hadoop 2.6.
>>>>>
>>>>> Cheers,
>>>>> Till
>>>>>
>>>>> On Mon, Jun 18, 2018 at 10:48 AM Garvit Sharma <garvit...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Ok, I have attached the log file.
>>>>>>
>>>>>> Please check and let me know.
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> On Mon, Jun 18, 2018 at 2:07 PM Amit Jain <aj201...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi Gravit,
>>>>>>>
>>>>>>> I think Till is interested to know about classpath details present
>>>>>>> at the start of JM and TM logs e.g. following logs provide classpath
>>>>>>> details used by TM in our case.
>>>>>>>
>>>>>>> 2018-06-17 19:01:30,656 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> --------------------------------------------------------------------------------
>>>>>>> 2018-06-17 19:01:30,658 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> Starting
>>>>>>> YARN TaskExecutor runner (Version: 1.5.0, Rev:c61b108, Date:24.05.2018 @
>>>>>>> 14:54:44 UTC)
>>>>>>> 2018-06-17 19:01:30,659 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - OS
>>>>>>> current user: yarn
>>>>>>> 2018-06-17 19:01:31,662 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> Current
>>>>>>> Hadoop/Kerberos user: hadoop
>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM:
>>>>>>> OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.171-b10
>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> Maximum
>>>>>>> heap size: 6647 MiBytes
>>>>>>> 2018-06-17 19:01:31,663 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> JAVA_HOME: /usr/lib/jvm/java-openjdk
>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - Hadoop
>>>>>>> version: 2.8.3
>>>>>>> 2018-06-17 19:01:31,664 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - JVM
>>>>>>> Options:
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Xms6936m
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Xmx6936m
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -XX:MaxDirectMemorySize=4072m
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Dlog.file=/var/log/hadoop-yarn/containers/application_1528342246614_0002/container_1528342246614_0002_01_282649/taskmanager.log
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Dlogback.configurationFile=file:./logback.xml
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> -Dlog4j.configuration=file:./log4j.properties
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> Program
>>>>>>> Arguments:
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> --configDir
>>>>>>> 2018-06-17 19:01:31,665 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner - .
>>>>>>> *2018-06-17 19:01:31,666 INFO
>>>>>>> org.apache.flink.yarn.YarnTaskExecutorRunner -
>>>>>>> Classpath:
>>>>>>> lib/flink-dist_2.11-1.5.0.jar:lib/flink-python_2.11-1.5.0.jar:lib/flink-shaded-hadoop2-uber-1.5.0.jar:lib/flink-shaded-include-yarn-0.9.1.jar:lib/guava-18.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.7.jar:log4j.properties:logback.xml:flink.jar:flink-conf.yaml::/etc/hadoop/conf:/usr/lib/hadoop/hadoop-common-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs.jar:/usr/lib/hadoop/hadoop-auth.jar:/usr/lib/hadoop/hadoop-archives-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-archive-logs-2.8.3-amzn-0.jar:/usr/lib/hadoop/hadoop-azure-datalake-2.8.3-amzn-0.jar.........*
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Amit
>>>>>>>
>>>>>>> On Mon, Jun 18, 2018 at 2:00 PM, Garvit Sharma <garvit...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Please refer to my previous mail for complete logs.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> On Mon, Jun 18, 2018 at 1:17 PM Till Rohrmann <trohrm...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Could you also please share the complete log file with us.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Till
>>>>>>>>>
>>>>>>>>> On Sat, Jun 16, 2018 at 5:22 PM Ted Yu <yuzhih...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> The error for core-default.xml is interesting.
>>>>>>>>>>
>>>>>>>>>> Flink doesn't have this file. Probably it came with Yarn. Please
>>>>>>>>>> check the hadoop version Flink was built with versus the hadoop
>>>>>>>>>> version in
>>>>>>>>>> your cluster.
>>>>>>>>>>
>>>>>>>>>> Thanks
>>>>>>>>>>
>>>>>>>>>> -------- Original message --------
>>>>>>>>>> From: Garvit Sharma <garvit...@gmail.com>
>>>>>>>>>> Date: 6/16/18 7:23 AM (GMT-08:00)
>>>>>>>>>> To: trohrm...@apache.org
>>>>>>>>>> Cc: Chesnay Schepler <ches...@apache.org>, user@flink.apache.org
>>>>>>>>>> Subject: Re: Exception while submitting jobs through Yarn
>>>>>>>>>>
>>>>>>>>>> I am not able to figure out, got stuck badly in this since last 1
>>>>>>>>>> week. Any little help would be appreciated.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,523 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> Parallelism set: 1 for 8
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,578 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> Parallelism set: 1 for 1
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,588 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 1 -> 8
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,591 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> Parallelism set: 1 for 5
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,597 DEBUG
>>>>>>>>>> org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator -
>>>>>>>>>> CONNECTED: KeyGroupStreamPartitioner - 5 -> 8
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,618 FATAL org.apache.hadoop.conf.Configuration
>>>>>>>>>> - error parsing conf core-default.xml
>>>>>>>>>>
>>>>>>>>>> javax.xml.parsers.ParserConfigurationException: Feature '
>>>>>>>>>> http://apache.org/xml/features/xinclude' is not recognized.
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown
>>>>>>>>>> Source)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2482)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2444)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2361)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.conf.Configuration.get(Configuration.java:1188)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider.getRecordFactory(RecordFactoryProvider.java:49)
>>>>>>>>>>
>>>>>>>>>> at org.apache.hadoop.yarn.util.Records.<clinit>(Records.java:32)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>
>>>>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>>>>>
>>>>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1692)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,620 WARN
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor
>>>>>>>>>> - Error while getting queue information from YARN: null
>>>>>>>>>>
>>>>>>>>>> 2018-06-16 19:25:10,621 DEBUG
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor -
>>>>>>>>>> Error details
>>>>>>>>>>
>>>>>>>>>> java.lang.ExceptionInInitializerError
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getQueueInfoRequest(YarnClientImpl.java:495)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getAllQueues(YarnClientImpl.java:525)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.checkYarnQueues(AbstractYarnClusterDescriptor.java:658)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.AbstractYarnClusterDescriptor.deployInternal(AbstractYarnClusterDescriptor.java:486)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:75)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.runProgram(CliFrontend.java:235)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:210)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
>>>>>>>>>>
>>>>>>>>>> at
>>>>>>>>>> org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Garvit Sharma
>>>>>>>> github.com/garvitlnmiit/
>>>>>>>>
>>>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>>>> determination that makes him master.
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Garvit Sharma
>>>>>> github.com/garvitlnmiit/
>>>>>>
>>>>>> No Body is a Scholar by birth, its only hard work and strong
>>>>>> determination that makes him master.
>>>>>>
>>>>>
>>>>
>>>> --
>>>>
>>>> Garvit Sharma
>>>> github.com/garvitlnmiit/
>>>>
>>>> No Body is a Scholar by birth, its only hard work and strong
>>>> determination that makes him master.
>>>>
>>>
>>
>> --
>>
>> Garvit Sharma
>> github.com/garvitlnmiit/
>>
>> No Body is a Scholar by birth, its only hard work and strong
>> determination that makes him master.
>>
> --
Garvit Sharma
github.com/garvitlnmiit/
No Body is a Scholar by birth, its only hard work and strong determination
that makes him master.
