Hmmmmmmmmmmm, this has gotten weird. Thinking the issue was due to the issue with reverse DNS not working inside Docker containers I did a bunch of rejiggering things such that now each of my contains get's configured with principals and keytabs that reflect their actual IP when they're running.
Unfortunately it's still failing to format the namenode. A sample error: 10.0.0.51:8485: DestHost:destPort hdfs-journalnode6:8485 , > LocalHost:localPort hdfs-namenode1/10.0.0.54:0. Failed on local > exception: java.io.IOException: Couldn't set up IO streams: > java.lang.IllegalArgumentException: Server has invalid Kerberos principal: > root > /10.0.0.52@HADOOP, expecting: root/10.0.0.54@HADOOP > In this scenario 10.0.0.54 is the IP of the Namenode which is attempting to format the Journalnodes. 10.0.0.52 is the IP of the hdfs-journalnode6 service. Why is the Namenode expecting the Journalnode to be configured with the Namenodes principal? That seems a bit weird to me? Or am I just misunderstanding Kerberos and Hadoop dramatically? On Thu, May 2, 2019 at 4:45 PM Arpit Agarwal <aagar...@cloudera.com> wrote: > You can use /etc/hosts entries as a workaround. > > If this is a PoC/test environment, a less secure workaround is host-less > principals. i.e. omit the _HOST pattern. Not usually recommended since > service instances will use the same principal and it may be easier to > impersonate a service if the keytab is compromised. > > > On Apr 30, 2019, at 1:11 PM, Adam Jorgensen < > adam.jorgen...@spandigital.com> wrote: > > Ahhhhh....reverse DNS you say............oh dear > > As per https://github.com/docker/for-linux/issues/365 it seems that > Reverse DNS has been rather broken in Docker since early 2018 :-( > > I'm going to have to do some digging to see if I can find some way to fix > this I guess, since the other option is a painful dance involving > registering Kerberos principals for the specific IPs > > On Tue, Apr 30, 2019 at 9:56 PM Arpit Agarwal <aagar...@cloudera.com> > wrote: > >> Likely your reverse DNS is not configured properly. You can check it by >> running ‘dig -x 10.0.0.238’. >> >> >> On Apr 30, 2019, at 12:24 PM, Adam Jorgensen < >> adam.jorgen...@spandigital.com> wrote: >> >> Hi all, my first post here. I'm looking for some help with an issue I'm >> having attempting to format my Namenode. I'm running a HA configuration and >> have configured Kerberos for authentication. >> Additionally, I am running Hadoop using Docker Swarm. >> >> The issue I'm having is that when I attempt to format the Namenode the >> operation fails with complaints that QJM Journalnode do not have a valid >> Kerberos principal. However, the issue is more specific in that it seems >> like the operation to format the Journalnodes attempts to use a Kerberos >> principal of the form SERVICE/IP@REALM whereas the principals I have >> configured use the hostname rather than the IP. >> >> If you take a look at the logging output captured below you will get a >> better idea of what the issue is. >> >> Has anyone run into this before? Is there a way I can tell the Namenode >> format to use the correct principals? >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:54,107 INFO namenode.NameNode: STARTUP_MSG: >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | >> /************************************************************ >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | STARTUP_MSG: Starting >> NameNode >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | STARTUP_MSG: host = >> hdfs-namenode1/10.0.0.60 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | STARTUP_MSG: args = >> [-format, -nonInteractive] >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | STARTUP_MSG: version >> = 3.1.2 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | STARTUP_MSG: >> classpath = >> /opt/hadoop-latest/etc/hadoop:/opt/hadoop-latest/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerby-config-1.0.1.jar:/opt/hadoop-latest/sha >> >> re/hadoop/common/lib/metrics-core-3.2.4.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerb-util-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/commons-lang3-3.4.jar:/opt/hadoop-latest/share/hadoop/common/lib/jackson-annotations-2.7.8.jar:/opt/hadoop-lates >> >> t/share/hadoop/common/lib/jetty-util-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/had >> >> oop-latest/share/hadoop/common/lib/commons-compress-1.18.jar:/opt/hadoop-latest/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerby-util-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/hadoop-l >> >> atest/share/hadoop/common/lib/accessors-smart-1.2.jar:/opt/hadoop-latest/share/hadoop/common/lib/httpcore-4.4.4.jar:/opt/hadoop-latest/share/hadoop/common/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/commons-configuration2-2.1.1.jar: >> >> /opt/hadoop-latest/share/hadoop/common/lib/jetty-server-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop-latest/share/hadoop/common/lib/jersey-servlet-1.19.jar:/opt/hadoop-latest/share/hadoop/common/lib/jackson-databi >> >> nd-2.7.8.jar:/opt/hadoop-latest/share/hadoop/common/lib/jul-to-slf4j-1.7.25.jar:/opt/hadoop-latest/share/hadoop/common/lib/jcip-annotations-1.0-1.jar:/opt/hadoop-latest/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/hadoop-latest/share/hadoop/common/lib/js >> >> on-smart-2.3.jar:/opt/hadoop-latest/share/hadoop/common/lib/commons-beanutils-1.9.3.jar:/opt/hadoop-latest/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar:/opt/hadoop-latest/share/hadoop/common/lib/curator-framework-2.13.0.jar:/opt/hadoop-latest/share/hadoop/co >> >> mmon/lib/netty-3.10.5.Final.jar:/opt/hadoop-latest/share/hadoop/common/lib/re2j-1.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/woodstox-core-5.0.3.jar:/opt/hadoop-latest/share/hadoop/common/lib/jersey-json-1.19.jar:/opt/hadoop-latest/share/hadoop/common/lib/ >> >> kerb-server-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerb-core-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/jersey-core-1.19.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerb-crypto-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerby >> >> -asn1-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/snappy-java-1.0.5.jar:/opt/hadoop-latest/share/hadoop/common/lib/guava-11.0.2.jar:/opt/hadoop-latest/share/hadoop/common/lib/httpclient-4.5.2.jar:/opt/hadoop-latest/share/hadoop/common/lib/stax2-api-3.1. >> >> 4.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerb-common-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/asm-5.0.4.jar:/opt/hadoop-latest/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-latest/share/hadoop/common/lib/jsr311-api-1.1.1.jar >> >> :/opt/hadoop-latest/share/hadoop/common/lib/jaxb-api-2.2.11.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerby-xdr-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/jetty-io-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerby-pkix-1.0.1.ja >> >> r:/opt/hadoop-latest/share/hadoop/common/lib/commons-net-3.6.jar:/opt/hadoop-latest/share/hadoop/common/lib/jackson-core-2.7.8.jar:/opt/hadoop-latest/share/hadoop/common/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop-latest/share/hadoop/common/lib/jackson-map >> >> per-asl-1.9.13.jar:/opt/hadoop-latest/share/hadoop/common/lib/curator-client-2.13.0.jar:/opt/hadoop-latest/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerb-client-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/curator >> >> -recipes-2.13.0.jar:/opt/hadoop-latest/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-latest/share/hadoop/common/lib/jetty-xml-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/common/lib/jetty-servlet-9.3.24.v20180605.jar:/opt/hadoop-latest/sha >> >> re/hadoop/common/lib/hadoop-auth-3.1.2.jar:/opt/hadoop-latest/share/hadoop/common/lib/audience-annotations-0.5.0.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerb-admin-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/jetty-http-9.3.24.v20180605.jar:/opt/h >> >> adoop-latest/share/hadoop/common/lib/jetty-security-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/hadoop-latest/share/hadoop/common/lib/jersey-server-1.19.jar:/opt/hadoop-latest/share/hadoop/common/lib/hadoop- >> >> annotations-3.1.2.jar:/opt/hadoop-latest/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/hadoop-latest/share/hadoop/common/lib/token-provider-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/hadoop-latest/share/hadoop/common >> >> /lib/jetty-webapp-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hadoop-latest/share/hadoop/common/lib/zookeeper-3.4.13.jar:/opt/hadoop-latest/share/hadoop/common/lib/avro-1.7.7.jar:/opt/hadoop-latest/share/hadoop/common/li >> >> b/javax.servlet-api-3.1.0.jar:/opt/hadoop-latest/share/hadoop/common/lib/gson-2.2.4.jar:/opt/hadoop-latest/share/hadoop/common/lib/commons-codec-1.11.jar:/opt/hadoop-latest/share/hadoop/common/lib/commons-io-2.5.jar:/opt/hadoop-latest/share/hadoop/common/lib/jsc >> >> h-0.1.54.jar:/opt/hadoop-latest/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/hadoop-latest/share/hadoop/common/lib/slf4j-api-1.7.25.jar:/opt/hadoop-latest/share/hadoop/common/lib/kerb-identity-1.0.1.jar:/opt/hadoop-latest/share/hadoop/common/hadoop-common-3.1.2 >> >> -tests.jar:/opt/hadoop-latest/share/hadoop/common/hadoop-nfs-3.1.2.jar:/opt/hadoop-latest/share/hadoop/common/hadoop-common-3.1.2.jar:/opt/hadoop-latest/share/hadoop/common/hadoop-kms-3.1.2.jar:/opt/hadoop-latest/share/hadoop/hdfs:/opt/hadoop-latest/share/hadoop >> >> /hdfs/lib/commons-math3-3.1.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerby-config-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerb-util-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/okio-1.6.0.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/common >> >> s-lang3-3.4.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jackson-annotations-2.7.8.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jetty-util-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-latest/share/hadoop/hdfs/li >> >> b/kerb-simplekdc-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-compress-1.18.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jettison-1.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jetty-ut >> >> il-ajax-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerby-util-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/accessors-smart-1.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/htt >> >> pcore-4.4.4.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/nimbus-jose-jwt-4.41.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-configuration2-2.1.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jetty-server-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hado >> >> op/hdfs/lib/paranamer-2.3.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jersey-servlet-1.19.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jackson-databind-2.7.8.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jcip-annotations-1.0-1.jar:/opt/hadoop-latest/share/hadoop >> >> /hdfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/json-smart-2.3.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-beanutils-1.9.3.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/hadoop-latest/share/hadoop/hdf >> >> s/lib/curator-framework-2.13.0.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/netty-3.10.5.Final.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/re2j-1.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/woodstox-core-5.0.3.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jer >> >> sey-json-1.19.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerb-server-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerb-core-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jersey-core-1.19.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerb-crypto-1.0.1 >> >> .jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerby-asn1-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/snappy-java-1.0.5.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/httpclient-4.5.2.jar:/opt/hadoop- >> >> latest/share/hadoop/hdfs/lib/stax2-api-3.1.4.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerb-common-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/asm-5.0.4.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-latest/share/h >> >> adoop/hdfs/lib/jsr311-api-1.1.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jaxb-api-2.2.11.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerby-xdr-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jetty-io-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/h >> >> dfs/lib/kerby-pkix-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-net-3.6.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jackson-core-2.7.8.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop-latest/share/hadoop/h >> >> dfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/curator-client-2.13.0.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/netty-all-4.0.52.Final.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerb-client-1.0.1.jar:/opt/hadoop-latest/share/h >> >> adoop/hdfs/lib/curator-recipes-2.13.0.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jetty-xml-9.3.24.v20180605.jar:/opt/hadoop-lat >> >> est/share/hadoop/hdfs/lib/jetty-servlet-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/hadoop-auth-3.1.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/audience-annotations-0.5.0.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/kerb-admin-1.0.1.jar:/opt >> >> /hadoop-latest/share/hadoop/hdfs/lib/jetty-http-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jetty-security-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-collections-3.2.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/json >> >> -simple-1.1.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jersey-server-1.19.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/hadoop-annotations-3.1.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jackson-xc-1.9.13.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/token- >> >> provider-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jetty-webapp-9.3.24.v20180605.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/zo >> >> okeeper-3.4.13.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/avro-1.7.7.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/gson-2.2.4.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-codec-1.11.ja >> >> r:/opt/hadoop-latest/share/hadoop/hdfs/lib/commons-io-2.5.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jsch-0.1.54.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/hadoop-latest/share/hadoop/hdfs/lib/okhttp-2.7.5.jar:/opt/hadoop-latest/share/ha >> >> doop/hdfs/lib/kerb-identity-1.0.1.jar:/opt/hadoop-latest/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.2-tests.jar:/opt/hadoop-latest/share/hadoop/hdfs/hadoop-hdfs-rbf-3.1.2.jar:/opt/hadoop-latest/ >> >> share/hadoop/hdfs/hadoop-hdfs-client-3.1.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/hadoop-hdfs-httpfs-3.1.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/hadoop-hdfs-native-client-3.1.2-tests.jar:/opt/hadoop-latest/share/hadoop/hdfs/hadoop-hdfs-3.1.2.jar:/opt/hadoop >> >> -latest/share/hadoop/hdfs/hadoop-hdfs-3.1.2-tests.jar:/opt/hadoop-latest/share/hadoop/hdfs/hadoop-hdfs-nfs-3.1.2.jar:/opt/hadoop-latest/share/hadoop/hdfs/hadoop-hdfs-client-3.1.2-tests.jar:/opt/hadoop-latest/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/hadoop- >> >> latest/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-mapreduce-client-app-3.1.2.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-mapreduce-client-common-3.1.2.jar:/opt/hadoop-latest/share/hadoop/mapreduce/h >> >> adoop-mapreduce-client-uploader-3.1.2.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-3.1.2-tests.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.1.2.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-m >> >> apreduce-client-jobclient-3.1.2.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-mapreduce-client-core-3.1.2.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-3.1.2.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-mapreduce >> >> -client-nativetask-3.1.2.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-3.1.2.jar:/opt/hadoop-latest/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn:/opt/hadoop-latest/share/hadoop/y >> >> arn/lib/metrics-core-3.2.4.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/swagger-annotations-1.5.4.jar:/opt/hadoop-latest/share/hadoop/ >> >> yarn/lib/guice-servlet-4.0.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/json-io-2.5.1.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/fst-2.50.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/opt/hadoop-latest/share/hadoop/yarn >> >> /lib/jackson-jaxrs-json-provider-2.7.8.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/HikariCP-java7-2.4.12.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/java-util-1.9.0.jar:/opt/hadoop-latest/share/hadoop >> >> /yarn/lib/jersey-guice-1.19.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/jersey-client-1.19.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/jackson-jaxrs-base-2.7.8.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/snakeyaml-1.16.jar:/opt/hadoop-latest/share/hadoop/yarn >> >> /lib/guice-4.0.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/objenesis-1.0.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/jackson-module-jaxb-annotations-2.7.8.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/ehcache-3.3.1.jar:/opt/hadoop-latest/share/hadoop/yarn/lib/d >> >> nsjava-2.1.7.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-services-core-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-server-timeline-pluginstorage-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-registry-3.1.2.jar:/opt/hadoop-lat >> >> est/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-api-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-services-api-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-serv >> >> er-nodemanager-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-server-tests-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-common-3.1.2.jar:/opt/hadoop-la >> >> test/share/hadoop/yarn/hadoop-yarn-server-common-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-3.1.2.jar:/opt/hadoop-lates >> >> t/share/hadoop/yarn/hadoop-yarn-server-web-proxy-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-client-3.1.2.jar:/opt/hadoop-latest/share/hadoop/yarn/hadoop-yarn-server >> -router-3.1.2.jar >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | STARTUP_MSG: build = >> https://github.com/apache/hadoop.git -r >> 1019dde65bcf12e05ef48ac71e84550d589e5d9a; compiled by 'sunilg' on >> 2019-01-29T01:39Z >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | STARTUP_MSG: java = >> 1.8.0_202 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | >> ************************************************************/ >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:54,115 INFO namenode.NameNode: registered UNIX signal handlers for >> [TERM, HUP, INT] >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:54,192 INFO namenode.NameNode: createNameNode [-format, >> -nonInteractive] >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,034 INFO security.UserGroupInformation: Login successful for user >> root/hdfs-namenode1@HADOOP using keytab file /etc/krb5.keytab >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,110 INFO common.Util: Assuming 'file' scheme for path >> /opt/hadoop-latest/data in configuration. >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,111 INFO common.Util: Assuming 'file' scheme for path >> /opt/hadoop-latest/data in configuration. >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | Formatting using >> clusterid: CID-ad09dcfb-0152-4ded-9f72-2be4dfd729b8 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,141 INFO namenode.FSEditLog: Edit logging is async:true >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,152 INFO namenode.FSNamesystem: KeyProvider: null >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,153 INFO namenode.FSNamesystem: fsLock is fair: true >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,154 INFO namenode.FSNamesystem: Detailed lock hold time metrics >> enabled: false >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,154 INFO namenode.FSNamesystem: fsOwner = >> root/hdfs-namenode1@HADOOP (auth:KERBEROS) >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,154 INFO namenode.FSNamesystem: supergroup = supergroup >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,155 INFO namenode.FSNamesystem: isPermissionEnabled = true >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,156 WARN hdfs.DFSUtilClient: Namenode for hadoop-hdfs-cluster >> remains unresolved for ID secondary. Check your hdfs-site.xml file to >> ensure namenodes are configured properly. >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,156 INFO namenode.FSNamesystem: Determined nameservice ID: >> hadoop-hdfs-cluster >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,156 INFO namenode.FSNamesystem: HA Enabled: true >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,192 INFO common.Util: >> dfs.datanode.fileio.profiling.sampling.percentage set to 0. Disabling file >> IO profiling >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,203 INFO blockmanagement.DatanodeManager: >> dfs.block.invalidate.limit: configured=1000, counted=60, effected=1000 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,203 INFO blockmanagement.DatanodeManager: >> dfs.namenode.datanode.registration.ip-hostname-check=false >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,206 INFO blockmanagement.BlockManager: >> dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,206 INFO blockmanagement.BlockManager: The block deletion will >> start around 2019 Apr 30 11:17:55 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,207 INFO util.GSet: Computing capacity for map BlocksMap >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,207 INFO util.GSet: VM type = 64-bit >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,208 INFO util.GSet: 2.0% max memory 3.5 GB = 70.8 MB >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,208 INFO util.GSet: capacity = 2^23 = 8388608 entries >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,226 INFO blockmanagement.BlockManager: >> dfs.block.access.token.enable = true >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,226 INFO blockmanagement.BlockManager: >> dfs.block.access.key.update.interval=600 min(s), >> dfs.block.access.token.lifetime=600 min(s), >> dfs.encrypt.data.transfer.algorithm=null >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,226 WARN hdfs.DFSUtilClient: Namenode for hadoop-hdfs-cluster >> remains unresolved for ID secondary. Check your hdfs-site.xml file to >> ensure namenodes are configured properly. >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,237 INFO Configuration.deprecation: No unit for >> dfs.namenode.safemode.extension(30000) assuming MILLISECONDS >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,237 INFO blockmanagement.BlockManagerSafeMode: >> dfs.namenode.safemode.threshold-pct = 0.9990000128746033 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,237 INFO blockmanagement.BlockManagerSafeMode: >> dfs.namenode.safemode.min.datanodes = 0 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,237 INFO blockmanagement.BlockManagerSafeMode: >> dfs.namenode.safemode.extension = 30000 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,237 INFO blockmanagement.BlockManager: defaultReplication >> = 3 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,237 INFO blockmanagement.BlockManager: maxReplication >> = 512 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,237 INFO blockmanagement.BlockManager: minReplication >> = 1 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,237 INFO blockmanagement.BlockManager: maxReplicationStreams >> = 2 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,240 INFO blockmanagement.BlockManager: redundancyRecheckInterval >> = 3000ms >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,240 INFO blockmanagement.BlockManager: encryptDataTransfer >> = false >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,240 INFO blockmanagement.BlockManager: maxNumBlocksToLog >> = 1000 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,257 INFO namenode.FSDirectory: GLOBAL serial map: bits=24 >> maxEntries=16777215 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,269 INFO util.GSet: Computing capacity for map INodeMap >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,269 INFO util.GSet: VM type = 64-bit >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,269 INFO util.GSet: 1.0% max memory 3.5 GB = 35.4 MB >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,269 INFO util.GSet: capacity = 2^22 = 4194304 entries >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,270 INFO namenode.FSDirectory: ACLs enabled? false >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,270 INFO namenode.FSDirectory: POSIX ACL inheritance enabled? true >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,270 INFO namenode.FSDirectory: XAttrs enabled? true >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,270 INFO namenode.NameNode: Caching file names occurring more than >> 10 times >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,274 INFO snapshot.SnapshotManager: Loaded config captureOpenFiles: >> false, skipCaptureAccessTimeOnlyChange: false, >> snapshotDiffAllowSnapRootDescendant: true, maxSnapshotLimit: 65536 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,276 INFO snapshot.SnapshotManager: SkipList is disabled >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,279 INFO util.GSet: Computing capacity for map cachedBlocks >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,279 INFO util.GSet: VM type = 64-bit >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,279 INFO util.GSet: 0.25% max memory 3.5 GB = 8.8 MB >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,279 INFO util.GSet: capacity = 2^20 = 1048576 entries >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,318 INFO metrics.TopMetrics: NNTop conf: >> dfs.namenode.top.window.num.buckets = 10 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,318 INFO metrics.TopMetrics: NNTop conf: >> dfs.namenode.top.num.users = 10 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,319 INFO metrics.TopMetrics: NNTop conf: >> dfs.namenode.top.windows.minutes = 1,5,25 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,322 INFO namenode.FSNamesystem: Retry cache on namenode is enabled >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,322 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total >> heap and retry cache entry expiry time is 600000 millis >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,323 INFO util.GSet: Computing capacity for map NameNodeRetryCache >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,323 INFO util.GSet: VM type = 64-bit >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,323 INFO util.GSet: 0.029999999329447746% max memory 3.5 GB = 1.1 >> MB >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,323 INFO util.GSet: capacity = 2^17 = 131072 entries >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,347 WARN client.QuorumJournalManager: Quorum journal URI ' >> qjournal://hdfs-journalnode1:8485;hdfs-journalnode2:8485;hdfs-journalnode3:8485;hdfs-journalnode4:8485;hdfs-journalnode5:8485;hdfs-jou >> rnalnode6:8485;hdfs-journalnode7:8485;hdfs-journalnode8:8485/hadoop-hdfs-cluster' >> has an even number of Journal Nodes specified. This is not recommended! >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,653 WARN namenode.NameNode: Encountered exception during format: >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | >> org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if >> JNs are ready for formatting. 7 exceptions thrown: >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.236:8485: >> DestHost:destPort hdfs-journalnode5:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode5@HADOOP, expecting: root/10.0.0.236@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.252:8485: >> DestHost:destPort hdfs-journalnode6:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode6@HADOOP, expecting: root/10.0.0.252@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.250:8485: >> DestHost:destPort hdfs-journalnode1:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode1@HADOOP, expecting: root/10.0.0.250@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.240:8485: >> DestHost:destPort hdfs-journalnode8:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode8@HADOOP, expecting: root/10.0.0.240@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.232:8485: >> DestHost:destPort hdfs-journalnode2:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode2@HADOOP, expecting: root/10.0.0.232@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.244:8485: >> DestHost:destPort hdfs-journalnode7:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode7@HADOOP, expecting: root/10.0.0.244@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.238:8485: >> DestHost:destPort hdfs-journalnode3:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode3@HADOOP, expecting: root/10.0.0.238@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:253) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:1142) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:196) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1155) >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1600) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,657 ERROR namenode.NameNode: Failed to start namenode. >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | >> org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if >> JNs are ready for formatting. 7 exceptions thrown: >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.236:8485: >> DestHost:destPort hdfs-journalnode5:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode5@HADOOP, expecting: root/10.0.0.236@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.252:8485: >> DestHost:destPort hdfs-journalnode6:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode6@HADOOP, expecting: root/10.0.0.252@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.250:8485: >> DestHost:destPort hdfs-journalnode1:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode1@HADOOP, expecting: root/10.0.0.250@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.240:8485: >> DestHost:destPort hdfs-journalnode8:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode8@HADOOP, expecting: root/10.0.0.240@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.232:8485: >> DestHost:destPort hdfs-journalnode2:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode2@HADOOP, expecting: root/10.0.0.232@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.244:8485: >> DestHost:destPort hdfs-journalnode7:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode7@HADOOP, expecting: root/10.0.0.244@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.238:8485: >> DestHost:destPort hdfs-journalnode3:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode3@HADOOP, expecting: root/10.0.0.238@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.qjournal.client.QuorumException.create(QuorumException.java:81) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.qjournal.client.QuorumCall.rethrowException(QuorumCall.java:286) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.hasSomeData(QuorumJournalManager.java:253) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.common.Storage.confirmFormat(Storage.java:1142) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.namenode.FSImage.confirmFormat(FSImage.java:196) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1155) >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1600) >> >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | at >> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,658 INFO util.ExitUtil: Exiting with status 1: >> org.apache.hadoop.hdfs.qjournal.client.QuorumException: Unable to check if >> JNs are ready for formatting. 7 exceptions thrown: >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.236:8485: >> DestHost:destPort hdfs-journalnode5:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode5@HADOOP, expecting: root/10.0.0.236@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.252:8485: >> DestHost:destPort hdfs-journalnode6:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode6@HADOOP, expecting: root/10.0.0.252@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.250:8485: >> DestHost:destPort hdfs-journalnode1:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode1@HADOOP, expecting: root/10.0.0.250@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.240:8485: >> DestHost:destPort hdfs-journalnode8:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode8@HADOOP, expecting: root/10.0.0.240@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.232:8485: >> DestHost:destPort hdfs-journalnode2:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode2@HADOOP, expecting: root/10.0.0.232@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.244:8485: >> DestHost:destPort hdfs-journalnode7:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode7@HADOOP, expecting: root/10.0.0.244@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 10.0.0.238:8485: >> DestHost:destPort hdfs-journalnode3:8485 , LocalHost:localPort >> hdfs-namenode1/10.0.0.60:0. Failed on local exception: >> java.io.IOException: Couldn't set up IO streams: >> java.lang.IllegalArgumentExc >> eption: Server has invalid Kerberos principal: >> root/hdfs-journalnode3@HADOOP, expecting: root/10.0.0.238@HADOOP >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | 2019-04-30 >> 11:17:55,660 INFO namenode.NameNode: SHUTDOWN_MSG: >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | >> /************************************************************ >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | SHUTDOWN_MSG: Shutting >> down NameNode at hdfs-namenode1/10.0.0.60 >> adss_hdfs-namenode1.1.ybygx5r50v8y@Harvester | >> ************************************************************/ >> >> >> >
