[
https://issues.apache.org/jira/browse/AMBARI-12745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmytro Sen updated AMBARI-12745:
--------------------------------
Attachment: AMBARI-12745.patch
> Nodemanagers fail to start because of wrong recovery.dir property
> -----------------------------------------------------------------
>
> Key: AMBARI-12745
> URL: https://issues.apache.org/jira/browse/AMBARI-12745
> Project: Ambari
> Issue Type: Bug
> Components: stacks
> Affects Versions: 2.1.1
> Reporter: Dmytro Sen
> Assignee: Dmytro Sen
> Priority: Blocker
> Fix For: 2.1.1
>
> Attachments: AMBARI-12745.patch
>
>
> $ yarn nodemanager -checkHealth
> {noformat}
> 15/08/07 15:45:24 INFO nodemanager.NodeManager: STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NodeManager
> STARTUP_MSG: host = os-u14-chwavu-oozie-ha-1-5/172.22.126.134
> STARTUP_MSG: args = [-checkHealth]
> STARTUP_MSG: version = 2.7.1.2.3.2.0-2602
> STARTUP_MSG: classpath =
> /usr/hdp/2.3.2.0-2602/hadoop/conf:/usr/hdp/2.3.2.0-2602/hadoop/conf:/usr/hdp/2.3.2.0-2602/hadoop/conf:....
> STARTUP_MSG: build = [email protected]:hortonworks/hadoop.git -r
> f66cf95e2e9367a74b0ec88b2df33458b6cff2d0; compiled by 'jenkins' on
> 2015-08-05T21:42Z
> STARTUP_MSG: java = 1.7.0_79
> ************************************************************/
> 15/08/07 15:45:24 INFO nodemanager.NodeManager: registered UNIX signal
> handlers for [TERM, HUP, INT]
> 15/08/07 15:45:26 INFO recovery.NMLeveldbStateStoreService: Using state
> database at /nodemanager/recovery-state/yarn-nm-state for recovery
> 15/08/07 15:45:26 INFO service.AbstractService: Service
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService
> failed in state INITED; cause:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at
> org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573)
> 15/08/07 15:45:26 INFO service.AbstractService: Service NodeManager failed in
> state INITED; cause: org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573)
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at
> org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 15/08/07 15:45:26 FATAL nodemanager.NodeManager: Error starting NodeManager
> org.apache.hadoop.service.ServiceStateException:
> org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at
> org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartRecoveryStore(NodeManager.java:177)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:219)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:525)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:573)
> Caused by: org.fusesource.leveldbjni.internal.NativeDB$DBException: IO error:
> /nodemanager/recovery-state/yarn-nm-state/LOCK: No such file or directory
> at
> org.fusesource.leveldbjni.internal.NativeDB.checkStatus(NativeDB.java:200)
> at org.fusesource.leveldbjni.internal.NativeDB.open(NativeDB.java:218)
> at org.fusesource.leveldbjni.JniDBFactory.open(JniDBFactory.java:168)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMLeveldbStateStoreService.initStorage(NMLeveldbStateStoreService.java:930)
> at
> org.apache.hadoop.yarn.server.nodemanager.recovery.NMStateStoreService.serviceInit(NMStateStoreService.java:204)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> ... 5 more
> 15/08/07 15:45:26 INFO nodemanager.NodeManager: SHUTDOWN_MSG:
> /************************************************************
> SHUTDOWN_MSG: Shutting down NodeManager at
> os-u14-chwavu-oozie-ha-1-5/172.22.126.134
> ************************************************************/
> yarn@os-u14-chwavu-oozie-ha-1-5:/grid/0/hadoop/yarn$
> /usr/hdp/current/hadoop-yarn-nodemanager2015-08-07 01:51:06,160 INFO
> nodemanager.NodeManager (LogAdapter.java:info(45)) - STARTUP_MSG:
> /************************************************************
> STARTUP_MSG: Starting NodeManager
> STARTUP_MSG: host = os-u14-chwavu-oozie-ha-1-5/172.22.126.134
> STARTUP_MSG: args = []
> STARTUP_MSG: version = 2.7.1.2.3.2.0-2602
> STARTUP_MSG: classpath =
> /usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/current/hadoop-client/conf:/usr/hdp/2.3.2.0-2602/hadoop/lib/log4j-1.2.17.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/xmlenc-0.52.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jetty-util-6.1.26.hwx.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-core-2.2.3.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/slf4j-log4j12-1.7.10.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/hadoop-lzo-0.6.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-common-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpmime-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jackson-xc-1.9.13.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jersey-server-1.9.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpcore-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/java-xmlbuilder-0.4.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-net-3.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/xz-1.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/javax.persistence-2.1.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jersey-core-1.9.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-yarn-plugin-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/slf4j-api-1.7.10.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/microsoft-windowsazure-storage-sdk-0.6.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jets3t-0.9.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/paranamer-2.3.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jettison-1.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/httpclient-4.2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-audit-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-io-2.4.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/servlet-api-2.5.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-httpclient-3.1.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-plugins-cred-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/ranger-hdfs-plugin-0.5.0.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/zookeeper-3.4.6.2.3.2.0-2602.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/jaxb-api-2.2.2.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/hdp/2.3.2.0-2602/hadoop/lib/hadoop-common-2.7.1.2.3.2.0-2602.jar:/usr/hd...skipping...
> /sbin/yarn-daemon.sh --config /tmp/hadoopConf start nodemanager
> starting nodemanager, logging to
> /grid/0/log/hadoop/yarn/yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.out
> yarn@os-u14-chwavu-oozie-ha-1-5:/grid/0/hadoop/yarn$ ll
> /grid/0/log/hadoop/yarn/yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.
> yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.log
> yarn-yarn-nodemanager-os-u14-chwavu-oozie-ha-1-5.out
> 2015-08-07 01:51:06,160 INFO nodemanager.NodeManager
> (LogAdapter.java:info(45)) - STARTUP_MSG:
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
