[
https://issues.apache.org/jira/browse/HDFS-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475894#comment-15475894
]
Dinesh edited comment on HDFS-9096 at 10/18/16 10:41 AM:
---------------------------------------------------------
Facing same issue when I Rollback (after rolling upgrade) from Hadoop 2.7.2 to
2.5.2; Could any one please tell, is this a known bug?
Based on below logs, please suggest can we consider this issue as a new bug,
My Name node log details:
|C:\Hadoop\bin>hdfs namenode -rollingUpgrade rollback
16/10/18 15:54:47 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = myservernode/192.168.1.21
STARTUP_MSG: args = [-rollingUpgrade, rollback]
STARTUP_MSG: version = 2.7.2
STARTUP_MSG: classpath = ....
16/10/18 15:54:47 INFO namenode.NameNode: createNameNode [-rollingUpgrade,
rollback]
16/10/18 15:54:48 INFO impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
16/10/18 15:54:48 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10
second(s).
16/10/18 15:54:48 INFO impl.MetricsSystemImpl: NameNode metrics system started
16/10/18 15:54:48 INFO namenode.NameNode: fs.defaultFS is hdfs://hacluster
16/10/18 15:54:48 INFO namenode.NameNode: Clients are to use hacluster to
access this namenode/service.
16/10/18 15:54:48 INFO hdfs.DFSUtil: Starting Web-server for hdfs at:
http://myservernode.root.server.lan:50070
16/10/18 15:54:48 INFO mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
16/10/18 15:54:48 INFO server.AuthenticationFilter: Unable to initialize
FileSignerSecretProvider, falling back to use random secrets.
16/10/18 15:54:48 INFO http.HttpRequestLog: Http request log for
http.requests.namenode is not defined
16/10/18 15:54:48 INFO http.HttpServer2: Added global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
16/10/18 15:54:48 INFO http.HttpServer2: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context hdfs
16/10/18 15:54:48 INFO http.HttpServer2: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
16/10/18 15:54:48 INFO http.HttpServer2: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
16/10/18 15:54:48 INFO http.HttpServer2: Added filter
'org.apache.hadoop.hdfs.web.AuthFilter'
(class=org.apache.hadoop.hdfs.web.AuthFilter)
16/10/18 15:54:48 INFO http.HttpServer2: addJerseyResourcePackage:
packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*
16/10/18 15:54:48 INFO http.HttpServer2: Jetty bound to port 50070
16/10/18 15:54:48 INFO mortbay.log: jetty-6.1.26
16/10/18 15:54:48 INFO mortbay.log: Started
HttpServer2$selectchannelconnectorwithsafestar...@myservernode.root.server.lan:50070
16/10/18 15:54:48 WARN namenode.FSNamesystem: Only one image storage directory
(dfs.namenode.name.dir) configured. Beware of data loss due to lack of
redundant storage directories!
16/10/18 15:54:48 INFO namenode.FSNamesystem: No KeyProvider found.
16/10/18 15:54:48 INFO namenode.FSNamesystem: fsLock is fair:true
16/10/18 15:54:48 INFO blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
16/10/18 15:54:48 INFO blockmanagement.DatanodeManager:
dfs.namenode.datanode.registration.ip-hostname-check=true
16/10/18 15:54:48 INFO blockmanagement.BlockManager:
dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/10/18 15:54:48 INFO blockmanagement.BlockManager: The block deletion will
start around 2016 Oct 18 15:54:48
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map BlocksMap
16/10/18 15:54:48 INFO util.GSet: VM type = 64-bit
16/10/18 15:54:48 INFO util.GSet: 2.0% max memory 455 MB = 9.1 MB
16/10/18 15:54:48 INFO util.GSet: capacity = 2^20 = 1048576 entries
16/10/18 15:54:48 INFO blockmanagement.BlockManager:
dfs.block.access.token.enable=false
16/10/18 15:54:48 INFO blockmanagement.BlockManager: defaultReplication
= 3
16/10/18 15:54:48 INFO blockmanagement.BlockManager: maxReplication
= 512
16/10/18 15:54:48 INFO blockmanagement.BlockManager: minReplication
= 1
16/10/18 15:54:48 INFO blockmanagement.BlockManager: maxReplicationStreams
= 2
16/10/18 15:54:48 INFO blockmanagement.BlockManager: replicationRecheckInterval
= 3000
16/10/18 15:54:48 INFO blockmanagement.BlockManager: encryptDataTransfer
= false
16/10/18 15:54:48 INFO blockmanagement.BlockManager: maxNumBlocksToLog
= 1000
16/10/18 15:54:48 INFO namenode.FSNamesystem: fsOwner = SYSTEM
(auth:SIMPLE)
16/10/18 15:54:48 INFO namenode.FSNamesystem: supergroup = supergroup
16/10/18 15:54:48 INFO namenode.FSNamesystem: isPermissionEnabled = false
16/10/18 15:54:48 INFO namenode.FSNamesystem: Determined nameservice ID:
hacluster
16/10/18 15:54:48 INFO namenode.FSNamesystem: HA Enabled: true
16/10/18 15:54:48 INFO namenode.FSNamesystem: Append Enabled: true
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map INodeMap
16/10/18 15:54:48 INFO util.GSet: VM type = 64-bit
16/10/18 15:54:48 INFO util.GSet: 1.0% max memory 455 MB = 4.5 MB
16/10/18 15:54:48 INFO util.GSet: capacity = 2^19 = 524288 entries
16/10/18 15:54:48 INFO namenode.FSDirectory: ACLs enabled? false
16/10/18 15:54:48 INFO namenode.FSDirectory: XAttrs enabled? true
16/10/18 15:54:48 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
16/10/18 15:54:48 INFO namenode.NameNode: Caching file names occuring more than
10 times
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map cachedBlocks
16/10/18 15:54:48 INFO util.GSet: VM type = 64-bit
16/10/18 15:54:48 INFO util.GSet: 0.25% max memory 455 MB = 1.1 MB
16/10/18 15:54:48 INFO util.GSet: capacity = 2^17 = 131072 entries
16/10/18 15:54:48 INFO namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/10/18 15:54:48 INFO namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
16/10/18 15:54:48 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension
= 30000
16/10/18 15:54:48 INFO metrics.TopMetrics: NNTop conf:
dfs.namenode.top.window.num.buckets = 10
16/10/18 15:54:48 INFO metrics.TopMetrics: NNTop conf:
dfs.namenode.top.num.users = 10
16/10/18 15:54:48 INFO metrics.TopMetrics: NNTop conf:
dfs.namenode.top.windows.minutes = 1,5,25
16/10/18 15:54:48 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/10/18 15:54:48 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
total heap and retry cache entry expiry time is 600000 millis
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/10/18 15:54:48 INFO util.GSet: VM type = 64-bit
16/10/18 15:54:48 INFO util.GSet: 0.029999999329447746% max memory 455 MB =
139.8 KB
16/10/18 15:54:48 INFO util.GSet: capacity = 2^14 = 16384 entries
16/10/18 15:54:48 INFO common.Storage: Lock on
\Metadata\data\dfs\namenode\in_use.lock acquired by nodename 13228@myservernode
16/10/18 15:54:49 INFO client.QuorumJournalManager: Starting recovery process
for unclosed journal segments...
16/10/18 15:54:49 INFO client.QuorumJournalManager: Successfully started new
epoch 6
16/10/18 15:54:49 INFO client.QuorumJournalManager: Beginning recovery of
unclosed segment starting at txid 5879
16/10/18 15:54:49 INFO client.QuorumJournalManager: Recovery prepare phase
complete. Responses:
192.168.1.22:8485: segmentState { startTxId: 5879 endTxId: 5942 isInProgress:
true } lastWriterEpoch: 5 lastCommittedTxId: 5941
192.168.1.23:8485: segmentState { startTxId: 5879 endTxId: 5942 isInProgress:
true } lastWriterEpoch: 5 lastCommittedTxId: 5941
16/10/18 15:54:49 INFO client.QuorumJournalManager: Using longest log:
172.16.100.93:8485=segmentState {
startTxId: 5879
endTxId: 5942
isInProgress: true
}
lastWriterEpoch: 5
lastCommittedTxId: 5941
16/10/18 15:54:49 INFO namenode.FileJournalManager: Recovering unfinalized
segments in \Metadata\data\dfs\namenode\current
16/10/18 15:54:49 INFO namenode.FileJournalManager: Finalizing edits file
\Metadata\data\dfs\namenode\current\edits_inprogress_0000000000000005718 ->
\Metadata\data\dfs\namenode\current\edits_0000000000000005718-0000000000000005718
16/10/18 15:54:50 ERROR namenode.FSImage: Failed to load image from
FSImageFile(file=\Metadata\data\dfs\namenode\current\fsimage_rollback_0000000000000005713,
cpktTxId=0000000000000005713)
java.io.IOException: Image version -57 is not equal to the software version -63
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:196)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:179)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:957)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:941)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:740)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:676)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/18 15:54:50 WARN namenode.FSNamesystem: Encountered exception loading
fsimage
java.io.IOException: Failed to load an FSImage file!
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:687)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/18 15:54:50 INFO mortbay.log: Stopped
HttpServer2$selectchannelconnectorwithsafestar...@myservernode.root.server.lan:50070
16/10/18 15:54:50 INFO impl.MetricsSystemImpl: Stopping NameNode metrics
system...
16/10/18 15:54:50 INFO impl.MetricsSystemImpl: NameNode metrics system stopped.
16/10/18 15:54:50 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown
complete.
16/10/18 15:54:50 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Failed to load an FSImage file!
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:687)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/18 15:54:50 INFO util.ExitUtil: Exiting with status 1
16/10/18 15:54:50 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at myservernode/192.168.1.21
************************************************************/|
was (Author: dineshkumarp):
Facing same issue when I Rollback (after rolling upgrade) from Hadoop 2.7.2 to
2.5.2; Could any one please tell, is this a known bug?
My Name node log details:
|C:\Hadoop\bin>hdfs namenode -rollingUpgrade rollback
16/10/18 15:54:47 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = myservernode/192.168.1.21
STARTUP_MSG: args = [-rollingUpgrade, rollback]
STARTUP_MSG: version = 2.7.2
STARTUP_MSG: classpath = ....
16/10/18 15:54:47 INFO namenode.NameNode: createNameNode [-rollingUpgrade,
rollback]
16/10/18 15:54:48 INFO impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
16/10/18 15:54:48 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10
second(s).
16/10/18 15:54:48 INFO impl.MetricsSystemImpl: NameNode metrics system started
16/10/18 15:54:48 INFO namenode.NameNode: fs.defaultFS is hdfs://hacluster
16/10/18 15:54:48 INFO namenode.NameNode: Clients are to use hacluster to
access this namenode/service.
16/10/18 15:54:48 INFO hdfs.DFSUtil: Starting Web-server for hdfs at:
http://myservernode.root.server.lan:50070
16/10/18 15:54:48 INFO mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
16/10/18 15:54:48 INFO server.AuthenticationFilter: Unable to initialize
FileSignerSecretProvider, falling back to use random secrets.
16/10/18 15:54:48 INFO http.HttpRequestLog: Http request log for
http.requests.namenode is not defined
16/10/18 15:54:48 INFO http.HttpServer2: Added global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
16/10/18 15:54:48 INFO http.HttpServer2: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context hdfs
16/10/18 15:54:48 INFO http.HttpServer2: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
16/10/18 15:54:48 INFO http.HttpServer2: Added filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
16/10/18 15:54:48 INFO http.HttpServer2: Added filter
'org.apache.hadoop.hdfs.web.AuthFilter'
(class=org.apache.hadoop.hdfs.web.AuthFilter)
16/10/18 15:54:48 INFO http.HttpServer2: addJerseyResourcePackage:
packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*
16/10/18 15:54:48 INFO http.HttpServer2: Jetty bound to port 50070
16/10/18 15:54:48 INFO mortbay.log: jetty-6.1.26
16/10/18 15:54:48 INFO mortbay.log: Started
HttpServer2$selectchannelconnectorwithsafestar...@myservernode.root.server.lan:50070
16/10/18 15:54:48 WARN namenode.FSNamesystem: Only one image storage directory
(dfs.namenode.name.dir) configured. Beware of data loss due to lack of
redundant storage directories!
16/10/18 15:54:48 INFO namenode.FSNamesystem: No KeyProvider found.
16/10/18 15:54:48 INFO namenode.FSNamesystem: fsLock is fair:true
16/10/18 15:54:48 INFO blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
16/10/18 15:54:48 INFO blockmanagement.DatanodeManager:
dfs.namenode.datanode.registration.ip-hostname-check=true
16/10/18 15:54:48 INFO blockmanagement.BlockManager:
dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
16/10/18 15:54:48 INFO blockmanagement.BlockManager: The block deletion will
start around 2016 Oct 18 15:54:48
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map BlocksMap
16/10/18 15:54:48 INFO util.GSet: VM type = 64-bit
16/10/18 15:54:48 INFO util.GSet: 2.0% max memory 455 MB = 9.1 MB
16/10/18 15:54:48 INFO util.GSet: capacity = 2^20 = 1048576 entries
16/10/18 15:54:48 INFO blockmanagement.BlockManager:
dfs.block.access.token.enable=false
16/10/18 15:54:48 INFO blockmanagement.BlockManager: defaultReplication
= 3
16/10/18 15:54:48 INFO blockmanagement.BlockManager: maxReplication
= 512
16/10/18 15:54:48 INFO blockmanagement.BlockManager: minReplication
= 1
16/10/18 15:54:48 INFO blockmanagement.BlockManager: maxReplicationStreams
= 2
16/10/18 15:54:48 INFO blockmanagement.BlockManager: replicationRecheckInterval
= 3000
16/10/18 15:54:48 INFO blockmanagement.BlockManager: encryptDataTransfer
= false
16/10/18 15:54:48 INFO blockmanagement.BlockManager: maxNumBlocksToLog
= 1000
16/10/18 15:54:48 INFO namenode.FSNamesystem: fsOwner = SYSTEM
(auth:SIMPLE)
16/10/18 15:54:48 INFO namenode.FSNamesystem: supergroup = supergroup
16/10/18 15:54:48 INFO namenode.FSNamesystem: isPermissionEnabled = false
16/10/18 15:54:48 INFO namenode.FSNamesystem: Determined nameservice ID:
hacluster
16/10/18 15:54:48 INFO namenode.FSNamesystem: HA Enabled: true
16/10/18 15:54:48 INFO namenode.FSNamesystem: Append Enabled: true
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map INodeMap
16/10/18 15:54:48 INFO util.GSet: VM type = 64-bit
16/10/18 15:54:48 INFO util.GSet: 1.0% max memory 455 MB = 4.5 MB
16/10/18 15:54:48 INFO util.GSet: capacity = 2^19 = 524288 entries
16/10/18 15:54:48 INFO namenode.FSDirectory: ACLs enabled? false
16/10/18 15:54:48 INFO namenode.FSDirectory: XAttrs enabled? true
16/10/18 15:54:48 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
16/10/18 15:54:48 INFO namenode.NameNode: Caching file names occuring more than
10 times
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map cachedBlocks
16/10/18 15:54:48 INFO util.GSet: VM type = 64-bit
16/10/18 15:54:48 INFO util.GSet: 0.25% max memory 455 MB = 1.1 MB
16/10/18 15:54:48 INFO util.GSet: capacity = 2^17 = 131072 entries
16/10/18 15:54:48 INFO namenode.FSNamesystem:
dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/10/18 15:54:48 INFO namenode.FSNamesystem:
dfs.namenode.safemode.min.datanodes = 0
16/10/18 15:54:48 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension
= 30000
16/10/18 15:54:48 INFO metrics.TopMetrics: NNTop conf:
dfs.namenode.top.window.num.buckets = 10
16/10/18 15:54:48 INFO metrics.TopMetrics: NNTop conf:
dfs.namenode.top.num.users = 10
16/10/18 15:54:48 INFO metrics.TopMetrics: NNTop conf:
dfs.namenode.top.windows.minutes = 1,5,25
16/10/18 15:54:48 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/10/18 15:54:48 INFO namenode.FSNamesystem: Retry cache will use 0.03 of
total heap and retry cache entry expiry time is 600000 millis
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/10/18 15:54:48 INFO util.GSet: VM type = 64-bit
16/10/18 15:54:48 INFO util.GSet: 0.029999999329447746% max memory 455 MB =
139.8 KB
16/10/18 15:54:48 INFO util.GSet: capacity = 2^14 = 16384 entries
16/10/18 15:54:48 INFO common.Storage: Lock on
\Metadata\data\dfs\namenode\in_use.lock acquired by nodename 13228@myservernode
16/10/18 15:54:49 INFO client.QuorumJournalManager: Starting recovery process
for unclosed journal segments...
16/10/18 15:54:49 INFO client.QuorumJournalManager: Successfully started new
epoch 6
16/10/18 15:54:49 INFO client.QuorumJournalManager: Beginning recovery of
unclosed segment starting at txid 5879
16/10/18 15:54:49 INFO client.QuorumJournalManager: Recovery prepare phase
complete. Responses:
192.168.1.22:8485: segmentState { startTxId: 5879 endTxId: 5942 isInProgress:
true } lastWriterEpoch: 5 lastCommittedTxId: 5941
192.168.1.23:8485: segmentState { startTxId: 5879 endTxId: 5942 isInProgress:
true } lastWriterEpoch: 5 lastCommittedTxId: 5941
16/10/18 15:54:49 INFO client.QuorumJournalManager: Using longest log:
172.16.100.93:8485=segmentState {
startTxId: 5879
endTxId: 5942
isInProgress: true
}
lastWriterEpoch: 5
lastCommittedTxId: 5941
16/10/18 15:54:49 INFO namenode.FileJournalManager: Recovering unfinalized
segments in \Metadata\data\dfs\namenode\current
16/10/18 15:54:49 INFO namenode.FileJournalManager: Finalizing edits file
\Metadata\data\dfs\namenode\current\edits_inprogress_0000000000000005718 ->
\Metadata\data\dfs\namenode\current\edits_0000000000000005718-0000000000000005718
16/10/18 15:54:50 ERROR namenode.FSImage: Failed to load image from
FSImageFile(file=\Metadata\data\dfs\namenode\current\fsimage_rollback_0000000000000005713,
cpktTxId=0000000000000005713)
java.io.IOException: Image version -57 is not equal to the software version -63
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:196)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:179)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:957)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:941)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:740)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:676)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/18 15:54:50 WARN namenode.FSNamesystem: Encountered exception loading
fsimage
java.io.IOException: Failed to load an FSImage file!
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:687)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/18 15:54:50 INFO mortbay.log: Stopped
HttpServer2$selectchannelconnectorwithsafestar...@myservernode.root.server.lan:50070
16/10/18 15:54:50 INFO impl.MetricsSystemImpl: Stopping NameNode metrics
system...
16/10/18 15:54:50 INFO impl.MetricsSystemImpl: NameNode metrics system stopped.
16/10/18 15:54:50 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown
complete.
16/10/18 15:54:50 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Failed to load an FSImage file!
at
org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:687)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/18 15:54:50 INFO util.ExitUtil: Exiting with status 1
16/10/18 15:54:50 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at myservernode/192.168.1.21
************************************************************/|
> Issue in Rollback (after rolling upgrade) from hadoop 2.7.1 to 2.4.0
> --------------------------------------------------------------------
>
> Key: HDFS-9096
> URL: https://issues.apache.org/jira/browse/HDFS-9096
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rolling upgrades
> Affects Versions: 2.4.0
> Reporter: Harpreet Kaur
>
> I tried to do rolling upgrade from hadoop 2.4.0 to hadoop 2.7.1. As per
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#dfsadmin_-rollingUpgrade
> one can rollback to previous release provided the finalise step is not done.
> I upgraded the setup but didnot finalise the upgrade and tried to rollback
> HDFS to 2.4.0
> I tried the following steps
> 1. Shutdown all NNs and DNs.
> 2. Restore the pre-upgrade release in all machines.
> 3. Start NN1 as Active with the "-rollingUpgrade
> rollback<http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#namenode_-rollingUpgrade>"
> option.
> I am getting the following error after 3rd step
> 15/09/01 17:53:35 INFO namenode.AclConfigFlag: ACLs enabled? false
> 15/09/01 17:53:35 INFO common.Storage: Lock on <<NameNode dir>>/in_use.lock
> acquired by nodename 12152@VM-2
> 15/09/01 17:53:35 WARN namenode.FSNamesystem: Encountered exception loading
> fsimage
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /data/yarn/namenode. Reported: -63. Expecting =
> -56.
> at
> org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
> at
> org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
> at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:608)
> at
> org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:309)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:882)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:639)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1370)
> 15/09/01 17:53:35 INFO mortbay.log: Stopped
> SelectChannelConnector@0.0.0.0:50070
> 15/09/01 17:53:35 INFO impl.MetricsSystemImpl: Stopping NameNode metrics
> system...
> 15/09/01 17:53:35 INFO impl.MetricsSystemImpl: NameNode metrics system
> stopped.
> 15/09/01 17:53:35 INFO impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
> 15/09/01 17:53:35 FATAL namenode.NameNode: Exception in namenode join
> From rolling upgrade documentation it can be inferred that rolling upgrade is
> supported Hadoop 2.4.0 onwards but rollingUpgrade rollback to Hadoop 2.4.0
> seems to be broken in Hadoop 2.4.0. It throws above mentioned error.
> Are there any other steps to perform rollback (from rolling upgrade) or is it
> not supported to rollback to Hadoop 2.4.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
