Hi,
I am receiving lots of warning messages in namenodes logs on ACTIVE NN in
my HA Hadoop setup. Below are the logs
"2017-06-21 12:11:26,523 WARN
org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough
replicas: expected size is 1 but only 0 storage types can be selected
(replication=2, selected=[], unavailable=[DISK], removed=[DISK],
policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[],
replicationFallbacks=[ARCHIVE]})
2017-06-21 12:11:26,523 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed
to place enough replicas, still in need of 1 to reach 2
(unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7,
storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
newBlock=true) All required storage types are unavailable:
unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7,
storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
2017-06-21 12:11:26,523 INFO org.apache.hadoop.hdfs.StateChange: BLOCK*
allocate blk_1073894332_153508, replicas=192.168.9.174:50010 for
/36962._COPYING_
2017-06-21 12:11:26,810 INFO org.apache.hadoop.hdfs.StateChange: DIR*
completeFile: /36962._COPYING_ is closed by
DFSClient_NONMAPREDUCE_146762699_1
2017-06-21 12:11:30,626 WARN
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed
to place enough replicas, still in need of 1 to reach 2
(unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7,
storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
newBlock=true) For more information, please enable DEBUG log level on
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy and
org.apache.hadoop.net.NetworkTopology
2017-06-21 12:11:30,626 WARN
org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough
replicas: expected size is 1 but only 0 storage types can be selected
(replication=2, selected=[], unavailable=[DISK], removed=[DISK],
policy=BlockStoragePolicy{HOT:7, storageTypes=[DISK], creationFallbacks=[],
replicationFallbacks=[ARCHIVE]})"
I am also encountering exceptions in active namenode related to LeaseManager
2017-06-21 12:13:16,706 INFO
org.apache.hadoop.hdfs.server.namenode.LeaseManager: [Lease. Holder:
DFSClient_NONMAPREDUCE_409197282_362092, pending creates: 1] has expired
hard limit
2017-06-21 12:13:16,706 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Recovering [Lease.
Holder: DFSClient_NONMAPREDUCE_409197282_362092, pending creates: 1],
src=/user/hadoop/2106201707/02d5adda-d90f-47cb-85d5-999a079f4d79
2017-06-21 12:13:16,706 WARN org.apache.hadoop.hdfs.StateChange: DIR*
NameSystem.internalReleaseLease: Failed to release lease for file
/user/hadoop/2106201707/02d5adda-d90f-47cb-85d5-999a079f4d79. Committed
blocks are waiting to be minimally replicated. Try again later.
2017-06-21 12:13:16,706 ERROR
org.apache.hadoop.hdfs.server.namenode.LeaseManager: Cannot release the path
/user/hadoop/2106201707/02d5adda-d90f-47cb-85d5-999a079f4d79 in the lease
[Lease. Holder: DFSClient_NONMAPREDUCE_409197282_362092, pending creates:
1]
org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException: DIR*
NameSystem.internalReleaseLease: Failed to release lease for file
/user/hadoop/2106201707/02d5adda-d90f-47cb-85d5-999a079f4d79. Committed
blocks are waiting to be minimally replicated. Try again later.
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.internalReleaseLease(FSN
amesystem.java:3200)
at
org.apache.hadoop.hdfs.server.namenode.LeaseManager.checkLeases(LeaseManager
.java:383)
at
org.apache.hadoop.hdfs.server.namenode.LeaseManager$Monitor.run(LeaseManager
.java:329)
at java.lang.Thread.run(Thread.java:745)
I have checked the two datanodes. Both are running and have enough space for
new data.
PS: I have 2 Namenode and 2 datanodes in Hadoop HA setup. The HA is setuped
using Qourom Journal Manager and Zookeeper server.
Any idea why these errors?
Regards
Omprakash Paliwal
HPC-Medical and Bioinformatics Applications Group
Centre for Development of Advanced Computing (C-DAC)
Pune University campus,
PUNE-411007
Maharashtra, India
email:ompraka...@cdac.in
Contact : +91-20-25704231
-------------------------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]
This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
-------------------------------------------------------------------------------------------------------------------------------
