[jira] [Updated] (HDFS-5445) PacketReceiver populates the packetLen field in PacketHeader incorrectly
[
https://issues.apache.org/jira/browse/HDFS-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kengo Seki updated HDFS-5445:
-
Status: Patch Available (was: Open)
PacketReceiver populates the packetLen field in PacketHeader incorrectly
Key: HDFS-5445
URL: https://issues.apache.org/jira/browse/HDFS-5445
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.2.0, 2.1.0-beta
Environment: Ubuntu 12.10, Hadoop 2.1.0-beta
Reporter: Jonathan Mace
Priority: Minor
Labels: easyfix
Attachments: HDFS-5445.001.patch
Original Estimate: 1h
Remaining Estimate: 1h
Summary: PacketReceiver reconstructs PacketHeaders with a packetLen 4 bytes
fewer than it should be. It doesn't cause any exceptions because the
reconstructed header is never reserialized, and the packetLen field is not
used in this part of the code.
In the BlockSender class, when a Packet is constructed it must be passed the
field packetLen, which is defined as the data length, checksum data length,
PLUS the length of the packetLen field itself (4 byte integer).
{code:title=BlockSender.java|borderStyle=solid}
484: private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
485: boolean transferTo, DataTransferThrottler throttler) throws
IOException {
...
491:int packetLen = dataLen + checksumDataLen + 4;
...
504:int headerLen = writePacketHeader(pkt, dataLen, packetLen);
...
586: }
...
792: private int writePacketHeader(ByteBuffer pkt, int dataLen, int
packetLen) {
793:pkt.clear();
794:// both syncBlock and syncPacket are false
795:PacketHeader header = new PacketHeader(packetLen, offset, seqno,
796:(dataLen == 0), dataLen, false);
...
802: }
{code}
In the PacketReceiver class, the PacketHeader is reconstructed using the
method setFieldsFromData. However, the 4 bytes for the packetLen field
length are missing.
{code:title=PacketReceiver.java|borderStyle=solid}
112: private void doRead(ReadableByteChannel ch, InputStream in)
113: throws IOException {
...
136:int payloadLen = curPacketBuf.getInt();
...
144:int dataPlusChecksumLen = payloadLen - Ints.BYTES;
...
181:curHeader.setFieldsFromData(dataPlusChecksumLen, headerBuf);
...
192: }
{code}
The solution would be instead to do:
{code:title=PacketReceiver.java|borderStyle=solid}
181:curHeader.setFieldsFromData(payloadLen, headerBuf);
{code}
I found this because I was making small modifications to the code that
exposed this inconsistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-5445) PacketReceiver populates the packetLen field in PacketHeader incorrectly
[
https://issues.apache.org/jira/browse/HDFS-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kengo Seki updated HDFS-5445:
-
Attachment: HDFS-5445.001.patch
Patch attached.
PacketReceiver populates the packetLen field in PacketHeader incorrectly
Key: HDFS-5445
URL: https://issues.apache.org/jira/browse/HDFS-5445
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.0-beta, 2.2.0
Environment: Ubuntu 12.10, Hadoop 2.1.0-beta
Reporter: Jonathan Mace
Priority: Minor
Labels: easyfix
Attachments: HDFS-5445.001.patch
Original Estimate: 1h
Remaining Estimate: 1h
Summary: PacketReceiver reconstructs PacketHeaders with a packetLen 4 bytes
fewer than it should be. It doesn't cause any exceptions because the
reconstructed header is never reserialized, and the packetLen field is not
used in this part of the code.
In the BlockSender class, when a Packet is constructed it must be passed the
field packetLen, which is defined as the data length, checksum data length,
PLUS the length of the packetLen field itself (4 byte integer).
{code:title=BlockSender.java|borderStyle=solid}
484: private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
485: boolean transferTo, DataTransferThrottler throttler) throws
IOException {
...
491:int packetLen = dataLen + checksumDataLen + 4;
...
504:int headerLen = writePacketHeader(pkt, dataLen, packetLen);
...
586: }
...
792: private int writePacketHeader(ByteBuffer pkt, int dataLen, int
packetLen) {
793:pkt.clear();
794:// both syncBlock and syncPacket are false
795:PacketHeader header = new PacketHeader(packetLen, offset, seqno,
796:(dataLen == 0), dataLen, false);
...
802: }
{code}
In the PacketReceiver class, the PacketHeader is reconstructed using the
method setFieldsFromData. However, the 4 bytes for the packetLen field
length are missing.
{code:title=PacketReceiver.java|borderStyle=solid}
112: private void doRead(ReadableByteChannel ch, InputStream in)
113: throws IOException {
...
136:int payloadLen = curPacketBuf.getInt();
...
144:int dataPlusChecksumLen = payloadLen - Ints.BYTES;
...
181:curHeader.setFieldsFromData(dataPlusChecksumLen, headerBuf);
...
192: }
{code}
The solution would be instead to do:
{code:title=PacketReceiver.java|borderStyle=solid}
181:curHeader.setFieldsFromData(payloadLen, headerBuf);
{code}
I found this because I was making small modifications to the code that
exposed this inconsistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-3443) Unable to catch up edits during standby to active switch due to NPE
[
https://issues.apache.org/jira/browse/HDFS-3443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272551#comment-14272551
]
Tsz Wo Nicholas Sze commented on HDFS-3443:
---
It seems that trunk still has this bug. How about adding a boolean for
indicating namenode starting up so that NameNodeRpcServer could refuse all
operations?
Unable to catch up edits during standby to active switch due to NPE
---
Key: HDFS-3443
URL: https://issues.apache.org/jira/browse/HDFS-3443
Project: Hadoop HDFS
Issue Type: Bug
Components: auto-failover
Reporter: suja s
Assignee: amith
Attachments: HDFS-3443_1.patch, HDFS-3443_1.patch
Start NN
Let NN standby services be started.
Before the editLogTailer is initialised start ZKFC and allow the
activeservices start to proceed further.
Here editLogTailer.catchupDuringFailover() will throw NPE.
void startActiveServices() throws IOException {
LOG.info(Starting services required for active state);
writeLock();
try {
FSEditLog editLog = dir.fsImage.getEditLog();
if (!editLog.isOpenForWrite()) {
// During startup, we're already open for write during initialization.
editLog.initJournalsForWrite();
// May need to recover
editLog.recoverUnclosedStreams();
LOG.info(Catching up to latest edits from old active before +
taking over writer role in edits logs.);
editLogTailer.catchupDuringFailover();
{noformat}
2012-05-18 16:51:27,585 WARN org.apache.hadoop.ipc.Server: IPC Server
Responder, call org.apache.hadoop.ha.HAServiceProtocol.getServiceStatus from
XX.XX.XX.55:58003: output error
2012-05-18 16:51:27,586 WARN org.apache.hadoop.ipc.Server: IPC Server handler
8 on 8020, call org.apache.hadoop.ha.HAServiceProtocol.transitionToActive
from XX.XX.XX.55:58004: error: java.lang.NullPointerException
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:602)
at
org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.startActiveServices(NameNode.java:1287)
at
org.apache.hadoop.hdfs.server.namenode.ha.ActiveState.enterState(ActiveState.java:61)
at
org.apache.hadoop.hdfs.server.namenode.ha.HAState.setStateInternal(HAState.java:63)
at
org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.setState(StandbyState.java:49)
at
org.apache.hadoop.hdfs.server.namenode.NameNode.transitionToActive(NameNode.java:1219)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.transitionToActive(NameNodeRpcServer.java:978)
at
org.apache.hadoop.ha.protocolPB.HAServiceProtocolServerSideTranslatorPB.transitionToActive(HAServiceProtocolServerSideTranslatorPB.java:107)
at
org.apache.hadoop.ha.proto.HAServiceProtocolProtos$HAServiceProtocolService$2.callBlockingMethod(HAServiceProtocolProtos.java:3633)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:427)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)
2012-05-18 16:51:27,586 INFO org.apache.hadoop.ipc.Server: IPC Server handler
9 on 8020 caught an exception
java.nio.channels.ClosedChannelException
at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:133)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
at org.apache.hadoop.ipc.Server.channelWrite(Server.java:2092)
at org.apache.hadoop.ipc.Server.access$2000(Server.java:107)
at
org.apache.hadoop.ipc.Server$Responder.processResponse(Server.java:930)
at org.apache.hadoop.ipc.Server$Responder.doRespond(Server.java:994)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1738)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7182) JMX metrics aren't accessible when NN is busy
[ https://issues.apache.org/jira/browse/HDFS-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272517#comment-14272517 ] Hudson commented on HDFS-7182: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #2001 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2001/]) HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt JMX metrics aren't accessible when NN is busy - Key: HDFS-7182 URL: https://issues.apache.org/jira/browse/HDFS-7182 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Fix For: 2.7.0 Attachments: HDFS-7182-2.patch, HDFS-7182-3.patch, HDFS-7182.patch HDFS-5693 has addressed all NN JMX metrics in hadoop 2.0.5. Since then couple new metrics have been added. It turns out RollingUpgradeStatus requires FSNamesystem read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7342) Lease Recovery doesn't happen some times
[ https://issues.apache.org/jira/browse/HDFS-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HDFS-7342: --- Fix Version/s: (was: 2.7.0) Lease Recovery doesn't happen some times Key: HDFS-7342 URL: https://issues.apache.org/jira/browse/HDFS-7342 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Ravi Prakash Attachments: HDFS-7342.1.patch, HDFS-7342.2.patch, HDFS-7342.3.patch In some cases, LeaseManager tries to recover a lease, but is not able to. HDFS-4882 describes a possibility of that. We should fix this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7596) NameNode should prune dead storages from storageMap
[ https://issues.apache.org/jira/browse/HDFS-7596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-7596: Resolution: Fixed Fix Version/s: 2.7.0 Status: Resolved (was: Patch Available) +1 for patch v02. I committed this to trunk and branch-2. Arpit, thank you for the patch. Lars, thank you for help with the code review. NameNode should prune dead storages from storageMap --- Key: HDFS-7596 URL: https://issues.apache.org/jira/browse/HDFS-7596 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 2.7.0 Attachments: HDFS-7596.01.patch, HDFS-7596.02.patch The NameNode must be able to prune storages that are no longer reported by the DataNode and that have no blocks associated. These stale storages can skew the balancer behavior. Detailed discussion on HDFS-7575. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7182) JMX metrics aren't accessible when NN is busy
[ https://issues.apache.org/jira/browse/HDFS-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272451#comment-14272451 ] Hudson commented on HDFS-7182: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #69 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/69/]) HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java JMX metrics aren't accessible when NN is busy - Key: HDFS-7182 URL: https://issues.apache.org/jira/browse/HDFS-7182 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Fix For: 2.7.0 Attachments: HDFS-7182-2.patch, HDFS-7182-3.patch, HDFS-7182.patch HDFS-5693 has addressed all NN JMX metrics in hadoop 2.0.5. Since then couple new metrics have been added. It turns out RollingUpgradeStatus requires FSNamesystem read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7182) JMX metrics aren't accessible when NN is busy
[ https://issues.apache.org/jira/browse/HDFS-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272465#comment-14272465 ] Hudson commented on HDFS-7182: -- FAILURE: Integrated in Hadoop-Yarn-trunk #803 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/803/]) HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java JMX metrics aren't accessible when NN is busy - Key: HDFS-7182 URL: https://issues.apache.org/jira/browse/HDFS-7182 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Fix For: 2.7.0 Attachments: HDFS-7182-2.patch, HDFS-7182-3.patch, HDFS-7182.patch HDFS-5693 has addressed all NN JMX metrics in hadoop 2.0.5. Since then couple new metrics have been added. It turns out RollingUpgradeStatus requires FSNamesystem read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-5445) PacketReceiver populates the packetLen field in PacketHeader incorrectly
[
https://issues.apache.org/jira/browse/HDFS-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272482#comment-14272482
]
Hadoop QA commented on HDFS-5445:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12691488/HDFS-5445.001.patch
against trunk revision a260406.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestCacheDirectives
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9180//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9180//console
This message is automatically generated.
PacketReceiver populates the packetLen field in PacketHeader incorrectly
Key: HDFS-5445
URL: https://issues.apache.org/jira/browse/HDFS-5445
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.0-beta, 2.2.0
Environment: Ubuntu 12.10, Hadoop 2.1.0-beta
Reporter: Jonathan Mace
Priority: Minor
Labels: easyfix
Attachments: HDFS-5445.001.patch
Original Estimate: 1h
Remaining Estimate: 1h
Summary: PacketReceiver reconstructs PacketHeaders with a packetLen 4 bytes
fewer than it should be. It doesn't cause any exceptions because the
reconstructed header is never reserialized, and the packetLen field is not
used in this part of the code.
In the BlockSender class, when a Packet is constructed it must be passed the
field packetLen, which is defined as the data length, checksum data length,
PLUS the length of the packetLen field itself (4 byte integer).
{code:title=BlockSender.java|borderStyle=solid}
484: private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
485: boolean transferTo, DataTransferThrottler throttler) throws
IOException {
...
491:int packetLen = dataLen + checksumDataLen + 4;
...
504:int headerLen = writePacketHeader(pkt, dataLen, packetLen);
...
586: }
...
792: private int writePacketHeader(ByteBuffer pkt, int dataLen, int
packetLen) {
793:pkt.clear();
794:// both syncBlock and syncPacket are false
795:PacketHeader header = new PacketHeader(packetLen, offset, seqno,
796:(dataLen == 0), dataLen, false);
...
802: }
{code}
In the PacketReceiver class, the PacketHeader is reconstructed using the
method setFieldsFromData. However, the 4 bytes for the packetLen field
length are missing.
{code:title=PacketReceiver.java|borderStyle=solid}
112: private void doRead(ReadableByteChannel ch, InputStream in)
113: throws IOException {
...
136:int payloadLen = curPacketBuf.getInt();
...
144:int dataPlusChecksumLen = payloadLen - Ints.BYTES;
...
181:curHeader.setFieldsFromData(dataPlusChecksumLen, headerBuf);
...
192: }
{code}
The solution would be instead to do:
{code:title=PacketReceiver.java|borderStyle=solid}
181:curHeader.setFieldsFromData(payloadLen, headerBuf);
{code}
I found this because I was making small modifications to the code that
exposed this inconsistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-5445) PacketReceiver populates the packetLen field in PacketHeader incorrectly
[
https://issues.apache.org/jira/browse/HDFS-5445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272523#comment-14272523
]
Kengo Seki commented on HDFS-5445:
--
Unit test failure seems unrelated this patch. See HDFS-7571.
PacketReceiver populates the packetLen field in PacketHeader incorrectly
Key: HDFS-5445
URL: https://issues.apache.org/jira/browse/HDFS-5445
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.1.0-beta, 2.2.0
Environment: Ubuntu 12.10, Hadoop 2.1.0-beta
Reporter: Jonathan Mace
Priority: Minor
Labels: easyfix
Attachments: HDFS-5445.001.patch
Original Estimate: 1h
Remaining Estimate: 1h
Summary: PacketReceiver reconstructs PacketHeaders with a packetLen 4 bytes
fewer than it should be. It doesn't cause any exceptions because the
reconstructed header is never reserialized, and the packetLen field is not
used in this part of the code.
In the BlockSender class, when a Packet is constructed it must be passed the
field packetLen, which is defined as the data length, checksum data length,
PLUS the length of the packetLen field itself (4 byte integer).
{code:title=BlockSender.java|borderStyle=solid}
484: private int sendPacket(ByteBuffer pkt, int maxChunks, OutputStream out,
485: boolean transferTo, DataTransferThrottler throttler) throws
IOException {
...
491:int packetLen = dataLen + checksumDataLen + 4;
...
504:int headerLen = writePacketHeader(pkt, dataLen, packetLen);
...
586: }
...
792: private int writePacketHeader(ByteBuffer pkt, int dataLen, int
packetLen) {
793:pkt.clear();
794:// both syncBlock and syncPacket are false
795:PacketHeader header = new PacketHeader(packetLen, offset, seqno,
796:(dataLen == 0), dataLen, false);
...
802: }
{code}
In the PacketReceiver class, the PacketHeader is reconstructed using the
method setFieldsFromData. However, the 4 bytes for the packetLen field
length are missing.
{code:title=PacketReceiver.java|borderStyle=solid}
112: private void doRead(ReadableByteChannel ch, InputStream in)
113: throws IOException {
...
136:int payloadLen = curPacketBuf.getInt();
...
144:int dataPlusChecksumLen = payloadLen - Ints.BYTES;
...
181:curHeader.setFieldsFromData(dataPlusChecksumLen, headerBuf);
...
192: }
{code}
The solution would be instead to do:
{code:title=PacketReceiver.java|borderStyle=solid}
181:curHeader.setFieldsFromData(payloadLen, headerBuf);
{code}
I found this because I was making small modifications to the code that
exposed this inconsistency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7182) JMX metrics aren't accessible when NN is busy
[ https://issues.apache.org/jira/browse/HDFS-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272542#comment-14272542 ] Hudson commented on HDFS-7182: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2020 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2020/]) HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java JMX metrics aren't accessible when NN is busy - Key: HDFS-7182 URL: https://issues.apache.org/jira/browse/HDFS-7182 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Fix For: 2.7.0 Attachments: HDFS-7182-2.patch, HDFS-7182-3.patch, HDFS-7182.patch HDFS-5693 has addressed all NN JMX metrics in hadoop 2.0.5. Since then couple new metrics have been added. It turns out RollingUpgradeStatus requires FSNamesystem read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7548) Corrupt block reporting delayed until datablock scanner thread detects it
[ https://issues.apache.org/jira/browse/HDFS-7548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-7548: - Status: Open (was: Patch Available) Thanks Daryn for the review. Cancelling the current patch to address Daryn's comments. Will update the patch shortly. @Nathan: What action needs to be taken if java.io.IOException: Input/output error occurs. Corrupt block reporting delayed until datablock scanner thread detects it - Key: HDFS-7548 URL: https://issues.apache.org/jira/browse/HDFS-7548 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.5.0 Reporter: Rushabh S Shah Assignee: Rushabh S Shah Attachments: HDFS-7548.patch When there is one datanode holding the block and that block happened to be corrupt, namenode would keep on trying to replicate the block repeatedly but it would only report the block as corrupt only when the data block scanner thread of the datanode picks up this bad block. Requesting improvement in namenode reporting so that corrupt replica would be reported when there is only 1 replica and the replication of that replica keeps on failing with the checksum error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7182) JMX metrics aren't accessible when NN is busy
[ https://issues.apache.org/jira/browse/HDFS-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272525#comment-14272525 ] Hudson commented on HDFS-7182: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk-Java8 #66 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/66/]) HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt JMX metrics aren't accessible when NN is busy - Key: HDFS-7182 URL: https://issues.apache.org/jira/browse/HDFS-7182 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Fix For: 2.7.0 Attachments: HDFS-7182-2.patch, HDFS-7182-3.patch, HDFS-7182.patch HDFS-5693 has addressed all NN JMX metrics in hadoop 2.0.5. Since then couple new metrics have been added. It turns out RollingUpgradeStatus requires FSNamesystem read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7182) JMX metrics aren't accessible when NN is busy
[ https://issues.apache.org/jira/browse/HDFS-7182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272534#comment-14272534 ] Hudson commented on HDFS-7182: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #70 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/70/]) HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. (jing9: rev 2a06101e61f2f7a08913c2367d9dbefb910e64a7) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt JMX metrics aren't accessible when NN is busy - Key: HDFS-7182 URL: https://issues.apache.org/jira/browse/HDFS-7182 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ming Ma Assignee: Ming Ma Fix For: 2.7.0 Attachments: HDFS-7182-2.patch, HDFS-7182-3.patch, HDFS-7182.patch HDFS-5693 has addressed all NN JMX metrics in hadoop 2.0.5. Since then couple new metrics have been added. It turns out RollingUpgradeStatus requires FSNamesystem read lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7342) Lease Recovery doesn't happen some times
[ https://issues.apache.org/jira/browse/HDFS-7342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Prakash updated HDFS-7342: --- Fix Version/s: 2.7.0 Lease Recovery doesn't happen some times Key: HDFS-7342 URL: https://issues.apache.org/jira/browse/HDFS-7342 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Ravi Prakash Attachments: HDFS-7342.1.patch, HDFS-7342.2.patch, HDFS-7342.3.patch In some cases, LeaseManager tries to recover a lease, but is not able to. HDFS-4882 describes a possibility of that. We should fix this -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7596) NameNode should prune dead storages from storageMap
[ https://issues.apache.org/jira/browse/HDFS-7596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272616#comment-14272616 ] Hudson commented on HDFS-7596: -- FAILURE: Integrated in Hadoop-trunk-Commit #6841 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/6841/]) HDFS-7596. NameNode should prune dead storages from storageMap. Contributed by Arpit Agarwal. (cnauroth: rev ef3c3a832c2f0c1e5ccdda2ff8ef84902912955f) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeDescriptor.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java NameNode should prune dead storages from storageMap --- Key: HDFS-7596 URL: https://issues.apache.org/jira/browse/HDFS-7596 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 2.7.0 Attachments: HDFS-7596.01.patch, HDFS-7596.02.patch The NameNode must be able to prune storages that are no longer reported by the DataNode and that have no blocks associated. These stale storages can skew the balancer behavior. Detailed discussion on HDFS-7575. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7596) NameNode should prune dead storages from storageMap
[ https://issues.apache.org/jira/browse/HDFS-7596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272653#comment-14272653 ] Arpit Agarwal commented on HDFS-7596: - Thanks for the review and commit Chris! NameNode should prune dead storages from storageMap --- Key: HDFS-7596 URL: https://issues.apache.org/jira/browse/HDFS-7596 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 2.7.0 Attachments: HDFS-7596.01.patch, HDFS-7596.02.patch The NameNode must be able to prune storages that are no longer reported by the DataNode and that have no blocks associated. These stale storages can skew the balancer behavior. Detailed discussion on HDFS-7575. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7533) Datanode sometimes does not shutdown on receiving upgrade shutdown command
[
https://issues.apache.org/jira/browse/HDFS-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272814#comment-14272814
]
Hadoop QA commented on HDFS-7533:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12691530/HDFS-7533.v1.txt
against trunk revision ef3c3a8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/9181//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9181//console
This message is automatically generated.
Datanode sometimes does not shutdown on receiving upgrade shutdown command
--
Key: HDFS-7533
URL: https://issues.apache.org/jira/browse/HDFS-7533
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Kihwal Lee
Assignee: Eric Payne
Attachments: HDFS-7533.v1.txt
When datanode is told to shutdown via the dfsadmin command during rolling
upgrade, it may not shutdown. This is because not all writers have responder
running, but sendOOB() tries anyway. This causes NPE and the shutdown thread
dies, halting the shutdown after only shutting down DataXceiverServer.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7533) Datanode sometimes does not shutdown on receiving upgrade shutdown command
[ https://issues.apache.org/jira/browse/HDFS-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated HDFS-7533: - Status: Patch Available (was: Open) Datanode sometimes does not shutdown on receiving upgrade shutdown command -- Key: HDFS-7533 URL: https://issues.apache.org/jira/browse/HDFS-7533 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Assignee: Eric Payne Attachments: HDFS-7533.v1.txt When datanode is told to shutdown via the dfsadmin command during rolling upgrade, it may not shutdown. This is because not all writers have responder running, but sendOOB() tries anyway. This causes NPE and the shutdown thread dies, halting the shutdown after only shutting down DataXceiverServer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7533) Datanode sometimes does not shutdown on receiving upgrade shutdown command
[ https://issues.apache.org/jira/browse/HDFS-7533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Payne updated HDFS-7533: - Attachment: HDFS-7533.v1.txt Datanode sometimes does not shutdown on receiving upgrade shutdown command -- Key: HDFS-7533 URL: https://issues.apache.org/jira/browse/HDFS-7533 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Assignee: Eric Payne Attachments: HDFS-7533.v1.txt When datanode is told to shutdown via the dfsadmin command during rolling upgrade, it may not shutdown. This is because not all writers have responder running, but sendOOB() tries anyway. This causes NPE and the shutdown thread dies, halting the shutdown after only shutting down DataXceiverServer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
