[jira] [Commented] (HDFS-7381) Decouple the management of block id and gen stamps from FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203342#comment-14203342
]
Hadoop QA commented on HDFS-7381:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680304/HDFS-7381.000.patch
against trunk revision 9a4e0d3.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
See
https://builds.apache.org/job/PreCommit-HDFS-Build/8698//artifact/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDFSUpgradeFromImage
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.qjournal.client.TestQuorumJournalManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8698//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8698//console
This message is automatically generated.
Decouple the management of block id and gen stamps from FSNamesystem
Key: HDFS-7381
URL: https://issues.apache.org/jira/browse/HDFS-7381
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7381.000.patch
The block layer should be responsible of managing block ids and generation
stamps. Currently the functionality is misplace into {{FSNamesystem}}.
This jira proposes to decouple them from the {{FSNamesystem}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7379) TestBalancer#testBalancerWithRamDisk creates test files incorrectly
[ https://issues.apache.org/jira/browse/HDFS-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203392#comment-14203392 ] Hudson commented on HDFS-7379: -- FAILURE: Integrated in Hadoop-Yarn-trunk #737 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/737/]) HDFS-7379. TestBalancer#testBalancerWithRamDisk creates test files incorrectly. Contributed by Xiaoyu Yao. (wheat9: rev 57760c0663288a7611c3609891ef92f1abf4bb53) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java TestBalancer#testBalancerWithRamDisk creates test files incorrectly --- Key: HDFS-7379 URL: https://issues.apache.org/jira/browse/HDFS-7379 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.6.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Fix For: 2.6.0 Attachments: HDFS-7379.01.patch There is a copy-paste error during for test file creation. The test is supposed to create two test files named path1 and path2 on RAM_DISK. But The error caused path1 to be created twice with the second creation overwrite (delete) the first one on RAM_DISK. This caused verification failure for path2 as it never gets created. The fix is to create test files with the correct names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7379) TestBalancer#testBalancerWithRamDisk creates test files incorrectly
[ https://issues.apache.org/jira/browse/HDFS-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203437#comment-14203437 ] Hudson commented on HDFS-7379: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1927 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1927/]) HDFS-7379. TestBalancer#testBalancerWithRamDisk creates test files incorrectly. Contributed by Xiaoyu Yao. (wheat9: rev 57760c0663288a7611c3609891ef92f1abf4bb53) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt TestBalancer#testBalancerWithRamDisk creates test files incorrectly --- Key: HDFS-7379 URL: https://issues.apache.org/jira/browse/HDFS-7379 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.6.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Fix For: 2.6.0 Attachments: HDFS-7379.01.patch There is a copy-paste error during for test file creation. The test is supposed to create two test files named path1 and path2 on RAM_DISK. But The error caused path1 to be created twice with the second creation overwrite (delete) the first one on RAM_DISK. This caused verification failure for path2 as it never gets created. The fix is to create test files with the correct names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7379) TestBalancer#testBalancerWithRamDisk creates test files incorrectly
[ https://issues.apache.org/jira/browse/HDFS-7379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203462#comment-14203462 ] Hudson commented on HDFS-7379: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1951 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1951/]) HDFS-7379. TestBalancer#testBalancerWithRamDisk creates test files incorrectly. Contributed by Xiaoyu Yao. (wheat9: rev 57760c0663288a7611c3609891ef92f1abf4bb53) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt TestBalancer#testBalancerWithRamDisk creates test files incorrectly --- Key: HDFS-7379 URL: https://issues.apache.org/jira/browse/HDFS-7379 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.6.0 Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Fix For: 2.6.0 Attachments: HDFS-7379.01.patch There is a copy-paste error during for test file creation. The test is supposed to create two test files named path1 and path2 on RAM_DISK. But The error caused path1 to be created twice with the second creation overwrite (delete) the first one on RAM_DISK. This caused verification failure for path2 as it never gets created. The fix is to create test files with the correct names. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7358) Clients may get stuck waiting when using ByteArrayManager
[ https://issues.apache.org/jira/browse/HDFS-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-7358: -- Attachment: h7358_20141108.patch h7358_20141108.patch: fixes TestHFlush failure. Clients may get stuck waiting when using ByteArrayManager - Key: HDFS-7358 URL: https://issues.apache.org/jira/browse/HDFS-7358 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Attachments: h7358_20141104.patch, h7358_20141104_wait_timeout.patch, h7358_20141105.patch, h7358_20141106.patch, h7358_20141107.patch, h7358_20141108.patch [~stack] reported that clients might get stuck waiting when using ByteArrayManager; see [his comments|https://issues.apache.org/jira/browse/HDFS-7276?focusedCommentId=14197036page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14197036]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
Chris Nauroth created HDFS-7382:
---
Summary: DataNode in secure mode may throw NullPointerException if
client connects before DataNode registers itself with NameNode.
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
{{SaslDataTransferServer#receive}} needs to check if the DataNode is listening
on a privileged port. It does this by checking the address from the
{{DatanodeID}}. However, there is a window of time when this will be {{null}}.
If a client is still holding a {{LocatedBlock}} that references that DataNode
and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7382:
Component/s: security
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7382:
Attachment: HDFS-7382.1.patch
We don't really need to rely on NameNode registration. A DataNode always knows
which port it's listening on for DataTransferProtocol, based solely on its own
local configuration. The attached patch implements the fix. I'm still passing
through the {{DatanodeID}}, because that's still useful for debug logging.
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: HDFS-7382.1.patch
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7382:
Status: Patch Available (was: Open)
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: HDFS-7382.1.patch
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-7382:
--
Hadoop Flags: Reviewed
+1 patch looks good.
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: HDFS-7382.1.patch
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7383) DataNode.requestShortCircuitFdsForRead may throw NullPointerException
Tsz Wo Nicholas Sze created HDFS-7383:
-
Summary: DataNode.requestShortCircuitFdsForRead may throw
NullPointerException
Key: HDFS-7383
URL: https://issues.apache.org/jira/browse/HDFS-7383
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.DataNode.requestShortCircuitFdsForRead(DataNode.java:1525)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitFds(DataXceiver.java:286)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitFds(Receiver.java:185)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:89)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7358) Clients may get stuck waiting when using ByteArrayManager
[
https://issues.apache.org/jira/browse/HDFS-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203698#comment-14203698
]
Hadoop QA commented on HDFS-7358:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680428/h7358_20141108.patch
against trunk revision 737d928.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8699//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8699//artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8699//console
This message is automatically generated.
Clients may get stuck waiting when using ByteArrayManager
-
Key: HDFS-7358
URL: https://issues.apache.org/jira/browse/HDFS-7358
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Attachments: h7358_20141104.patch, h7358_20141104_wait_timeout.patch,
h7358_20141105.patch, h7358_20141106.patch, h7358_20141107.patch,
h7358_20141108.patch
[~stack] reported that clients might get stuck waiting when using
ByteArrayManager; see [his
comments|https://issues.apache.org/jira/browse/HDFS-7276?focusedCommentId=14197036page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14197036].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203697#comment-14203697
]
Hadoop QA commented on HDFS-7382:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680433/HDFS-7382.1.patch
against trunk revision 737d928.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHDFSForHA
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8700//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8700//console
This message is automatically generated.
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: HDFS-7382.1.patch
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7383) DataNode.requestShortCircuitFdsForRead may throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-7383:
--
Status: Patch Available (was: Open)
DataNode.requestShortCircuitFdsForRead may throw NullPointerException
-
Key: HDFS-7383
URL: https://issues.apache.org/jira/browse/HDFS-7383
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Attachments: h7383_20141108.patch
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.DataNode.requestShortCircuitFdsForRead(DataNode.java:1525)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitFds(DataXceiver.java:286)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitFds(Receiver.java:185)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:89)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7383) DataNode.requestShortCircuitFdsForRead may throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-7383:
--
Attachment: h7383_20141108.patch
h7383_20141108.patch: checks null.
DataNode.requestShortCircuitFdsForRead may throw NullPointerException
-
Key: HDFS-7383
URL: https://issues.apache.org/jira/browse/HDFS-7383
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Attachments: h7383_20141108.patch
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.DataNode.requestShortCircuitFdsForRead(DataNode.java:1525)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitFds(DataXceiver.java:286)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitFds(Receiver.java:185)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:89)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7358) Clients may get stuck waiting when using ByteArrayManager
[ https://issues.apache.org/jira/browse/HDFS-7358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203719#comment-14203719 ] Tsz Wo Nicholas Sze commented on HDFS-7358: --- Something wrong with test-patch? This is no findbugs warning in the given newPatchFindbugsWarningshadoop-hdfs.html. Clients may get stuck waiting when using ByteArrayManager - Key: HDFS-7358 URL: https://issues.apache.org/jira/browse/HDFS-7358 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Attachments: h7358_20141104.patch, h7358_20141104_wait_timeout.patch, h7358_20141105.patch, h7358_20141106.patch, h7358_20141107.patch, h7358_20141108.patch [~stack] reported that clients might get stuck waiting when using ByteArrayManager; see [his comments|https://issues.apache.org/jira/browse/HDFS-7276?focusedCommentId=14197036page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14197036]. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7312) Update DistCp v1 to optionally not use tmp location
[ https://issues.apache.org/jira/browse/HDFS-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203727#comment-14203727 ] Yongjun Zhang commented on HDFS-7312: - Hi [~jprosser], Thanks for reporting the issue. FYI, HADOOP-11281.described the same issue. It's nice that you are working on. Update DistCp v1 to optionally not use tmp location --- Key: HDFS-7312 URL: https://issues.apache.org/jira/browse/HDFS-7312 Project: Hadoop HDFS Issue Type: Improvement Components: tools Affects Versions: 2.5.1 Reporter: Joseph Prosser Assignee: Joseph Prosser Priority: Minor Original Estimate: 72h Remaining Estimate: 72h DistCp v1 currently copies files to a tmp location and then renames that to the specified destination. This can cause performance issues on filesystems such as S3. A -skiptmp flag will be added to bypass this step and copy directly to the destination. This feature mirrors a similar one added to HBase ExportSnapshot [HBASE-9|https://issues.apache.org/jira/browse/HBASE-9] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203741#comment-14203741
]
Yongjun Zhang commented on HDFS-7382:
-
HI [~cnauroth],
Thanks for reporting the issue and working on the solution. I reviewed the fix,
it looks good to me. I happen to notice a typo in the related code at line 113:
{code}
} else if (datanodeId.getXferPort() 1024) {
LOG.debug(
SASL server skipping handshake in unsecured configuration for
+ peer = {}, datanodeId = {}, peer, datanodeId);
return new IOStreamPair(underlyingIn, underlyingOut);
} else if (dnConf.getSaslPropsResolver() != null) {
{code}
In the debug message above, the unsecured meant secured. Can we fix it in
this patch?
Another thing is, I wish the 1024 is a constant defined in the security util
somewhere, but it pre-existed. So I think it we were to replace it, it does not
have to be the scope of this jira.
Thanks again.
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: HDFS-7382.1.patch
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7383) DataNode.requestShortCircuitFdsForRead may throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203770#comment-14203770
]
Hadoop QA commented on HDFS-7383:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680441/h7383_20141108.patch
against trunk revision 6caa810.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancer
org.apache.hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.util.TestByteArrayManager
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8701//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8701//console
This message is automatically generated.
DataNode.requestShortCircuitFdsForRead may throw NullPointerException
-
Key: HDFS-7383
URL: https://issues.apache.org/jira/browse/HDFS-7383
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Attachments: h7383_20141108.patch
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.DataNode.requestShortCircuitFdsForRead(DataNode.java:1525)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitFds(DataXceiver.java:286)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitFds(Receiver.java:185)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:89)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7382:
Attachment: HDFS-7382.2.patch
Thank you for the code reviews, Nicholas and Yongjun.
Yongjun, I'm uploading patch v2 with the log message fixed like you suggested.
Nice catch! I'm going to commit this version based on the prior +1 from
Nicholas, and this new version is only changing a string literal in a log
message.
You're right that there are a few different places checking for a port number
1024. We might benefit from a shared {{isPrivilegedPort}} method somewhere.
One interesting aspect is that Windows has no concept of privileged ports, so
perhaps this method would return false always on Windows. I won't address it
here, but I do encourage you to file a jira for that chnage.
bq. -1 tests included. The patch doesn't appear to include any new or modified
tests.
This turned up in internal system testing that involved restart DataNodes while
running jobs. It would be very difficult to put this into a predictable test.
The failure in {{TestWebHDFSForHA}} looks unrelated. The test passed locally.
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: HDFS-7382.1.patch, HDFS-7382.2.patch
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203795#comment-14203795
]
Hadoop QA commented on HDFS-7382:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680464/HDFS-7382.2.patch
against trunk revision 9ba8d8c.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8703//console
This message is automatically generated.
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: HDFS-7382.1.patch, HDFS-7382.2.patch
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203796#comment-14203796
]
Hudson commented on HDFS-7382:
--
FAILURE: Integrated in Hadoop-trunk-Commit #6494 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/6494/])
HDFS-7382. DataNode in secure mode may throw NullPointerException if client
connects before DataNode registers itself with NameNode. Contributed by Chris
Nauroth. (cnauroth: rev 9ba8d8c7eb65eeb6fe673f04e493d9eedd95a822)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiver.java
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: HDFS-7382.1.patch, HDFS-7382.2.patch
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7382) DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode.
[
https://issues.apache.org/jira/browse/HDFS-7382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7382:
Resolution: Fixed
Fix Version/s: 2.6.0
Status: Resolved (was: Patch Available)
I committed this to trunk, branch-2 and branch-2.6. Nicholas and Yongjun,
thank you again for the code reviews.
DataNode in secure mode may throw NullPointerException if client connects
before DataNode registers itself with NameNode.
-
Key: HDFS-7382
URL: https://issues.apache.org/jira/browse/HDFS-7382
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode, security
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Fix For: 2.6.0
Attachments: HDFS-7382.1.patch, HDFS-7382.2.patch
{{SaslDataTransferServer#receive}} needs to check if the DataNode is
listening on a privileged port. It does this by checking the address from
the {{DatanodeID}}. However, there is a window of time when this will be
{{null}}. If a client is still holding a {{LocatedBlock}} that references
that DataNode and chooses to connect, then there is a risk of getting a
{{NullPointerException}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7381) Decouple the management of block id and gen stamps from FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14203801#comment-14203801
]
Hadoop QA commented on HDFS-7381:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12680304/HDFS-7381.000.patch
against trunk revision 6caa810.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:red}-1 eclipse:eclipse{color}. The patch failed to build with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8702//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8702//console
This message is automatically generated.
Decouple the management of block id and gen stamps from FSNamesystem
Key: HDFS-7381
URL: https://issues.apache.org/jira/browse/HDFS-7381
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-7381.000.patch
The block layer should be responsible of managing block ids and generation
stamps. Currently the functionality is misplace into {{FSNamesystem}}.
This jira proposes to decouple them from the {{FSNamesystem}} class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7383) DataNode.requestShortCircuitFdsForRead may throw NullPointerException
[
https://issues.apache.org/jira/browse/HDFS-7383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-7383:
Hadoop Flags: Reviewed
+1 for the patch. The test failures look unrelated, and I verified that the
tests pass in my local environment. Thank you for the patch, Nicholas.
DataNode.requestShortCircuitFdsForRead may throw NullPointerException
-
Key: HDFS-7383
URL: https://issues.apache.org/jira/browse/HDFS-7383
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Tsz Wo Nicholas Sze
Assignee: Tsz Wo Nicholas Sze
Attachments: h7383_20141108.patch
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.DataNode.requestShortCircuitFdsForRead(DataNode.java:1525)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.requestShortCircuitFds(DataXceiver.java:286)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opRequestShortCircuitFds(Receiver.java:185)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:89)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
at java.lang.Thread.run(Thread.java:745)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6735) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream
[ https://issues.apache.org/jira/browse/HDFS-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HDFS-6735: Attachment: HDFS-6735-v3.txt I classified the state in DFSInputStream into state used by read only and state used by both read and pread. With that here's a new proposed patch. * makes LocatedBlocks immutable (which was intended it seems) * pread no longer affects currentNode (that was unintended I think) * guards state shared between read and pread with an extra sharedLock (the state used for read only is still guarded by a lock on this, which we need to take anyway to avoid concurrent stateful reads against the same input stream) * removed all synchronized on private method that were only called from methods already synchronized (good practice anyway) * makes cachingStrategy volatile (made more sense than locking there) * should be free of deadlocks (never acquire lock on this with sharedLock held, but the reverse is possible) * pos, blockEnd, currentLocatedBlock are not updated in getBlockAt unless called on behalf of read (not for pread, hence locking on this not needed there) I have not tested this, yet. Please have a careful look and let me know what you think. We might want to further disentangle the mixed state. (And just maybe the best solution would be for HBase to have an input stream for each thread doing read and one for all threads doing preads - and not do any of this...?) A minor optimization to avoid pread() be blocked by read() inside the same DFSInputStream - Key: HDFS-6735 URL: https://issues.apache.org/jira/browse/HDFS-6735 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-6735-v2.txt, HDFS-6735-v3.txt, HDFS-6735.txt In current DFSInputStream impl, there're a couple of coarser-grained locks in read/pread path, and it has became a HBase read latency pain point so far. In HDFS-6698, i made a minor patch against the first encourtered lock, around getFileLength, in deed, after reading code and testing, it shows still other locks we could improve. In this jira, i'll make a patch against other locks, and a simple test case to show the issue and the improved result. This is important for HBase application, since in current HFile read path, we issue all read()/pread() requests in the same DFSInputStream for one HFile. (Multi streams solution is another story i had a plan to do, but probably will take more time than i expected) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
