[jira] Updated: (HDFS-1080) SecondaryNameNode image transfer should use the defined http address rather than local ip address
[ https://issues.apache.org/jira/browse/HDFS-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1080: -- Status: Resolved (was: Patch Available) Hadoop Flags: [Incompatible change, Reviewed] (was: [Incompatible change]) Fix Version/s: 0.22.0 Resolution: Fixed I've committed this. Resolving as fixed. SecondaryNameNode image transfer should use the defined http address rather than local ip address - Key: HDFS-1080 URL: https://issues.apache.org/jira/browse/HDFS-1080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.22.0 Attachments: HDFS-1080-trunk-1.patch, HDFS-1080-trunk.patch, HDFS-1080-Y20.patch Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1080) SecondaryNameNode image transfer should use the defined http address rather than local ip address
[ https://issues.apache.org/jira/browse/HDFS-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1080: -- Attachment: HDFS-1080-trunk-1.patch Updated patch per Konstantin's comments. Checkpointer needed a bit more work to get the relevant information to the right to the needed function, but should work exactly the same. I'm going to open a JIRA to combine this code duplication until the 2ndarryNN is removed. Konstantin pointed out that this will be an incompatible change in that, were one specifying localhost as the 2nd/CP Nodes' address and relying on that value to be resolved on the node. For now on, one will need to specify the exact address to use, which is much more correct in my opinion, particularly on production systems that may rely on IP aliasing. I'll mark this as incompatible. SecondaryNameNode image transfer should use the defined http address rather than local ip address - Key: HDFS-1080 URL: https://issues.apache.org/jira/browse/HDFS-1080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1080-trunk-1.patch, HDFS-1080-trunk.patch, HDFS-1080-Y20.patch Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1080) SecondaryNameNode image transfer should use the defined http address rather than local ip address
[ https://issues.apache.org/jira/browse/HDFS-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1080: -- Hadoop Flags: [Incompatible change] SecondaryNameNode image transfer should use the defined http address rather than local ip address - Key: HDFS-1080 URL: https://issues.apache.org/jira/browse/HDFS-1080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1080-trunk-1.patch, HDFS-1080-trunk.patch, HDFS-1080-Y20.patch Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1080) SecondaryNameNode image transfer should use the defined http address rather than local ip address
[ https://issues.apache.org/jira/browse/HDFS-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1080: -- Status: Open (was: Patch Available) SecondaryNameNode image transfer should use the defined http address rather than local ip address - Key: HDFS-1080 URL: https://issues.apache.org/jira/browse/HDFS-1080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1080-trunk-1.patch, HDFS-1080-trunk.patch, HDFS-1080-Y20.patch Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1080) SecondaryNameNode image transfer should use the defined http address rather than local ip address
[ https://issues.apache.org/jira/browse/HDFS-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1080: -- Status: Patch Available (was: Open) Re-submitting to Hudson for new patch. SecondaryNameNode image transfer should use the defined http address rather than local ip address - Key: HDFS-1080 URL: https://issues.apache.org/jira/browse/HDFS-1080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1080-trunk-1.patch, HDFS-1080-trunk.patch, HDFS-1080-Y20.patch Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-811) Add metrics, failure reporting and additional tests for HDFS-457
[
https://issues.apache.org/jira/browse/HDFS-811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12876948#action_12876948
]
Jakob Homan commented on HDFS-811:
--
Before this goes in, there are still quite a few undescribed asserts and a fair
amount of unnecessary white space changes that should be removed. Also,
TestDataNodeVolumeFailure2? Oy. It that the sequel? Would a more descriptive
name be reasonable?
Add metrics, failure reporting and additional tests for HDFS-457
Key: HDFS-811
URL: https://issues.apache.org/jira/browse/HDFS-811
Project: Hadoop HDFS
Issue Type: Test
Components: test
Affects Versions: 0.21.0, 0.22.0
Reporter: Ravi Phulari
Assignee: Eli Collins
Priority: Minor
Fix For: 0.21.0, 0.22.0
Attachments: hdfs-811-1.patch, hdfs-811-2.patch, hdfs-811-3.patch,
hdfs-811-4.patch
HDFS-457 introduced a improvement which allows datanode to continue if a
volume for replica storage fails. Previously a datanode resigned if any
volume failed.
Description of HDFS-457
{quote}
Current implementation shuts DataNode down completely when one of the
configured volumes of the storage fails.
This is rather wasteful behavior because it decreases utilization (good
storage becomes unavailable) and imposes extra load on the system
(replication of the blocks from the good volumes). These problems will become
even more prominent when we move to mixed (heterogeneous) clusters with many
more volumes per Data Node.
{quote}
I suggest following additional tests for this improvement.
#1 Test successive volume failures ( Minimum 4 volumes )
#2 Test if each volume failure reports reduction in available DFS space and
remaining space.
#3 Test if failure of all volumes on a data nodes leads to the data node
failure.
#4 Test if correcting failed storage disk brings updates and increments
available DFS space.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1191) SecondaryNameNode give the loopback address (127.0.0.1) to NameNode under some condition
[ https://issues.apache.org/jira/browse/HDFS-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12876195#action_12876195 ] Jakob Homan commented on HDFS-1191: --- This has also been dealt with in HDFS-1080, for which the forward port is coming soon - really - and I believe may be a more appropriate solution, as it keeps more to the defined behavior as specified in the configuration. Do you think we can close this as a duplicate? SecondaryNameNode give the loopback address (127.0.0.1) to NameNode under some condition Key: HDFS-1191 URL: https://issues.apache.org/jira/browse/HDFS-1191 Project: Hadoop HDFS Issue Type: Bug Reporter: Jeff Zhang Attachments: HDFS_1191.patch This issue is because of JDK's bug (although this won't been fixed ) http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4665037 In the method putFSImage of SecondaryNameNode.java, InetAddress.getLocalHost().getHostAddress() will return 127.0.0.1 under some condition which will cause the NameNode can't fetch the merged fsimage from SecondaryNameNode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1080) SecondaryNameNode image transfer should use the defined http address rather than local ip address
[ https://issues.apache.org/jira/browse/HDFS-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1080: -- Attachment: HDFS-1080-trunk.patch Patch for trunk. Exactly the same as the 20 patch, which we've tested manually extensively here. This code is buried quite deep in the guts of the secondary namenode; I think a new unit test isn't feasible here, without a serious, potentially destabilizing refactor. I'd like to go ahead without one. This issue occurs, and prevents the 2ndNN from functioning correctly, when a machine is running on a secure cluster and using IP aliasing to run as a different host than is returned from getLocalHost. The NN will attempt to connect to the local host (say 192.168.0.1) rather than the alias (say secure-2nn), and this will fail the Kerberized SSL authentication and prevent the merged image from being downloaded. SecondaryNameNode image transfer should use the defined http address rather than local ip address - Key: HDFS-1080 URL: https://issues.apache.org/jira/browse/HDFS-1080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1080-trunk.patch, HDFS-1080-Y20.patch Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1080) SecondaryNameNode image transfer should use the defined http address rather than local ip address
[ https://issues.apache.org/jira/browse/HDFS-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1080: -- Status: Patch Available (was: Open) Submitting patch. SecondaryNameNode image transfer should use the defined http address rather than local ip address - Key: HDFS-1080 URL: https://issues.apache.org/jira/browse/HDFS-1080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1080-trunk.patch, HDFS-1080-Y20.patch Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1191) SecondaryNameNode give the loopback address (127.0.0.1) to NameNode under some condition
[ https://issues.apache.org/jira/browse/HDFS-1191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12876375#action_12876375 ] Jakob Homan commented on HDFS-1191: --- By which I meant HDFS-62... doh. SecondaryNameNode give the loopback address (127.0.0.1) to NameNode under some condition Key: HDFS-1191 URL: https://issues.apache.org/jira/browse/HDFS-1191 Project: Hadoop HDFS Issue Type: Bug Reporter: Jeff Zhang Attachments: HDFS_1191.patch This issue is because of JDK's bug (although this won't been fixed ) http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4665037 In the method putFSImage of SecondaryNameNode.java, InetAddress.getLocalHost().getHostAddress() will return 127.0.0.1 under some condition which will cause the NameNode can't fetch the merged fsimage from SecondaryNameNode. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-62) SecondaryNamenode may report incorrect info host name
[
https://issues.apache.org/jira/browse/HDFS-62?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12876377#action_12876377
]
Jakob Homan commented on HDFS-62:
-
Seeing as there hasn't been any movement on this in more than a year, and that
HDFS-1080 also solves this issue, I'd like to close this as won't fix. I think
I prefer the 1080 approach for two reasons: it relies on user-provided
configuration to guarantee the hostname is reported as expected, rather than
how we may be able to decipher, and also it's a smaller change with fewer
moving parts.
Thoughts?
SecondaryNamenode may report incorrect info host name
-
Key: HDFS-62
URL: https://issues.apache.org/jira/browse/HDFS-62
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Carlos Valiente
Assignee: Todd Lipcon
Priority: Minor
Attachments: HADOOP-5626.patch, hadoop-5626.txt
I have set up {{dfs.secondary.http.address}} like this:
{code}
property
namedfs.secondary.http.address/name
valuesecondary.example.com:50090/value
/property
{code}
In my setup {{secondary.example.com}} resolves to an IP address (say,
192.168.0.10) which is not the same as the host's name (as returned by
{{InetAddress.getLocalHost().getHostAddress()}}, say 192.168.0.1).
In this situation, edit log related transfers fail. From the namenode log:
{code}
2009-04-05 13:32:39,128 INFO
org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log from
192.168.0.10
2009-04-05 13:32:39,168 WARN org.mortbay.log: /getimage: java.io.IOException:
GetImage failed. java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:519)
at java.net.Socket.connect(Socket.java:469)
at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:394)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:529)
at sun.net.www.http.HttpClient.init(HttpClient.java:233)
at sun.net.www.http.HttpClient.New(HttpClient.java:306)
at sun.net.www.http.HttpClient.New(HttpClient.java:323)
at
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:837)
at
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:778)
at
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:703)
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1026)
at
org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
...
{code}
From the secondary namenode log:
{code}
2009-04-05 13:42:39,238 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Exception in
doCheckpoint:
2009-04-05 13:42:39,238 ERROR
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode:
java.io.FileNotFoundException:
http://nn.example.com:50070/getimage?putimage=1port=50090machine=
192.168.0.1token=-19:1243068779:0:1238929357000:1238929031783
at
sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1288)
at
org.apache.hadoop.hdfs.server.namenode.TransferFsImage.getFileClient(TransferFsImage.java:151)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.putFSImage(SecondaryNameNode.java:294)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.doCheckpoint(SecondaryNameNode.java:333)
at
org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.run(SecondaryNameNode.java:239)
at java.lang.Thread.run(Thread.java:619)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1185) Remove duplicate now() functions in DataNode, FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1185: -- Hadoop Flags: [Reviewed] Assignee: Jeff Ames +1. Existing tests cover this refactor. Failed contrib test is unrelated. Remove duplicate now() functions in DataNode, FSNamesystem -- Key: HDFS-1185 URL: https://issues.apache.org/jira/browse/HDFS-1185 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, name-node Reporter: Jeff Ames Assignee: Jeff Ames Priority: Minor Attachments: HDFS-1185-2.patch An identical now() function is defined in Util, DataNode, and FSNamesystem. This patch removes the latter two and converts all calls to use the Util.now function. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1185) Remove duplicate now() functions in DataNode, FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1185: -- Status: Resolved (was: Patch Available) Resolution: Fixed I've just committed this. Thanks Jeff! Resolving as fixed. Remove duplicate now() functions in DataNode, FSNamesystem -- Key: HDFS-1185 URL: https://issues.apache.org/jira/browse/HDFS-1185 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, name-node Reporter: Jeff Ames Assignee: Jeff Ames Priority: Minor Attachments: HDFS-1185-2.patch An identical now() function is defined in Util, DataNode, and FSNamesystem. This patch removes the latter two and converts all calls to use the Util.now function. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1183) Remove some duplicate code in NamenodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1183: -- +1. Core failed test is known bad (https://issues.apache.org/jira/browse/HDFS-615), contrib test failure is unrelated, and existing tests cover this refactor. Remove some duplicate code in NamenodeJspHelper.java Key: HDFS-1183 URL: https://issues.apache.org/jira/browse/HDFS-1183 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jeff Ames Assignee: Jeff Ames Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1183.patch This patch refactors out some duplicate code in generateNodeData and generateDecommissioningNodeData in NamenodeJspHelper.java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1183) Remove some duplicate code in NamenodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1183: -- Affects Version/s: 0.20.1 Remove some duplicate code in NamenodeJspHelper.java Key: HDFS-1183 URL: https://issues.apache.org/jira/browse/HDFS-1183 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.20.1 Reporter: Jeff Ames Assignee: Jeff Ames Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1183.patch This patch refactors out some duplicate code in generateNodeData and generateDecommissioningNodeData in NamenodeJspHelper.java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1184) Replace tabs in code with spaces
[ https://issues.apache.org/jira/browse/HDFS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1184: -- Affects Version/s: 0.20.1 Replace tabs in code with spaces Key: HDFS-1184 URL: https://issues.apache.org/jira/browse/HDFS-1184 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.20.1 Reporter: Jeff Ames Assignee: Jeff Ames Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1184.patch Replace some code indentation that uses tabs with spaces, to match the coding convention -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1185) Remove duplicate now() functions in DataNode, FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1185: -- Fix Version/s: 0.22.0 Affects Version/s: 0.20.1 Remove duplicate now() functions in DataNode, FSNamesystem -- Key: HDFS-1185 URL: https://issues.apache.org/jira/browse/HDFS-1185 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, name-node Affects Versions: 0.20.1 Reporter: Jeff Ames Assignee: Jeff Ames Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1185-2.patch An identical now() function is defined in Util, DataNode, and FSNamesystem. This patch removes the latter two and converts all calls to use the Util.now function. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1184) Replace tabs in code with spaces
[ https://issues.apache.org/jira/browse/HDFS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1184: -- Hadoop Flags: [Reviewed] +1. Looks good. I'll commit it in the morning. Replace tabs in code with spaces Key: HDFS-1184 URL: https://issues.apache.org/jira/browse/HDFS-1184 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jeff Priority: Minor Attachments: HDFS-1184.patch Replace some code indentation that uses tabs with spaces, to match the coding convention -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1184) Replace tabs in code with spaces
[ https://issues.apache.org/jira/browse/HDFS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reassigned HDFS-1184: - Assignee: Jeff Replace tabs in code with spaces Key: HDFS-1184 URL: https://issues.apache.org/jira/browse/HDFS-1184 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jeff Assignee: Jeff Priority: Minor Attachments: HDFS-1184.patch Replace some code indentation that uses tabs with spaces, to match the coding convention -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1184) Replace tabs in code with spaces
[ https://issues.apache.org/jira/browse/HDFS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1184: -- Status: Patch Available (was: Open) Replace tabs in code with spaces Key: HDFS-1184 URL: https://issues.apache.org/jira/browse/HDFS-1184 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jeff Assignee: Jeff Priority: Minor Attachments: HDFS-1184.patch Replace some code indentation that uses tabs with spaces, to match the coding convention -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1183) Remove some duplicate code in NamenodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1183: -- Status: Patch Available (was: Open) Remove some duplicate code in NamenodeJspHelper.java Key: HDFS-1183 URL: https://issues.apache.org/jira/browse/HDFS-1183 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jeff Priority: Minor Attachments: HDFS-1183.patch This patch refactors out some duplicate code in generateNodeData and generateDecommissioningNodeData in NamenodeJspHelper.java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1183) Remove some duplicate code in NamenodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1183: -- Hadoop Flags: [Reviewed] Assignee: Jeff Fix Version/s: 0.22.0 +1. Looks good. Will commit in the morning, pending Hudson. Remove some duplicate code in NamenodeJspHelper.java Key: HDFS-1183 URL: https://issues.apache.org/jira/browse/HDFS-1183 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jeff Assignee: Jeff Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1183.patch This patch refactors out some duplicate code in generateNodeData and generateDecommissioningNodeData in NamenodeJspHelper.java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names
[
https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12875240#action_12875240
]
Jakob Homan commented on HDFS-1110:
---
Looking good.
Review:
* If you keep {{NamespaceDedupe}}, which I would recommend as I do think it
adds value in and of itself, it's probably best to move its user-facing bits
with the rest of the offline image viewers. {{OfflineImageViewer.java}} handles
all the command line arguments and such.
* {{NamespaceDedupe.java}}:51 line goes more than 80 characters.
* Nit: {{TestNameDictionary::testNameReuse()}} at first looked to me like a
unit test that hadn't annotated. Maybe verifyNameReuse?
* The static class {{ByteArray}} seems like a candidate either for being a
stand-alone class or wrapped by {{NameDictionary}}; it's not really an integral
part of {{FSDirectory}}.
* The {{NameDictionary.lookup(name, value)}} method seems a bit odd in its
usage. Both times it's used via dictionary.lookup(name, name), which makes me
wonder if this is the right API. Do we expect {{NameDictionary}} to be used
elsewhere such that this abstraction is worth the odd API?
Overall I think this is a good thing to do. The 12 second startup cost compared
to the almost 2 gb savings seems worth it to me. There should be a linear
tradeoff such that small clusters should see essentially no impact and large
clusters pay a very small penalty at startup but have the benefits for their
entire runtime.
A useful improvement later on may be a safemode command to repopulate the
dictionary, which would take into account changes since cluster startup,
particularly newly popular filenames.
Namenode heap optimization - reuse objects for commonly used file names
---
Key: HDFS-1110
URL: https://issues.apache.org/jira/browse/HDFS-1110
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Fix For: 0.22.0
Attachments: hdfs-1110.2.patch, hdfs-1110.3.patch, hdfs-1110.patch
There are a lot of common file names used in HDFS, mainly created by
mapreduce, such as file names starting with part. Reusing byte[]
corresponding to these recurring file names will save significant heap space
used for storing the file names in millions of INodeFile objects.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1185) Remove duplicate now() functions in DataNode, FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12875262#action_12875262 ] Jakob Homan commented on HDFS-1185: --- Konstantin just beat me to the static import suggestion. I've added you to the HDFS contributors so you should be able to use submit patch to trigger Hudson's automated test patch; this also lets reviewers know the patch ready for review. Thanks for the contributions. Remove duplicate now() functions in DataNode, FSNamesystem -- Key: HDFS-1185 URL: https://issues.apache.org/jira/browse/HDFS-1185 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, name-node Reporter: Jeff Priority: Minor Attachments: HDFS-1185.patch An identical now() function is defined in Util, DataNode, and FSNamesystem. This patch removes the latter two and converts all calls to use the Util.now function. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1184) Replace tabs in code with spaces
[ https://issues.apache.org/jira/browse/HDFS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1184: -- Status: Open (was: Patch Available) Replace tabs in code with spaces Key: HDFS-1184 URL: https://issues.apache.org/jira/browse/HDFS-1184 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jeff Assignee: Jeff Priority: Minor Attachments: HDFS-1184.patch Replace some code indentation that uses tabs with spaces, to match the coding convention -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1184) Replace tabs in code with spaces
[ https://issues.apache.org/jira/browse/HDFS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1184: -- Status: Patch Available (was: Open) Re-triggering Hudson. Replace tabs in code with spaces Key: HDFS-1184 URL: https://issues.apache.org/jira/browse/HDFS-1184 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jeff Assignee: Jeff Priority: Minor Attachments: HDFS-1184.patch Replace some code indentation that uses tabs with spaces, to match the coding convention -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1096) allow dfsadmin/mradmin refresh of superuser proxy group mappings
[ https://issues.apache.org/jira/browse/HDFS-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12875301#action_12875301 ] Jakob Homan commented on HDFS-1096: --- +1 on hdfs-109-fix.patch. Fixes the build for me. allow dfsadmin/mradmin refresh of superuser proxy group mappings Key: HDFS-1096 URL: https://issues.apache.org/jira/browse/HDFS-1096 Project: Hadoop HDFS Issue Type: New Feature Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1096-3.patch, HDFS-1096-BP20-4.patch, HDFS-1096-BP20-6-fix.patch, HDFS-1096-BP20-6.patch, HDFS-1096-BP20-7.patch, HDFS-1096-fix.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names
[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12875356#action_12875356 ] Jakob Homan commented on HDFS-1110: --- bq. Change the method name from lookup to put That sounds good. To me, this seems more like a cache (or as Suresh pointed out, interning of Strings), than a dictionary, but the distinction is definitely blurry. bq. Add methods get and remove for completeness This would be extra complexity that wouldn't be called by anyone, correct? I'd hold off on that functionality until it's needed. Namenode heap optimization - reuse objects for commonly used file names --- Key: HDFS-1110 URL: https://issues.apache.org/jira/browse/HDFS-1110 Project: Hadoop HDFS Issue Type: Improvement Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.22.0 Attachments: hdfs-1110.2.patch, hdfs-1110.3.patch, hdfs-1110.patch There are a lot of common file names used in HDFS, mainly created by mapreduce, such as file names starting with part. Reusing byte[] corresponding to these recurring file names will save significant heap space used for storing the file names in millions of INodeFile objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1183) Remove some duplicate code in NamenodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1183: -- Status: Open (was: Patch Available) Remove some duplicate code in NamenodeJspHelper.java Key: HDFS-1183 URL: https://issues.apache.org/jira/browse/HDFS-1183 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jeff Ames Assignee: Jeff Ames Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1183.patch This patch refactors out some duplicate code in generateNodeData and generateDecommissioningNodeData in NamenodeJspHelper.java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1183) Remove some duplicate code in NamenodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1183: -- Status: Patch Available (was: Open) Hudson's back. Re-triggering. Remove some duplicate code in NamenodeJspHelper.java Key: HDFS-1183 URL: https://issues.apache.org/jira/browse/HDFS-1183 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jeff Ames Assignee: Jeff Ames Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1183.patch This patch refactors out some duplicate code in generateNodeData and generateDecommissioningNodeData in NamenodeJspHelper.java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20-BetterJsvcHandling.patch Updating patch, not for commit. We found that manually building the jsvc binary did not work well; there were problems if the build machine is of different architecture (we run 32 bit JVMs for DNs/TTs). New process is to download the file, either directly from Apache, or overridden via the -Djsvc.location param. This greatly simplifies the jsvc process. Also for those interested, in our testing here at Y!, this solution has worked great for securing the datanodes. We've run into no problems with running the datanodes under jsvc. Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: commons-daemon-1.0.2-src.tar.gz, HDFS-1150-BF-Y20-LOG-DIRS-2.patch, HDFS-1150-BF-Y20-LOG-DIRS.patch, HDFS-1150-BF1-Y20.patch, hdfs-1150-bugfix-1.1.patch, hdfs-1150-bugfix-1.2.patch, hdfs-1150-bugfix-1.patch, HDFS-1150-Y20-BetterJsvcHandling.patch, HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1178) The NameNode servlets should not use RPC to connect to the NameNode
[ https://issues.apache.org/jira/browse/HDFS-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12873163#action_12873163 ] Jakob Homan commented on HDFS-1178: --- +1 for direct-nn.patch. The NameNode servlets should not use RPC to connect to the NameNode --- Key: HDFS-1178 URL: https://issues.apache.org/jira/browse/HDFS-1178 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Owen O'Malley Assignee: Owen O'Malley Attachments: direct-nn.patch, hdfs-1178-y20.patch, hdfs-1178-y20.patch Currently some of the NameNode servlets use RPC to connect from the NameNode to itself. They should do it more directly with the NameNode object. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1177) Ivy2.0 has bugs: let's upgrate to 2.1.0
[ https://issues.apache.org/jira/browse/HDFS-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1177: -- Hadoop Flags: [Reviewed] +1, although we may as well delete that previous, commented out version of ivy as well. Common's already been moved to this version; we should open a corresponding JIRA for MR. It's best to have all the kids using the same toys. Ivy2.0 has bugs: let's upgrate to 2.1.0 --- Key: HDFS-1177 URL: https://issues.apache.org/jira/browse/HDFS-1177 Project: Hadoop HDFS Issue Type: Bug Components: build Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Attachments: HADOOP-6792.patch Ivy before 2.1 has bugs in checksums calculation from sha1 files. It might prevent the build from getting some artifacts. Let's upgrade to Ivy 2.1.0 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1027) Update year to 2010.
[
https://issues.apache.org/jira/browse/HDFS-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HDFS-1027:
--
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Fix Version/s: (was: 0.21.0)
(was: 0.20.1)
(was: 0.20.2)
(was: 0.20.3)
Resolution: Fixed
+1. I've committed this. Thanks, Ravi. Resolving as fixed.
Update year to 2010.
-
Key: HDFS-1027
URL: https://issues.apache.org/jira/browse/HDFS-1027
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.20.1, 0.20.2
Reporter: Ravi Phulari
Assignee: Ravi Phulari
Priority: Trivial
Fix For: 0.22.0
Attachments: HDFS-1027.patch
Copyright year needs to be updated from 2009 to 2010.
{code:xml}
!-- The following are used to construct a copyright statement --
year2009/year
vendorThe Apache Software Foundation./vendor
copyright-linkhttp://www.apache.org/licenses//copyright-link
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-992) Re-factor block access token implementation to conform to the generic Token interface in Common
[ https://issues.apache.org/jira/browse/HDFS-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan resolved HDFS-992. -- Fix Version/s: 0.22.0 Resolution: Fixed I've committed this to trunk. Resolving as fixed. Thanks Jitendra and Kan. Re-factor block access token implementation to conform to the generic Token interface in Common --- Key: HDFS-992 URL: https://issues.apache.org/jira/browse/HDFS-992 Project: Hadoop HDFS Issue Type: New Feature Components: security Reporter: Kan Zhang Assignee: Kan Zhang Fix For: 0.22.0 Attachments: h992-12.patch, h992-18.patch, h992-20.patch, h992-21.patch, h992-23.patch, h992-26.patch, h992-27.patch, h992-28.patch, h992-29.patch, h992-BK-0.20-07.1.patch, h992-BK-0.20-07.patch This makes it possible to use block access token as shared key for client-to-datanode authentication over RPC. However, access authorization is still based on block access token semantics. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-992) Re-factor block access token implementation to conform to the generic Token interface in Common
[ https://issues.apache.org/jira/browse/HDFS-992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12872009#action_12872009 ] Jakob Homan commented on HDFS-992: -- +1 for updated patch. Re-factor block access token implementation to conform to the generic Token interface in Common --- Key: HDFS-992 URL: https://issues.apache.org/jira/browse/HDFS-992 Project: Hadoop HDFS Issue Type: New Feature Components: security Reporter: Kan Zhang Assignee: Kan Zhang Fix For: 0.22.0 Attachments: h992-12.patch, h992-18.patch, h992-20.patch, h992-21.patch, h992-23.patch, h992-26.patch, h992-27.patch, h992-28.patch, h992-29.patch, h992-BK-0.20-07.1.patch, h992-BK-0.20-07.patch This makes it possible to use block access token as shared key for client-to-datanode authentication over RPC. However, access authorization is still based on block access token semantics. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-992) Re-factor block access token implementation to conform to the generic Token interface in Common
[ https://issues.apache.org/jira/browse/HDFS-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-992: - Status: Patch Available (was: Open) Triggering Hudson Re-factor block access token implementation to conform to the generic Token interface in Common --- Key: HDFS-992 URL: https://issues.apache.org/jira/browse/HDFS-992 Project: Hadoop HDFS Issue Type: New Feature Components: security Reporter: Kan Zhang Assignee: Kan Zhang Attachments: h992-12.patch, h992-18.patch, h992-20.patch, h992-21.patch, h992-23.patch, h992-26.patch, h992-27.patch, h992-28.patch, h992-BK-0.20-07.1.patch, h992-BK-0.20-07.patch This makes it possible to use block access token as shared key for client-to-datanode authentication over RPC. However, access authorization is still based on block access token semantics. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-992) Re-factor block access token implementation to conform to the generic Token interface in Common
[ https://issues.apache.org/jira/browse/HDFS-992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-992: - Hadoop Flags: [Reviewed] +1 pending test-patch results. This really should have been split into two parts: one the automatic refactoring done by Eclipse and the other the logic refactoring, to make reviewing easier and the history cleaner. Re-factor block access token implementation to conform to the generic Token interface in Common --- Key: HDFS-992 URL: https://issues.apache.org/jira/browse/HDFS-992 Project: Hadoop HDFS Issue Type: New Feature Components: security Reporter: Kan Zhang Assignee: Kan Zhang Attachments: h992-12.patch, h992-18.patch, h992-20.patch, h992-21.patch, h992-23.patch, h992-26.patch, h992-27.patch, h992-28.patch, h992-BK-0.20-07.1.patch, h992-BK-0.20-07.patch This makes it possible to use block access token as shared key for client-to-datanode authentication over RPC. However, access authorization is still based on block access token semantics. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-538) DistributedFileSystem::listStatus incorrectly returns null for empty result sets
[
https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12870092#action_12870092
]
Jakob Homan commented on HDFS-538:
--
@Todd - why the change in release note? The previous version better covered the
change and provided a clue to clients at the change they need to make.
DistributedFileSystem::listStatus incorrectly returns null for empty result
sets
Key: HDFS-538
URL: https://issues.apache.org/jira/browse/HDFS-538
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Jakob Homan
Assignee: Jakob Homan
Fix For: 0.21.0
Attachments: HDFS-538.patch
Currently the listStatus method returns null if no files match the request.
This differs from the Checksum/LocalFileSystem implementation, which returns
an empty array, and the nontvery-explict prescription of the FileSystem
interface: {...@return the statuses of the files/directories in the given
patch}} It's better to return an empty collection than have to add extra
null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-538) DistributedFileSystem::listStatus incorrectly returns null for empty result sets
[
https://issues.apache.org/jira/browse/HDFS-538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HDFS-538:
-
Release Note: FileSystem.listStatus() previously returned null for empty or
nonexistent directories; will now return empty array for empty directories and
throw FileNotFoundException for non-existent directory. Client code should be
updated for new semantics. (was: FileSystem.listStatus() previously returned
null for empty or nonexistent directories. It has been changed to throw
FileNotFoundException if the directory does not exist and to return an empty
array if the directory is empty.)
DistributedFileSystem::listStatus incorrectly returns null for empty result
sets
Key: HDFS-538
URL: https://issues.apache.org/jira/browse/HDFS-538
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Jakob Homan
Assignee: Jakob Homan
Fix For: 0.21.0
Attachments: HDFS-538.patch
Currently the listStatus method returns null if no files match the request.
This differs from the Checksum/LocalFileSystem implementation, which returns
an empty array, and the nontvery-explict prescription of the FileSystem
interface: {...@return the statuses of the files/directories in the given
patch}} It's better to return an empty collection than have to add extra
null checks. The method should return an empty array.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-BF-Y20-LOG-DIRS.patch One more bug fix patch? Sure! Make logs go in the right place... Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: commons-daemon-1.0.2-src.tar.gz, HDFS-1150-BF-Y20-LOG-DIRS.patch, HDFS-1150-BF1-Y20.patch, hdfs-1150-bugfix-1.1.patch, hdfs-1150-bugfix-1.2.patch, hdfs-1150-bugfix-1.patch, HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-BF-Y20-LOG-DIRS-2.patch Our Ops noticed the new jsvc pushes the classpath so long they can no longer monitor the class name, so adding in a fake env variable to identify the started process Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: commons-daemon-1.0.2-src.tar.gz, HDFS-1150-BF-Y20-LOG-DIRS-2.patch, HDFS-1150-BF-Y20-LOG-DIRS.patch, HDFS-1150-BF1-Y20.patch, hdfs-1150-bugfix-1.1.patch, hdfs-1150-bugfix-1.2.patch, hdfs-1150-bugfix-1.patch, HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-BF1-Y20.patch Add some knobs to the secure datanode starter. Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-BF1-Y20.patch, HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1153) The navigation to /dfsnodelist.jsp with invalid input parameters produces NPE and HTTP 500 error
[ https://issues.apache.org/jira/browse/HDFS-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1153: -- Hadoop Flags: [Reviewed] +1 The navigation to /dfsnodelist.jsp with invalid input parameters produces NPE and HTTP 500 error - Key: HDFS-1153 URL: https://issues.apache.org/jira/browse/HDFS-1153 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.1, 0.20.2 Reporter: Ravi Phulari Assignee: Ravi Phulari Fix For: 0.20.3 Attachments: HDFS-1153.patch Navigation to dfsnodelist.jsp with invalid input parameters produces NPE and HTTP 500 error. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20S-ready-6.patch Patch is ready to go, I believe. #6 Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20S-ready-7.patch Final patch based on peer review from Jitendra and Devaraj. Fixed a couple javadoc issues and troublesome jdk reference. Tests look OK, test-patch has two false findbugs warnings from pre-existing errors that were refactored into new methods. Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20S-ready-8.patch Added license header to build.sh Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12867736#action_12867736 ] Jakob Homan commented on HDFS-1150: --- bq. It appears the assumption is that the attacker won't be able to get root privileges. This is indeed an assumption we've had for all the security work. Should one get root, they can get krb keytabs and at that point, game's over. This approach doesn't fix that assumption, but is consistent with it. Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20S-Rough-3.patch Merged with trunk and Jitendra's patch... Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20S-Rough-4.patch How about one that actually compiles? Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20S-ready-5.patch Patch where everything works for review. jsvc is behaving a bit odd - I can't get it to redirect stdout and stderr where they should go... Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866762#action_12866762 ] Jakob Homan commented on HDFS-1150: --- The issue occurs when a client reads or writes blocks to a dn (and along the pipeline during writing). While the datanode is able to use the block access token to verify that the client has permission to do so, the client has no way of verifying that it's talking to a genuine datanode, rather than an imposter process that has come up on the datanode's port (say, after the datanode has crashed). To correct this, rather than change the DataTransferProtocol, and potentially introduce bugs into the write pipeline, we're looking at using the common's jsvc/daemon library to start a secure datanode as root, grab the necessary resources (eg privileged ports) and then drop privileges. This allows clients a reasonable certainty that the datanode they're talking to was started securely and is who they expect them to be. Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20S-Rough.txt First, very rough patch against Y20S branch for secure datanode wrapper that uses jsvc/commons daemon to start up as root and then drop privs. Still need to modify shell scripts to use new class, as well as add to build file to compile jsvc (similar to MR's LinuxTaskController). Having trouble bringing in commons daemon via ivy... Not for commit, etc, etc. Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20S-Rough-2.patch Updated patch with correcty ivy lib... Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1061) Memory footprint optimization for INodeFile object.
[ https://issues.apache.org/jira/browse/HDFS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1061: -- Hadoop Flags: [Incompatible change, Reviewed] Contrib test failure unrelated. +1. Marking as incompatible change due to more strict bounds checking. Memory footprint optimization for INodeFile object. Key: HDFS-1061 URL: https://issues.apache.org/jira/browse/HDFS-1061 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.22.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1061-1.patch, HDFS-1061-2.patch, HDFS-1061-3.patch, HDFS-1061.patch I am proposing a footprint optimization to merge blockReplication and preferredBlockSize fields into one 'long header' field in INodeFile class. This saves 8 bytes per INodeFile object on a 64 bit JVM. This memory optimization is transparent and changes are very minimal. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1061) Memory footprint optimization for INodeFile object.
[ https://issues.apache.org/jira/browse/HDFS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1061: -- Status: Resolved (was: Patch Available) Resolution: Fixed I've just committed this. Thanks, Bharath! Resolving as fixed. Memory footprint optimization for INodeFile object. Key: HDFS-1061 URL: https://issues.apache.org/jira/browse/HDFS-1061 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.22.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1061-1.patch, HDFS-1061-2.patch, HDFS-1061-3.patch, HDFS-1061.patch I am proposing a footprint optimization to merge blockReplication and preferredBlockSize fields into one 'long header' field in INodeFile class. This saves 8 bytes per INodeFile object on a 64 bit JVM. This memory optimization is transparent and changes are very minimal. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1061) Memory footprint optimization for INodeFile object.
[ https://issues.apache.org/jira/browse/HDFS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12865875#action_12865875 ] Jakob Homan commented on HDFS-1061: --- Thanks for the tests Bharath. * For number of replicas does a checking for less than 0 make sense? Should it be = 0. A replica count of 0 seems odd... * For the tests that are expected to throw an exception, it's not necessary to include the always-fails assert. Junit will take care of this. (Also, Junit provides the fail(msg) method, which is equivalent to assertTrue(false)). Other than that looks good. Memory footprint optimization for INodeFile object. Key: HDFS-1061 URL: https://issues.apache.org/jira/browse/HDFS-1061 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.22.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1061-1.patch, HDFS-1061-2.patch, HDFS-1061.patch I am proposing a footprint optimization to merge blockReplication and preferredBlockSize fields into one 'long header' field in INodeFile class. This saves 8 bytes per INodeFile object on a 64 bit JVM. This memory optimization is transparent and changes are very minimal. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1136) FileChecksumServlets.RedirectServlet doesn't carry forward the delegation token
[ https://issues.apache.org/jira/browse/HDFS-1136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1136: -- Hadoop Flags: [Reviewed] Assignee: Boris Shkolnik +1 FileChecksumServlets.RedirectServlet doesn't carry forward the delegation token Key: HDFS-1136 URL: https://issues.apache.org/jira/browse/HDFS-1136 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1136-BP20-1.patch, HDFS-1136-BP20-2.patch FileChecksumServlets.RedirectServlet doesn't carry forward the delegation token in the redircted URL when redirecting to a random data node to get checksum from there -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-999) Secondary namenode should login using kerberos if security is configured
[ https://issues.apache.org/jira/browse/HDFS-999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-999: - Attachment: HDFS-999-BF1-Y20.patch Bug fix to original patch for Y20S. Trunk version soon. Not for commit. Secondary namenode should login using kerberos if security is configured Key: HDFS-999 URL: https://issues.apache.org/jira/browse/HDFS-999 Project: Hadoop HDFS Issue Type: New Feature Reporter: Boris Shkolnik Assignee: Boris Shkolnik Fix For: 0.21.0 Attachments: HDFS-999-BF1-Y20.patch, HDFS-999-BP20.patch, HDFS-999.patch Right now, if NameNode is configured to use Kerberos, SecondaryNameNode will fail to start. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-165) NPE in datanode.handshake()
[ https://issues.apache.org/jira/browse/HDFS-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-165: - Status: Resolved (was: Patch Available) Resolution: Won't Fix NPE in datanode.handshake() --- Key: HDFS-165 URL: https://issues.apache.org/jira/browse/HDFS-165 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.1 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Fix For: 0.22.0 Attachments: HDFS-165.patch It appears possible to raise an NPE in DataNode.handshake() if the startup protocol gets interrupted or fails in some manner -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-320) Namenode should return lease recovery request with other requests
[ https://issues.apache.org/jira/browse/HDFS-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reassigned HDFS-320: Assignee: Kan Zhang Namenode should return lease recovery request with other requests - Key: HDFS-320 URL: https://issues.apache.org/jira/browse/HDFS-320 Project: Hadoop HDFS Issue Type: Improvement Reporter: Kan Zhang Assignee: Kan Zhang Attachments: HADOOP-5481.patch HADOOP-5034 modified NN to return both replication and deletion requests to DN in one reply to a heartbeat. However, the lease recovery request is still sent separately by itself. Is there a reason for this? If not, I suggest we combine them together. This will make it less confusing when adding new types of requests, which are combinable as well. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-801) Add SureLogic annotations' jar into Ivy and Eclipse configs
[ https://issues.apache.org/jira/browse/HDFS-801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-801: - Status: Open (was: Patch Available) Add SureLogic annotations' jar into Ivy and Eclipse configs --- Key: HDFS-801 URL: https://issues.apache.org/jira/browse/HDFS-801 Project: Hadoop HDFS Issue Type: Improvement Components: build, tools Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Edwin Chan Attachments: hdfs_3.1.0.patch, hdfs_3.1.0.patch In order to use SureLogic analysis tools and allow their concurrency analysis annotations in HDFS code the annotations library has to be automatically pulled from a Maven repo. Also, it has to be added to Eclipse .classpath template. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-801) Add SureLogic annotations' jar into Ivy and Eclipse configs
[ https://issues.apache.org/jira/browse/HDFS-801?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-801: - Status: Patch Available (was: Open) Re-submitting to Hudson since it's been a while since the last run, before commit. Add SureLogic annotations' jar into Ivy and Eclipse configs --- Key: HDFS-801 URL: https://issues.apache.org/jira/browse/HDFS-801 Project: Hadoop HDFS Issue Type: Improvement Components: build, tools Affects Versions: 0.22.0 Reporter: Konstantin Boudnik Assignee: Edwin Chan Attachments: hdfs_3.1.0.patch, hdfs_3.1.0.patch In order to use SureLogic analysis tools and allow their concurrency analysis annotations in HDFS code the annotations library has to be automatically pulled from a Maven repo. Also, it has to be added to Eclipse .classpath template. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-708) A stress-test tool for HDFS.
[ https://issues.apache.org/jira/browse/HDFS-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-708: - Status: Open (was: Patch Available) Canceling patch pending review updates. A stress-test tool for HDFS. Key: HDFS-708 URL: https://issues.apache.org/jira/browse/HDFS-708 Project: Hadoop HDFS Issue Type: New Feature Components: test, tools Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Joshua Harlow Fix For: 0.22.0 Attachments: slive.patch, SLiveTest.pdf It would be good to have a tool for automatic stress testing HDFS, which would provide IO-intensive load on HDFS cluster. The idea is to start the tool, let it run overnight, and then be able to analyze possible failures. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-165) NPE in datanode.handshake()
[ https://issues.apache.org/jira/browse/HDFS-165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12862338#action_12862338 ] Jakob Homan commented on HDFS-165: -- bq. I will merge [this patch] into the lifecycle patch rather than split out (as I have done here) Steve, I read this to mean this patch is no longer necessary and can be closed as WontFix? Does this sound good to you? NPE in datanode.handshake() --- Key: HDFS-165 URL: https://issues.apache.org/jira/browse/HDFS-165 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.1 Reporter: Steve Loughran Assignee: Steve Loughran Priority: Minor Fix For: 0.22.0 Attachments: HDFS-165.patch It appears possible to raise an NPE in DataNode.handshake() if the startup protocol gets interrupted or fails in some manner -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-760) fs -put fails if dfs.umask is set to 63
[
https://issues.apache.org/jira/browse/HDFS-760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan resolved HDFS-760.
--
Resolution: Fixed
This was fixed by HADOOP-6521.
fs -put fails if dfs.umask is set to 63
-
Key: HDFS-760
URL: https://issues.apache.org/jira/browse/HDFS-760
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.21.0
Reporter: Tsz Wo (Nicholas), SZE
Fix For: 0.21.0, 0.22.0
Add the following to hdfs-site.conf
{noformat}
property
namedfs.umask/name
value63/value
/property
{noformat}
Then run hadoop fs -put
{noformat}
-bash-3.1$ ./bin/hadoop fs -put README.txt r.txt
09/11/09 23:09:07 WARN conf.Configuration: mapred.task.id is deprecated.
Instead, use mapreduce.task.attempt.id
put: 63
Usage: java FsShell [-put localsrc ... dst]
-bash-3.1$
{noformat}
Observed the above behavior in 0.21.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1110) Namenode heap optimization - reuse objects for commonly used file names
[ https://issues.apache.org/jira/browse/HDFS-1110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12861567#action_12861567 ] Jakob Homan commented on HDFS-1110: --- bq. Build a tool to generate list names that is used more than 10 times from fsimage. Also, we don't actually have to build a separate tool, as the offline image viewer can quickly be extended to provide these numbers and generate the new file. The numbers above were generated from just a few lines in a new viewer. Namenode heap optimization - reuse objects for commonly used file names --- Key: HDFS-1110 URL: https://issues.apache.org/jira/browse/HDFS-1110 Project: Hadoop HDFS Issue Type: Improvement Reporter: Suresh Srinivas Assignee: Suresh Srinivas Fix For: 0.22.0 Attachments: hdfs-1110.patch There are a lot of common file names used in HDFS, mainly created by mapreduce, such as file names starting with part. Reusing byte[] corresponding to these recurring file names will save significant heap space used for storing the file names in millions of INodeFile objects. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-921) Convert TestDFSClientRetries::testNotYetReplicatedErrors to Mockito
[ https://issues.apache.org/jira/browse/HDFS-921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12861050#action_12861050 ] Jakob Homan commented on HDFS-921: -- bq. Actually, I was about to suggest to move this test into src/test/unit because now it looks like a real unit test. Shall I open a separate JIRA for that? What do you think? The src/test/unit directory is vexing. We should definitely have a JIRA for moving those tests that can be truly called unit tests to it; I'm just afraid it'll be a lonely little directory tree at the moment. If we did, TestDFSClientRetries actually probably wouldn't qualify because a new test that got added recently uses MiniDFSCluster and takes more than a minute to run... It's worth a JIRA though. Convert TestDFSClientRetries::testNotYetReplicatedErrors to Mockito --- Key: HDFS-921 URL: https://issues.apache.org/jira/browse/HDFS-921 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Jakob Homan Assignee: Jakob Homan Fix For: 0.22.0 Attachments: HDFS-921-2.patch, HDFS-921.patch When TestDFSClientRetries::testNotYetReplicatedErrors was written, Mockito was not available and the NameNode was mocked by manually extending ClientProtocol and implementing all the methods, most with empty bodies. Now that we have Mockito, this code can be removed and replaced with an actual mock. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-925) Make it harder to accidentally close a shared DFSClient
[
https://issues.apache.org/jira/browse/HDFS-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan reassigned HDFS-925:
Assignee: Steve Loughran
Make it harder to accidentally close a shared DFSClient
---
Key: HDFS-925
URL: https://issues.apache.org/jira/browse/HDFS-925
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs client
Affects Versions: 0.21.0
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor
Fix For: 0.22.0
Attachments: HADOOP-5933.patch, HADOOP-5933.patch, HDFS-925.patch
Every so often I get stack traces telling me that DFSClient is closed,
usually in {{org.apache.hadoop.hdfs.DFSClient.checkOpen() }} . The root cause
of this is usually that one thread has closed a shared fsclient while another
thread still has a reference to it. If the other thread then asks for a new
client it will get one -and the cache repopulated- but if has one already,
then I get to see a stack trace.
It's effectively a race condition between clients in different threads.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1047) Install/deploy source jars to Maven repo
[ https://issues.apache.org/jira/browse/HDFS-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1047: -- Hadoop Flags: [Reviewed] +1. I manually tested by applying all three of the patches, generating the source jar files, verifying their contents and starting a Maven-based project in Eclipse that depended on the local versions and verifying that Eclipse pulled in the source jars as necessary. This will be a big help. Install/deploy source jars to Maven repo Key: HDFS-1047 URL: https://issues.apache.org/jira/browse/HDFS-1047 Project: Hadoop HDFS Issue Type: Improvement Components: build Affects Versions: 0.22.0 Reporter: Patrick Angeles Assignee: Patrick Angeles Fix For: 0.22.0 Attachments: hdfs-1047.patch Making source jars available in the Maven repository enables most IDEs to easily show library code to Hadoop developers. This issue is related to HADOOP-6635 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1047) Install/deploy source jars to Maven repo
[ https://issues.apache.org/jira/browse/HDFS-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1047: -- Status: Resolved (was: Patch Available) Resolution: Fixed I've committed this. Thanks Patrick! Resolving as fixed. Install/deploy source jars to Maven repo Key: HDFS-1047 URL: https://issues.apache.org/jira/browse/HDFS-1047 Project: Hadoop HDFS Issue Type: Improvement Components: build Affects Versions: 0.22.0 Reporter: Patrick Angeles Assignee: Patrick Angeles Fix For: 0.22.0 Attachments: hdfs-1047.patch Making source jars available in the Maven repository enables most IDEs to easily show library code to Hadoop developers. This issue is related to HADOOP-6635 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-666) Unit test for FsShell -text
[ https://issues.apache.org/jira/browse/HDFS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-666: - Status: Open (was: Patch Available) Unit test for FsShell -text --- Key: HDFS-666 URL: https://issues.apache.org/jira/browse/HDFS-666 Project: Hadoop HDFS Issue Type: Test Components: test Reporter: Chris Douglas Assignee: Chris Douglas Priority: Minor Attachments: H666-0.patch, H666-1.patch, H666-2.patch HADOOP-6293 modified FsShell \-text to accept arbitrary paths. The unit test is in TestDFSShell, so it will be updated in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-666) Unit test for FsShell -text
[ https://issues.apache.org/jira/browse/HDFS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-666: - Status: Patch Available (was: Open) Hadoop Flags: [Reviewed] +1. Patch looks good. Re-submitting to Hudson just because there's been such a long delay since its last run. Unit test for FsShell -text --- Key: HDFS-666 URL: https://issues.apache.org/jira/browse/HDFS-666 Project: Hadoop HDFS Issue Type: Test Components: test Reporter: Chris Douglas Assignee: Chris Douglas Priority: Minor Attachments: H666-0.patch, H666-1.patch, H666-2.patch HADOOP-6293 modified FsShell \-text to accept arbitrary paths. The unit test is in TestDFSShell, so it will be updated in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-921) Convert TestDFSClientRetries::testNotYetReplicatedErrors to Mockito
[ https://issues.apache.org/jira/browse/HDFS-921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-921: - Attachment: HDFS-921-2.patch Patch went stale. Updated version. Uses new parameter to addBlock. Otherwise unchanged. Cos: I'm a lazy, lazy man. I let Eclipse handle all my imports... :) Convert TestDFSClientRetries::testNotYetReplicatedErrors to Mockito --- Key: HDFS-921 URL: https://issues.apache.org/jira/browse/HDFS-921 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-921-2.patch, HDFS-921.patch When TestDFSClientRetries::testNotYetReplicatedErrors was written, Mockito was not available and the NameNode was mocked by manually extending ClientProtocol and implementing all the methods, most with empty bodies. Now that we have Mockito, this code can be removed and replaced with an actual mock. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-921) Convert TestDFSClientRetries::testNotYetReplicatedErrors to Mockito
[ https://issues.apache.org/jira/browse/HDFS-921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-921: - Status: Open (was: Patch Available) Convert TestDFSClientRetries::testNotYetReplicatedErrors to Mockito --- Key: HDFS-921 URL: https://issues.apache.org/jira/browse/HDFS-921 Project: Hadoop HDFS Issue Type: Improvement Components: test Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-921-2.patch, HDFS-921.patch When TestDFSClientRetries::testNotYetReplicatedErrors was written, Mockito was not available and the NameNode was mocked by manually extending ClientProtocol and implementing all the methods, most with empty bodies. Now that we have Mockito, this code can be removed and replaced with an actual mock. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-666) Unit test for FsShell -text
[ https://issues.apache.org/jira/browse/HDFS-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-666: - Status: Resolved (was: Patch Available) Resolution: Fixed Contrib test failure is unrelated. I've committed this. Thanks Chris! Resolving as fixed. Unit test for FsShell -text --- Key: HDFS-666 URL: https://issues.apache.org/jira/browse/HDFS-666 Project: Hadoop HDFS Issue Type: Test Components: test Reporter: Chris Douglas Assignee: Chris Douglas Priority: Minor Attachments: H666-0.patch, H666-1.patch, H666-2.patch HADOOP-6293 modified FsShell \-text to accept arbitrary paths. The unit test is in TestDFSShell, so it will be updated in this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1054) Remove unnecessary sleep after failure in nextBlockOutputStream
[ https://issues.apache.org/jira/browse/HDFS-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1054: -- Status: Resolved (was: Patch Available) Resolution: Fixed Thanks for the revision. I can't reproduce the failed test. Contrib failure is unrelated. +1. I've committed this. Thanks Todd. Resolving as fixed. Remove unnecessary sleep after failure in nextBlockOutputStream --- Key: HDFS-1054 URL: https://issues.apache.org/jira/browse/HDFS-1054 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-1054.txt, hdfs-1054.txt If DFSOutputStream fails to create a pipeline, it currently sleeps 6 seconds before retrying. I don't see a great reason to wait at all, much less 6 seconds (especially now that HDFS-630 ensures that a retry won't go back to the bad node). We should at least make it configurable, and perhaps something like backoff makes some sense. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1101) TestDiskError.testLocalDirs() fails
[
https://issues.apache.org/jira/browse/HDFS-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HDFS-1101:
--
Status: Open (was: Patch Available)
This patch no longer cleanly applies after HDFS-909. Also, I'm not getting the
test failure anymore. That patch had two of the changes to MiniDFSCluster that
are included in this patch.
TestDiskError.testLocalDirs() fails
---
Key: HDFS-1101
URL: https://issues.apache.org/jira/browse/HDFS-1101
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.22.0
Attachments: H1101-0.patch, H1101-1.patch, TestDiskErrorLocal.patch
{{TestDiskError.testLocalDirs()}} fails with {{FileNotFoundException}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1092) Use logging rather than System.err in MiniDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1092: -- Summary: Use logging rather than System.err in MiniDFSCluster (was: MiniDFSCluster - System.err usage ) Hadoop Flags: [Reviewed] Assignee: Kay Kay Fix Version/s: (was: 0.21.0) Affects Version/s: 0.22.0 (was: 0.20.2) Priority: Minor (was: Major) +1. I have committed this. Thanks for the contribution, Kay Kay. Resolving as fixed. Use logging rather than System.err in MiniDFSCluster Key: HDFS-1092 URL: https://issues.apache.org/jira/browse/HDFS-1092 Project: Hadoop HDFS Issue Type: Improvement Components: test Affects Versions: 0.22.0 Reporter: Kay Kay Assignee: Kay Kay Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1092.patch, HDFS-1092.patch In waitClusterUp(), (Waiting for the Mini HDFS Cluster to start...) is routed to System.err , diverting the filters in the logger configuration. replace by commons logging for consistency since this output appears out of the blue, in the clients (HBase etc.) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1092) Use logging rather than System.err in MiniDFSCluster
[ https://issues.apache.org/jira/browse/HDFS-1092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1092: -- Status: Resolved (was: Patch Available) Resolution: Fixed Use logging rather than System.err in MiniDFSCluster Key: HDFS-1092 URL: https://issues.apache.org/jira/browse/HDFS-1092 Project: Hadoop HDFS Issue Type: Improvement Components: test Affects Versions: 0.22.0 Reporter: Kay Kay Assignee: Kay Kay Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1092.patch, HDFS-1092.patch In waitClusterUp(), (Waiting for the Mini HDFS Cluster to start...) is routed to System.err , diverting the filters in the logger configuration. replace by commons logging for consistency since this output appears out of the blue, in the clients (HBase etc.) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1061) Memory footprint optimization for INodeFile object.
[ https://issues.apache.org/jira/browse/HDFS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12860584#action_12860584 ] Jakob Homan commented on HDFS-1061: --- The patch looks good, but Hudson is right - we should have some basic sanity unit tests here, if for no other reason, than to catch any mistakes that may be introduced later that don't jive with the assumptions made in the patch. The new logic in INodeFile is easily testable. I'll +1 and commit once some basic unit tests are included. Memory footprint optimization for INodeFile object. Key: HDFS-1061 URL: https://issues.apache.org/jira/browse/HDFS-1061 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.22.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1061.patch I am proposing a footprint optimization to merge blockReplication and preferredBlockSize fields into one 'long header' field in INodeFile class. This saves 8 bytes per INodeFile object on a 64 bit JVM. This memory optimization is transparent and changes are very minimal. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1061) Memory footprint optimization for INodeFile object.
[ https://issues.apache.org/jira/browse/HDFS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1061: -- Status: Open (was: Patch Available) Canceling patch pending new unit test. Memory footprint optimization for INodeFile object. Key: HDFS-1061 URL: https://issues.apache.org/jira/browse/HDFS-1061 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.22.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1061.patch I am proposing a footprint optimization to merge blockReplication and preferredBlockSize fields into one 'long header' field in INodeFile class. This saves 8 bytes per INodeFile object on a 64 bit JVM. This memory optimization is transparent and changes are very minimal. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1101) TestDiskError.testLocalDirs() fails
[
https://issues.apache.org/jira/browse/HDFS-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12860588#action_12860588
]
Jakob Homan commented on HDFS-1101:
---
+1 on H1101-0.patch
TestDiskError.testLocalDirs() fails
---
Key: HDFS-1101
URL: https://issues.apache.org/jira/browse/HDFS-1101
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.22.0
Attachments: H1101-0.patch, TestDiskErrorLocal.patch
{{TestDiskError.testLocalDirs()}} fails with {{FileNotFoundException}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1101) TestDiskError.testLocalDirs() fails
[
https://issues.apache.org/jira/browse/HDFS-1101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jakob Homan updated HDFS-1101:
--
Status: Resolved (was: Patch Available)
Hadoop Flags: [Reviewed]
Assignee: Chris Douglas (was: Konstantin Shvachko)
Resolution: Fixed
I've committed this. Thanks Chris and Konstantin! Resolving as fixed.
TestDiskError.testLocalDirs() fails
---
Key: HDFS-1101
URL: https://issues.apache.org/jira/browse/HDFS-1101
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Assignee: Chris Douglas
Fix For: 0.22.0
Attachments: H1101-0.patch, TestDiskErrorLocal.patch
{{TestDiskError.testLocalDirs()}} fails with {{FileNotFoundException}}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1047) Install/deploy source jars to Maven repo
[ https://issues.apache.org/jira/browse/HDFS-1047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reassigned HDFS-1047: - Assignee: Patrick Angeles Install/deploy source jars to Maven repo Key: HDFS-1047 URL: https://issues.apache.org/jira/browse/HDFS-1047 Project: Hadoop HDFS Issue Type: Improvement Components: build Affects Versions: 0.22.0 Reporter: Patrick Angeles Assignee: Patrick Angeles Fix For: 0.22.0 Attachments: hdfs-1047.patch Making source jars available in the Maven repository enables most IDEs to easily show library code to Hadoop developers. This issue is related to HADOOP-6635 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1054) Remove unnecessary sleep after failure in nextBlockOutputStream
[ https://issues.apache.org/jira/browse/HDFS-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12860625#action_12860625 ] Jakob Homan commented on HDFS-1054: --- Patch looks good except for unnecessary changes to log output (combining two log entries into one). We've not nailed down the log output format yet, but there's still no need to change it without need, in case someone is currently parsing its output. Remove unnecessary sleep after failure in nextBlockOutputStream --- Key: HDFS-1054 URL: https://issues.apache.org/jira/browse/HDFS-1054 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-1054.txt If DFSOutputStream fails to create a pipeline, it currently sleeps 6 seconds before retrying. I don't see a great reason to wait at all, much less 6 seconds (especially now that HDFS-630 ensures that a retry won't go back to the bad node). We should at least make it configurable, and perhaps something like backoff makes some sense. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1054) Remove unnecessary sleep after failure in nextBlockOutputStream
[ https://issues.apache.org/jira/browse/HDFS-1054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12860628#action_12860628 ] Jakob Homan commented on HDFS-1054: --- Changing to INFO should work fine. Remove unnecessary sleep after failure in nextBlockOutputStream --- Key: HDFS-1054 URL: https://issues.apache.org/jira/browse/HDFS-1054 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Attachments: hdfs-1054.txt If DFSOutputStream fails to create a pipeline, it currently sleeps 6 seconds before retrying. I don't see a great reason to wait at all, much less 6 seconds (especially now that HDFS-630 ensures that a retry won't go back to the bad node). We should at least make it configurable, and perhaps something like backoff makes some sense. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1014) Error in reading delegation tokens from edit logs.
[ https://issues.apache.org/jira/browse/HDFS-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1014: -- Status: Resolved (was: Patch Available) Fix Version/s: 0.22.0 Resolution: Fixed I've just committed this. Resolving as fixed. Thanks for the contribution, Jitendra. Error in reading delegation tokens from edit logs. -- Key: HDFS-1014 URL: https://issues.apache.org/jira/browse/HDFS-1014 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.22.0 Attachments: HDFS-1014-y20.1.patch, HDFS-1014.2.patch, HDFS-1014.3.patch When delegation tokens are read from the edit logs...same object is used to read the identifier and is stored in the token cache. This is wrong because same object is getting updated. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1014) Error in reading delegation tokens from edit logs.
[ https://issues.apache.org/jira/browse/HDFS-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12858646#action_12858646 ] Jakob Homan commented on HDFS-1014: --- +1 on trunk patch (3). Error in reading delegation tokens from edit logs. -- Key: HDFS-1014 URL: https://issues.apache.org/jira/browse/HDFS-1014 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1014-y20.1.patch, HDFS-1014.2.patch, HDFS-1014.3.patch When delegation tokens are read from the edit logs...same object is used to read the identifier and is stored in the token cache. This is wrong because same object is getting updated. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1080) SecondaryNameNode image transfer should use the defined http address rather than local ip address
SecondaryNameNode image transfer should use the defined http address rather than local ip address - Key: HDFS-1080 URL: https://issues.apache.org/jira/browse/HDFS-1080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Jakob Homan Assignee: Jakob Homan Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1080) SecondaryNameNode image transfer should use the defined http address rather than local ip address
[ https://issues.apache.org/jira/browse/HDFS-1080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1080: -- Attachment: HDFS-1080-Y20.patch Patch to use defined addr, for Y!20. Trunk patch soon... SecondaryNameNode image transfer should use the defined http address rather than local ip address - Key: HDFS-1080 URL: https://issues.apache.org/jira/browse/HDFS-1080 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1080-Y20.patch Currently when telling the NN where to get the merged image, SNN uses getLocalHost.getAddr(), which may not return the correct ip addr. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1081) Performance regression in DistributedFileSystem::getFileBlockLocations in secure systems
Performance regression in DistributedFileSystem::getFileBlockLocations in secure systems Key: HDFS-1081 URL: https://issues.apache.org/jira/browse/HDFS-1081 Project: Hadoop HDFS Issue Type: Improvement Components: security Reporter: Jakob Homan Assignee: Jakob Homan We've seen a significant decrease in the performance of DistributedFileSystem::getFileBlockLocations() with security turned on Y20. This JIRA is for correcting and tracking it both on Y20 and trunk. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel to generate block report
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-854: - Hadoop Flags: [Reviewed] +1. TestDataNodeScanner passes with the patch on my box - Hudson's acting up again. Other test failure is unrelated. Unless there are more comments, I'll commit this this evening. Datanode should scan devices in parallel to generate block report - Key: HDFS-854 URL: https://issues.apache.org/jira/browse/HDFS-854 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.22.0 Reporter: dhruba borthakur Assignee: Dmytro Molkov Fix For: 0.22.0 Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1 A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster. A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-854) Datanode should scan devices in parallel to generate block report
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-854: - Resolution: Fixed Status: Resolved (was: Patch Available) I have committed this. Resolving as fixed. Thanks for the contribution, Dmytro! Datanode should scan devices in parallel to generate block report - Key: HDFS-854 URL: https://issues.apache.org/jira/browse/HDFS-854 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.22.0 Reporter: dhruba borthakur Assignee: Dmytro Molkov Fix For: 0.22.0 Attachments: HDFS-854-2.patch, HDFS-854.patch, HDFS-854.patch.1 A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster. A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-854) Datanode should scan devices in parallel to generate block report
[ https://issues.apache.org/jira/browse/HDFS-854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12849476#action_12849476 ] Jakob Homan commented on HDFS-854: -- The test failure is not related in all likelihood. The patch overall looks good. One change: dfs.datanode.directoryscan.threads should be defined and referenced via DFSConfigKeys.java. One question: Is it appropriate to just log the error when compiling block reports (line 125 in patch), or should more drastic action be taken? One request: Naming patches with a final number really confuses all my editors, how about HDFS-854-1.patch (although numbering patches is greatly appreciated!) Datanode should scan devices in parallel to generate block report - Key: HDFS-854 URL: https://issues.apache.org/jira/browse/HDFS-854 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.22.0 Reporter: dhruba borthakur Assignee: Dmytro Molkov Fix For: 0.22.0 Attachments: HDFS-854.patch, HDFS-854.patch.1 A Datanode should scan its disk devices in parallel so that the time to generate a block report is reduced. This will reduce the startup time of a cluster. A datanode has 12 disk (each of 1 TB) to store HDFS blocks. There is a total of 150K blocks on these 12 disks. It takes the datanode upto 20 minutes to scan these devices to generate the first block report. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1045) In secure clusters, re-login is necessary for https clients before opening connections
[ https://issues.apache.org/jira/browse/HDFS-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan reassigned HDFS-1045: - Assignee: Jakob Homan In secure clusters, re-login is necessary for https clients before opening connections -- Key: HDFS-1045 URL: https://issues.apache.org/jira/browse/HDFS-1045 Project: Hadoop HDFS Issue Type: Bug Components: security Reporter: Jakob Homan Assignee: Jakob Homan Attachments: HDFS-1045-Y20.patch Ticket credentials expire and therefore clients opening https connections (only the NN and SNN doing image/edits exchange) should re-login before opening those connections. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1045) In secure clusters, re-login is necessary for https clients before opening connections
In secure clusters, re-login is necessary for https clients before opening connections -- Key: HDFS-1045 URL: https://issues.apache.org/jira/browse/HDFS-1045 Project: Hadoop HDFS Issue Type: Bug Components: security Reporter: Jakob Homan Ticket credentials expire and therefore clients opening https connections (only the NN and SNN doing image/edits exchange) should re-login before opening those connections. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1045) In secure clusters, re-login is necessary for https clients before opening connections
[ https://issues.apache.org/jira/browse/HDFS-1045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1045: -- Attachment: HDFS-1045-Y20.patch Patch for Y20 branch. Trunk version soon... In secure clusters, re-login is necessary for https clients before opening connections -- Key: HDFS-1045 URL: https://issues.apache.org/jira/browse/HDFS-1045 Project: Hadoop HDFS Issue Type: Bug Components: security Reporter: Jakob Homan Attachments: HDFS-1045-Y20.patch Ticket credentials expire and therefore clients opening https connections (only the NN and SNN doing image/edits exchange) should re-login before opening those connections. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1036) in DelegationTokenFetch dfs.getURI returns no port
[ https://issues.apache.org/jira/browse/HDFS-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1036: -- +1 in DelegationTokenFetch dfs.getURI returns no port -- Key: HDFS-1036 URL: https://issues.apache.org/jira/browse/HDFS-1036 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Attachments: HDFS-1036-BP20.patch dfs.getUri().getPort() returns -1. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
