[jira] [Commented] (HDFS-1964) Incorrect HTML unescaping in DatanodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13038974#comment-13038974 ] Hudson commented on HDFS-1964: -- Integrated in Hadoop-Hdfs-trunk-Commit #683 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/683/]) HDFS-1964. Fix incorrect HTML unescaping in DatanodeJspHelper. Contributed by Aaron T. Myers. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1127390 Files : * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/DatanodeJspHelper.java * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDatanodeJsp.java * /hadoop/hdfs/trunk/CHANGES.txt Incorrect HTML unescaping in DatanodeJspHelper.java --- Key: HDFS-1964 URL: https://issues.apache.org/jira/browse/HDFS-1964 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0, 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.22.0, 0.23.0 Attachments: hdfs-1964-trunk.0.patch, hdfs-1964-trunk.1.patch HDFS-1575 introduced some HTML unescaping of parameters so that viewing a file would work for paths containing HTML-escaped characters, but in two of the places did the unescaping either too early or too late. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1623) High Availability Framework for HDFS NN
[ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-1623: --- Attachment: NameNode HA_v2.pdf NameNode HA_v2.pdf Attached version 2 of Namenode HA document. Added a significant amount of design details. High Availability Framework for HDFS NN --- Key: HDFS-1623 URL: https://issues.apache.org/jira/browse/HDFS-1623 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: HDFS-High-Availability.pdf, NameNode HA_v2.pdf, Namenode HA Framework.pdf -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1623) High Availability Framework for HDFS NN
[ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-1623: --- Attachment: (was: NameNode HA_v2.pdf) High Availability Framework for HDFS NN --- Key: HDFS-1623 URL: https://issues.apache.org/jira/browse/HDFS-1623 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: HDFS-High-Availability.pdf, NameNode HA_v2.pdf, Namenode HA Framework.pdf -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1623) High Availability Framework for HDFS NN
[ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039019#comment-13039019 ] Sanjay Radia commented on HDFS-1623: How does NN fail-over impact federation? Eg does viewfs have any special requirements as a client? In case of federation, each NN needs its own warm or hot failover. Cannot do N+k failover because of the large memory requirements for a NN unless one wants to limit it to cold failover. Viewfs is separate layer and not required for federation; one could choose to use symbolic links or one could choose to not provide a global namespace. But if one were to use viewfs, I am not aware of any *additional* failover issues. Viewfs has bindings like: /user -hdfs://nnOfUser/ The failover issues are the same as if hdfs://nnofUser was your default file system. High Availability Framework for HDFS NN --- Key: HDFS-1623 URL: https://issues.apache.org/jira/browse/HDFS-1623 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: HDFS-High-Availability.pdf, NameNode HA_v2.pdf, Namenode HA Framework.pdf -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1623) High Availability Framework for HDFS NN
[ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039022#comment-13039022 ] Sanjay Radia commented on HDFS-1623: Wrt requirement #2, ... fail over to new version of HDFS ... is out of scope for this framework. For failover during upgrades, the DNs need to be able to communicate to both versions of the NN. This is true for dot releases and will only be true if we support rolling upgrades. I will clarify this on the next version of the document. High Availability Framework for HDFS NN --- Key: HDFS-1623 URL: https://issues.apache.org/jira/browse/HDFS-1623 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: HDFS-High-Availability.pdf, NameNode HA_v2.pdf, Namenode HA Framework.pdf -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1941) Remove -genclusterid from NameNode startup options
[ https://issues.apache.org/jira/browse/HDFS-1941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039090#comment-13039090 ] Hudson commented on HDFS-1941: -- Integrated in Hadoop-Hdfs-trunk #677 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/677/]) Reverting the change r1125031 - HDFS-1941. Remove -genclusterid option from namenode command. suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1127311 Files : * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/common/HdfsConstants.java * /hadoop/hdfs/trunk/CHANGES.txt Remove -genclusterid from NameNode startup options -- Key: HDFS-1941 URL: https://issues.apache.org/jira/browse/HDFS-1941 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.23.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Priority: Minor Fix For: 0.23.0 Attachments: HDFS-1941-1.patch Currently, namenode -genclusterid is a helper utility to generate unique clusterid. This option is useless once namenode -format automatically generates the clusterid. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1964) Incorrect HTML unescaping in DatanodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039091#comment-13039091 ] Hudson commented on HDFS-1964: -- Integrated in Hadoop-Hdfs-trunk #677 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk/677/]) HDFS-1964. Fix incorrect HTML unescaping in DatanodeJspHelper. Contributed by Aaron T. Myers. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1127390 Files : * /hadoop/hdfs/trunk/src/java/org/apache/hadoop/hdfs/server/datanode/DatanodeJspHelper.java * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/server/datanode/TestDatanodeJsp.java * /hadoop/hdfs/trunk/CHANGES.txt Incorrect HTML unescaping in DatanodeJspHelper.java --- Key: HDFS-1964 URL: https://issues.apache.org/jira/browse/HDFS-1964 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0, 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.22.0, 0.23.0 Attachments: hdfs-1964-trunk.0.patch, hdfs-1964-trunk.1.patch HDFS-1575 introduced some HTML unescaping of parameters so that viewing a file would work for paths containing HTML-escaped characters, but in two of the places did the unescaping either too early or too late. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1169) Can't read binary data off HDFS via thrift API
[ https://issues.apache.org/jira/browse/HDFS-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johnny Boy updated HDFS-1169: - Attachment: thriftfs.jar HadoopThriftServer with changed encoding for reading and writing Can't read binary data off HDFS via thrift API -- Key: HDFS-1169 URL: https://issues.apache.org/jira/browse/HDFS-1169 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.2 Reporter: Erik Forsberg Attachments: HadoopThriftServer.java, hadoopfs.thrift, thriftfs.jar Trying to access binary data stored in HDFS (in my case, TypedByte files generated by Dumbo) via thrift talking to org.apache.hadoop.thriftfs.HadoopThriftServer, the data I get back is mangled. For example, when I read a file which contains the value 0xa2, it's coming back as 0xef 0xbf 0xbd, also known as the Unicode replacement character. I think this is because the read method in HadoopThriftServer.java is trying to convert the data read from HDFS into UTF-8 via the String() constructor. This essentially makes the HDFS thrift API useless for me :-(. Not being an expert on Thrift, but would it be possible to modify the API so that it uses the binary type listed on http://wiki.apache.org/thrift/ThriftTypes? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1169) Can't read binary data off HDFS via thrift API
[ https://issues.apache.org/jira/browse/HDFS-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Johnny Boy updated HDFS-1169: - Attachment: thriftfs.jar Modified version of HadoopThriftServer Can't read binary data off HDFS via thrift API -- Key: HDFS-1169 URL: https://issues.apache.org/jira/browse/HDFS-1169 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.2 Reporter: Erik Forsberg Attachments: HadoopThriftServer.java, hadoopfs.thrift, thriftfs.jar, thriftfs.jar Trying to access binary data stored in HDFS (in my case, TypedByte files generated by Dumbo) via thrift talking to org.apache.hadoop.thriftfs.HadoopThriftServer, the data I get back is mangled. For example, when I read a file which contains the value 0xa2, it's coming back as 0xef 0xbf 0xbd, also known as the Unicode replacement character. I think this is because the read method in HadoopThriftServer.java is trying to convert the data read from HDFS into UTF-8 via the String() constructor. This essentially makes the HDFS thrift API useless for me :-(. Not being an expert on Thrift, but would it be possible to modify the API so that it uses the binary type listed on http://wiki.apache.org/thrift/ThriftTypes? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1169) Can't read binary data off HDFS via thrift API
[ https://issues.apache.org/jira/browse/HDFS-1169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039124#comment-13039124 ] Johnny Boy commented on HDFS-1169: -- Bleh double post and I don't know how to remove it :) The comment should be that it was enough to change utf-8 into latin1 for me to get the binary data to work, see attached file above Can't read binary data off HDFS via thrift API -- Key: HDFS-1169 URL: https://issues.apache.org/jira/browse/HDFS-1169 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.2 Reporter: Erik Forsberg Attachments: HadoopThriftServer.java, hadoopfs.thrift, thriftfs.jar, thriftfs.jar Trying to access binary data stored in HDFS (in my case, TypedByte files generated by Dumbo) via thrift talking to org.apache.hadoop.thriftfs.HadoopThriftServer, the data I get back is mangled. For example, when I read a file which contains the value 0xa2, it's coming back as 0xef 0xbf 0xbd, also known as the Unicode replacement character. I think this is because the read method in HadoopThriftServer.java is trying to convert the data read from HDFS into UTF-8 via the String() constructor. This essentially makes the HDFS thrift API useless for me :-(. Not being an expert on Thrift, but would it be possible to modify the API so that it uses the binary type listed on http://wiki.apache.org/thrift/ThriftTypes? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-977) DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
[
https://issues.apache.org/jira/browse/HDFS-977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J Chouraria updated HDFS-977:
---
Fix Version/s: 0.23.0
Release Note: DataNode.createInterDataNodeProtocolProxy() guarded a log at
the wrong level
Status: Patch Available (was: Open)
DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
---
Key: HDFS-977
URL: https://issues.apache.org/jira/browse/HDFS-977
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Harsh J Chouraria
Priority: Trivial
Fix For: 0.23.0
Attachments: HDFS-977.r1.diff
My IDE tells me that this code snippet in
{{DataNode.createInterDataNodeProtocolProxy()}} is guarding the info log
entry at debug level, and it should be reviewed
{code}
if (InterDatanodeProtocol.LOG.isDebugEnabled()) {
InterDatanodeProtocol.LOG.info(InterDatanodeProtocol addr= + addr);
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-977) DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
[
https://issues.apache.org/jira/browse/HDFS-977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039132#comment-13039132
]
Hadoop QA commented on HDFS-977:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12478501/HDFS-977.r1.diff
against trunk revision 1127390.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/622//console
This message is automatically generated.
DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
---
Key: HDFS-977
URL: https://issues.apache.org/jira/browse/HDFS-977
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Harsh J Chouraria
Priority: Trivial
Fix For: 0.23.0
Attachments: HDFS-977.r1.diff
My IDE tells me that this code snippet in
{{DataNode.createInterDataNodeProtocolProxy()}} is guarding the info log
entry at debug level, and it should be reviewed
{code}
if (InterDatanodeProtocol.LOG.isDebugEnabled()) {
InterDatanodeProtocol.LOG.info(InterDatanodeProtocol addr= + addr);
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1636) If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation
[ https://issues.apache.org/jira/browse/HDFS-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J Chouraria updated HDFS-1636: Attachment: HDFS-1636.r2.diff Added sort of a test case for this. If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation -- Key: HDFS-1636 URL: https://issues.apache.org/jira/browse/HDFS-1636 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Chouraria Priority: Minor Fix For: 0.23.0 Attachments: HDFS-1636.r1.diff, HDFS-1636.r2.diff Right now, running namenode -format when dfs.name.dir is configured to a dir which exists but is empty still asks for confirmation. This is unnecessary since it isn't blowing away any real data. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1636) If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation
[ https://issues.apache.org/jira/browse/HDFS-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J Chouraria updated HDFS-1636: Fix Version/s: 0.23.0 Release Note: If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation. Status: Patch Available (was: Open) If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation -- Key: HDFS-1636 URL: https://issues.apache.org/jira/browse/HDFS-1636 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Chouraria Priority: Minor Fix For: 0.23.0 Attachments: HDFS-1636.r1.diff, HDFS-1636.r2.diff Right now, running namenode -format when dfs.name.dir is configured to a dir which exists but is empty still asks for confirmation. This is unnecessary since it isn't blowing away any real data. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1620) FSConstants vs FsConstants
[
https://issues.apache.org/jira/browse/HDFS-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039145#comment-13039145
]
Harsh J Chouraria commented on HDFS-1620:
-
The failing test:
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery.testBlockPoolStorageStates
is unrelated to this patch (age 93 as per hudson).
FSConstants vs FsConstants
---
Key: HDFS-1620
URL: https://issues.apache.org/jira/browse/HDFS-1620
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 0.22.0
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Harsh J Chouraria
Priority: Minor
Fix For: 0.23.0
Attachments: HDFS-1620.r1.diff
We have {{org.apache.hadoop.hdfs.protocol.*FSConstants*}} and
{{org.apache.hadoop.fs.*FsConstants*}}. Elegant or confused?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1640) Transmission of large files in compressed format will save the network traffic and can improve the data transfer time.
[ https://issues.apache.org/jira/browse/HDFS-1640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039159#comment-13039159 ] Harsh J Chouraria commented on HDFS-1640: - @Option 2 -- This can be done at the writer level itself. Much too expensive maintaining a per-file property for compression and implementing the same to be guaranteed as well. Besides, compressing on the fly for tx would still invoke the same amount of IO. No savings, right? Transmission of large files in compressed format will save the network traffic and can improve the data transfer time. -- Key: HDFS-1640 URL: https://issues.apache.org/jira/browse/HDFS-1640 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, hdfs client Reporter: Uma Maheswara Rao G *In Write scenario:* DFSClient can Compress the data when transmitting it over the network, Data Nodes can forward the same compressed data to other Data Nodes in pipeline and as well as it can decompress that data and write on to their local disks. *In Read Scenario:* Data Node can compress the data when transmitting it over the network. DFSClient can decompress it and write on to the local stream. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1636) If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation
[ https://issues.apache.org/jira/browse/HDFS-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039160#comment-13039160 ] Hadoop QA commented on HDFS-1636: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480416/HDFS-1636.r2.diff against trunk revision 1127390. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSStorageStateRecovery +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/623//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/623//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/623//console This message is automatically generated. If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation -- Key: HDFS-1636 URL: https://issues.apache.org/jira/browse/HDFS-1636 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Chouraria Priority: Minor Fix For: 0.23.0 Attachments: HDFS-1636.r1.diff, HDFS-1636.r2.diff Right now, running namenode -format when dfs.name.dir is configured to a dir which exists but is empty still asks for confirmation. This is unnecessary since it isn't blowing away any real data. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1636) If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation
[ https://issues.apache.org/jira/browse/HDFS-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039178#comment-13039178 ] Harsh J Chouraria commented on HDFS-1636: - Failing test looks unrelated. If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation -- Key: HDFS-1636 URL: https://issues.apache.org/jira/browse/HDFS-1636 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Chouraria Priority: Minor Fix For: 0.23.0 Attachments: HDFS-1636.r1.diff, HDFS-1636.r2.diff Right now, running namenode -format when dfs.name.dir is configured to a dir which exists but is empty still asks for confirmation. This is unnecessary since it isn't blowing away any real data. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1636) If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation
[ https://issues.apache.org/jira/browse/HDFS-1636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039195#comment-13039195 ] Todd Lipcon commented on HDFS-1636: --- Hey Harsh. There's now an issue where the NN will throw an NPE if one of the directories exists but isn't accessible. eg I did chmod 000 /my/namedir and run namenode -format and got: 11/05/25 09:43:32 ERROR namenode.NameNode: java.lang.NullPointerException at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1487) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1671) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1730) Consider looking at HADOOP-7322 for a utility method on its way into trunk to deal with this. If dfs.name.dir points to an empty dir, namenode format shouldn't require confirmation -- Key: HDFS-1636 URL: https://issues.apache.org/jira/browse/HDFS-1636 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.23.0 Reporter: Todd Lipcon Assignee: Harsh J Chouraria Priority: Minor Fix For: 0.23.0 Attachments: HDFS-1636.r1.diff, HDFS-1636.r2.diff Right now, running namenode -format when dfs.name.dir is configured to a dir which exists but is empty still asks for confirmation. This is unnecessary since it isn't blowing away any real data. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-977) DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
[
https://issues.apache.org/jira/browse/HDFS-977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039198#comment-13039198
]
Todd Lipcon commented on HDFS-977:
--
Can you re-upload as a -p0 patch?
DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
---
Key: HDFS-977
URL: https://issues.apache.org/jira/browse/HDFS-977
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Harsh J Chouraria
Priority: Trivial
Fix For: 0.23.0
Attachments: HDFS-977.r1.diff
My IDE tells me that this code snippet in
{{DataNode.createInterDataNodeProtocolProxy()}} is guarding the info log
entry at debug level, and it should be reviewed
{code}
if (InterDatanodeProtocol.LOG.isDebugEnabled()) {
InterDatanodeProtocol.LOG.info(InterDatanodeProtocol addr= + addr);
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-977) DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
[
https://issues.apache.org/jira/browse/HDFS-977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Harsh J Chouraria updated HDFS-977:
---
Attachment: HDFS-977.r2.diff
Sorry about that!
DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
---
Key: HDFS-977
URL: https://issues.apache.org/jira/browse/HDFS-977
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Harsh J Chouraria
Priority: Trivial
Fix For: 0.23.0
Attachments: HDFS-977.r1.diff, HDFS-977.r2.diff
My IDE tells me that this code snippet in
{{DataNode.createInterDataNodeProtocolProxy()}} is guarding the info log
entry at debug level, and it should be reviewed
{code}
if (InterDatanodeProtocol.LOG.isDebugEnabled()) {
InterDatanodeProtocol.LOG.info(InterDatanodeProtocol addr= + addr);
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1727) fsck command can display command usage if user passes any illegal argument
[
https://issues.apache.org/jira/browse/HDFS-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039208#comment-13039208
]
Todd Lipcon commented on HDFS-1727:
---
A few comments:
- I think the {{null == dir}} check needs an {{else}} clause that prints an
error that only one path may be passed. That is to say, if you run hadoop fsck
/foo /bar, it should print an error rather than ignoring /bar.
- To make the tests run faster, I think it would be better for the two test
cases to be combined and share the same MiniDFSCluster.
- The test case would be easier to follow if you used an argument like
-asdfjisadfijs or -thisIsNotAValidFlag instead of -listcorruptfileblocks.
- The error message could be formatted nicer. Instead of Invalid arguments
passed 'foo' I would use the same formatting as the FsShell errors - ie fsck:
Illegal option 'foo'
fsck command can display command usage if user passes any illegal argument
--
Key: HDFS-1727
URL: https://issues.apache.org/jira/browse/HDFS-1727
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.20.1, 0.23.0
Reporter: Uma Maheswara Rao G
Priority: Minor
Attachments: HDFS-1727.patch
In fsck command if user passes the arguments like
./hadoop fsck -test -files -blocks -racks
In this case it will take / and will display whole DFS information regarding
to files,blocks,racks.
But here, we are hiding the user mistake. Instead of this, we can display the
command usage if user passes any invalid argument like above.
If user passes illegal optional arguments like
./hadoop fsck /test -listcorruptfileblocks instead of
./hadoop fsck /test -list-corruptfileblocks also we can display the proper
command usage
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1727) fsck command can display command usage if user passes any illegal argument
[ https://issues.apache.org/jira/browse/HDFS-1727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039210#comment-13039210 ] Todd Lipcon commented on HDFS-1727: --- One more comment: please remove the @throws Exception from the test javadoc. It doesn't give any extra information, so just clutters the code. fsck command can display command usage if user passes any illegal argument -- Key: HDFS-1727 URL: https://issues.apache.org/jira/browse/HDFS-1727 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.1, 0.23.0 Reporter: Uma Maheswara Rao G Priority: Minor Attachments: HDFS-1727.patch In fsck command if user passes the arguments like ./hadoop fsck -test -files -blocks -racks In this case it will take / and will display whole DFS information regarding to files,blocks,racks. But here, we are hiding the user mistake. Instead of this, we can display the command usage if user passes any invalid argument like above. If user passes illegal optional arguments like ./hadoop fsck /test -listcorruptfileblocks instead of ./hadoop fsck /test -list-corruptfileblocks also we can display the proper command usage -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1592) Datanode startup doesn't honor volumes.tolerated
[ https://issues.apache.org/jira/browse/HDFS-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039240#comment-13039240 ] Jitendra Nath Pandey commented on HDFS-1592: +1 for the patch. Datanode startup doesn't honor volumes.tolerated - Key: HDFS-1592 URL: https://issues.apache.org/jira/browse/HDFS-1592 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.204.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Fix For: 0.20.204.0, 0.23.0 Attachments: HDFS-1592-1.patch, HDFS-1592-2.patch, HDFS-1592-3.patch, HDFS-1592-4.patch, HDFS-1592-5.patch, HDFS-1592-rel20.patch Datanode startup doesn't honor volumes.tolerated for hadoop 20 version. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-977) DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
[
https://issues.apache.org/jira/browse/HDFS-977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039251#comment-13039251
]
Hadoop QA commented on HDFS-977:
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12480430/HDFS-977.r2.diff
against trunk revision 1127390.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/624//testReport/
Findbugs warnings:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/624//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/624//console
This message is automatically generated.
DataNode.createInterDataNodeProtocolProxy() guards a log at the wrong level
---
Key: HDFS-977
URL: https://issues.apache.org/jira/browse/HDFS-977
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Assignee: Harsh J Chouraria
Priority: Trivial
Fix For: 0.23.0
Attachments: HDFS-977.r1.diff, HDFS-977.r2.diff
My IDE tells me that this code snippet in
{{DataNode.createInterDataNodeProtocolProxy()}} is guarding the info log
entry at debug level, and it should be reviewed
{code}
if (InterDatanodeProtocol.LOG.isDebugEnabled()) {
InterDatanodeProtocol.LOG.info(InterDatanodeProtocol addr= + addr);
}
{code}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1884) Recurring failures in TestDFSStorageStateRecovery due to HADOOP-7146
[ https://issues.apache.org/jira/browse/HDFS-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Foley updated HDFS-1884: - Description: (was: Changing summary to make it more informative in the HDFS-1852 roll-up.) Recurring failures in TestDFSStorageStateRecovery due to HADOOP-7146 Key: HDFS-1884 URL: https://issues.apache.org/jira/browse/HDFS-1884 Project: Hadoop HDFS Issue Type: Sub-task Components: test Reporter: Matt Foley Assignee: Aaron T. Myers Attachments: hdfs-1884.0.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1983) Fix path display for copy rm
[ https://issues.apache.org/jira/browse/HDFS-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-1983: -- Status: Patch Available (was: Open) Fix path display for copy rm -- Key: HDFS-1983 URL: https://issues.apache.org/jira/browse/HDFS-1983 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.23.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-1983.patch This will also fix a few misc broken tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1983) Fix path display for copy rm
[ https://issues.apache.org/jira/browse/HDFS-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-1983: -- Attachment: HDFS-1983.patch Fixes all remaining issues in DFSShell and HDFSCLI tests. Fix path display for copy rm -- Key: HDFS-1983 URL: https://issues.apache.org/jira/browse/HDFS-1983 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.23.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-1983.patch This will also fix a few misc broken tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1943) fail to start datanode while start-dfs.sh is executed by root user
[ https://issues.apache.org/jira/browse/HDFS-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039320#comment-13039320 ] Bharath Mundlapudi commented on HDFS-1943: -- +1 to this patch. fail to start datanode while start-dfs.sh is executed by root user -- Key: HDFS-1943 URL: https://issues.apache.org/jira/browse/HDFS-1943 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Wei Yongjun Priority: Blocker Fix For: 0.23.0 Attachments: HDFS-1943.patch When start-dfs.sh is run by root user, we got the following error message: # start-dfs.sh Starting namenodes on [localhost ] localhost: namenode running as process 2556. Stop it first. localhost: starting datanode, logging to /usr/hadoop/hadoop-common-0.23.0-SNAPSHOT/bin/../logs/hadoop-root-datanode-cspf01.out localhost: Unrecognized option: -jvm localhost: Could not create the Java virtual machine. The -jvm options should be passed to jsvc when we starting a secure datanode, but it still passed to java when start-dfs.sh is run by root while secure datanode is disabled. This is a bug of bin/hdfs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1983) Fix path display for copy rm
[ https://issues.apache.org/jira/browse/HDFS-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039331#comment-13039331 ] Todd Lipcon commented on HDFS-1983: --- - looks like some hard tabs snuck in (in runCmd()) - can you use Joiner.on( ).join(args) instead of using the StringBuilder in the same function? Otherwise, looks good Fix path display for copy rm -- Key: HDFS-1983 URL: https://issues.apache.org/jira/browse/HDFS-1983 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.23.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-1983.patch This will also fix a few misc broken tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1131) each DFSClient instance uses a daemon thread for lease checking, there should be a singleton daemon for all DFSClient instances
[ https://issues.apache.org/jira/browse/HDFS-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-1131. -- Resolution: Duplicate each DFSClient instance uses a daemon thread for lease checking, there should be a singleton daemon for all DFSClient instances --- Key: HDFS-1131 URL: https://issues.apache.org/jira/browse/HDFS-1131 Project: Hadoop HDFS Issue Type: Improvement Reporter: Alejandro Abdelnur Priority: Critical When accessing HDFS from a server application that acts on behalf of several users each user has its own file system handle (DFSClient), this creates one daemon thread per file system instance. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1964) Incorrect HTML unescaping in DatanodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-1964: - Attachment: hdfs-1964-22.0.patch Sure, Todd. Here's a patch for branch-0.22. Incorrect HTML unescaping in DatanodeJspHelper.java --- Key: HDFS-1964 URL: https://issues.apache.org/jira/browse/HDFS-1964 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0, 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.22.0, 0.23.0 Attachments: hdfs-1964-22.0.patch, hdfs-1964-trunk.0.patch, hdfs-1964-trunk.1.patch HDFS-1575 introduced some HTML unescaping of parameters so that viewing a file would work for paths containing HTML-escaped characters, but in two of the places did the unescaping either too early or too late. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1983) Fix path display for copy rm
[ https://issues.apache.org/jira/browse/HDFS-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-1983: -- Attachment: HDFS-1983-2.patch Removed the tabs. Time to change my vimrc... We don't appear to currently have a dep on google's common base, so I didn't change to using a Joiner. Let me know if that's not ok. Fix path display for copy rm -- Key: HDFS-1983 URL: https://issues.apache.org/jira/browse/HDFS-1983 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.23.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-1983-2.patch, HDFS-1983.patch This will also fix a few misc broken tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1943) fail to start datanode while start-dfs.sh is executed by root user
[ https://issues.apache.org/jira/browse/HDFS-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039367#comment-13039367 ] Jakob Homan commented on HDFS-1943: --- +1 fail to start datanode while start-dfs.sh is executed by root user -- Key: HDFS-1943 URL: https://issues.apache.org/jira/browse/HDFS-1943 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Wei Yongjun Priority: Blocker Fix For: 0.23.0 Attachments: HDFS-1943.patch When start-dfs.sh is run by root user, we got the following error message: # start-dfs.sh Starting namenodes on [localhost ] localhost: namenode running as process 2556. Stop it first. localhost: starting datanode, logging to /usr/hadoop/hadoop-common-0.23.0-SNAPSHOT/bin/../logs/hadoop-root-datanode-cspf01.out localhost: Unrecognized option: -jvm localhost: Could not create the Java virtual machine. The -jvm options should be passed to jsvc when we starting a secure datanode, but it still passed to java when start-dfs.sh is run by root while secure datanode is disabled. This is a bug of bin/hdfs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1943) fail to start datanode while start-dfs.sh is executed by root user
[ https://issues.apache.org/jira/browse/HDFS-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039369#comment-13039369 ] Jakob Homan commented on HDFS-1943: --- It's not a good idea to run the dn as root, but this shouldn't be the way it's prevented. Unit test failures can't be related. Committing. fail to start datanode while start-dfs.sh is executed by root user -- Key: HDFS-1943 URL: https://issues.apache.org/jira/browse/HDFS-1943 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Wei Yongjun Priority: Blocker Fix For: 0.23.0 Attachments: HDFS-1943.patch When start-dfs.sh is run by root user, we got the following error message: # start-dfs.sh Starting namenodes on [localhost ] localhost: namenode running as process 2556. Stop it first. localhost: starting datanode, logging to /usr/hadoop/hadoop-common-0.23.0-SNAPSHOT/bin/../logs/hadoop-root-datanode-cspf01.out localhost: Unrecognized option: -jvm localhost: Could not create the Java virtual machine. The -jvm options should be passed to jsvc when we starting a secure datanode, but it still passed to java when start-dfs.sh is run by root while secure datanode is disabled. This is a bug of bin/hdfs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1943) fail to start datanode while start-dfs.sh is executed by root user
[ https://issues.apache.org/jira/browse/HDFS-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1943: -- Resolution: Fixed Fix Version/s: (was: 0.23.0) Edit log branch (HDFS-1073) Assignee: Wei Yongjun Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed to trunk. Resolving. Thanks, Wei! fail to start datanode while start-dfs.sh is executed by root user -- Key: HDFS-1943 URL: https://issues.apache.org/jira/browse/HDFS-1943 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Wei Yongjun Assignee: Wei Yongjun Priority: Blocker Fix For: Edit log branch (HDFS-1073) Attachments: HDFS-1943.patch When start-dfs.sh is run by root user, we got the following error message: # start-dfs.sh Starting namenodes on [localhost ] localhost: namenode running as process 2556. Stop it first. localhost: starting datanode, logging to /usr/hadoop/hadoop-common-0.23.0-SNAPSHOT/bin/../logs/hadoop-root-datanode-cspf01.out localhost: Unrecognized option: -jvm localhost: Could not create the Java virtual machine. The -jvm options should be passed to jsvc when we starting a secure datanode, but it still passed to java when start-dfs.sh is run by root while secure datanode is disabled. This is a bug of bin/hdfs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1943) fail to start datanode while start-dfs.sh is executed by root user
[ https://issues.apache.org/jira/browse/HDFS-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1943: -- Fix Version/s: (was: Edit log branch (HDFS-1073)) 0.23.0 fail to start datanode while start-dfs.sh is executed by root user -- Key: HDFS-1943 URL: https://issues.apache.org/jira/browse/HDFS-1943 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Wei Yongjun Assignee: Wei Yongjun Priority: Blocker Fix For: 0.23.0 Attachments: HDFS-1943.patch When start-dfs.sh is run by root user, we got the following error message: # start-dfs.sh Starting namenodes on [localhost ] localhost: namenode running as process 2556. Stop it first. localhost: starting datanode, logging to /usr/hadoop/hadoop-common-0.23.0-SNAPSHOT/bin/../logs/hadoop-root-datanode-cspf01.out localhost: Unrecognized option: -jvm localhost: Could not create the Java virtual machine. The -jvm options should be passed to jsvc when we starting a secure datanode, but it still passed to java when start-dfs.sh is run by root while secure datanode is disabled. This is a bug of bin/hdfs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1983) Fix path display for copy rm
[ https://issues.apache.org/jira/browse/HDFS-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039395#comment-13039395 ] Todd Lipcon commented on HDFS-1983: --- We should be picking up guava via common, which now depends on it. No big deal, though. +1 on this patch. Unfortunately Hudson seems stalled, but since this is just a test change, I will manually run the modified tests before commit. Fix path display for copy rm -- Key: HDFS-1983 URL: https://issues.apache.org/jira/browse/HDFS-1983 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.23.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-1983-2.patch, HDFS-1983.patch This will also fix a few misc broken tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1983) Fix path display for copy rm
[ https://issues.apache.org/jira/browse/HDFS-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1983: -- Resolution: Fixed Fix Version/s: 0.23.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Related passed manually. Committed to trunk, thanks Daryn! Fix path display for copy rm -- Key: HDFS-1983 URL: https://issues.apache.org/jira/browse/HDFS-1983 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.23.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 0.23.0 Attachments: HDFS-1983-2.patch, HDFS-1983.patch This will also fix a few misc broken tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1936) Updating the layout version from HDFS-1822 causes upgrade problems.
[ https://issues.apache.org/jira/browse/HDFS-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039398#comment-13039398 ] Suresh Srinivas commented on HDFS-1936: --- imgVersion -23 This is a bug in the existing code. Delegation token was added in -24. So the check should be -24. Removed // added in -23 I would revert this. The comments are replete with when a particular field is added. I prefer not adding these details to comments, we can keep that documentation really up to date. DataNode.getUpgradeManagerDatanode, which treats the DN like a singleton... This is very similar to what is in 22. 22 uses a static method as well. I will look into this. If this is not closely related to this change, will address it in a separate bug. Updating the layout version from HDFS-1822 causes upgrade problems. --- Key: HDFS-1936 URL: https://issues.apache.org/jira/browse/HDFS-1936 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.22.0, 0.23.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Blocker Fix For: 0.22.0, 0.23.0 Attachments: HDFS-1936.22.patch, HDFS-1936.trunk.patch, hadoop-22-dfs-dir.tgz, hdfs-1936-with-testcase.txt In HDFS-1822 and HDFS-1842, the layout versions for 203, 204, 22 and trunk were changed. Some of the namenode logic that depends on layout version is broken because of this. Read the comment for more description. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1996) ivy: hdfs test jar should be independent to common test jar
ivy: hdfs test jar should be independent to common test jar
---
Key: HDFS-1996
URL: https://issues.apache.org/jira/browse/HDFS-1996
Project: Hadoop HDFS
Issue Type: Improvement
Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Eric Yang
hdfs tests and common tests may require different libraries, e.g. common tests
need ftpserver for testing {{FTPFileSystem}} but hdfs does not.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1997) Image transfer process misreports client side exceptions
Image transfer process misreports client side exceptions Key: HDFS-1997 URL: https://issues.apache.org/jira/browse/HDFS-1997 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Todd Lipcon Assignee: Todd Lipcon -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1997) Image transfer process misreports client side exceptions
[ https://issues.apache.org/jira/browse/HDFS-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1997: -- Component/s: name-node Description: If the image transfer process receives a client-side error when transferring edit logs, it will throw an exception before it has completely read all of the input from the server-side servlet. Then, the finally clause will throw a new error, since the received length is less than the length given in the header. This masks the client-side exception and makes it look like a network error or a server-side problem. Affects Version/s: Edit log branch (HDFS-1073) Fix Version/s: Edit log branch (HDFS-1073) Image transfer process misreports client side exceptions Key: HDFS-1997 URL: https://issues.apache.org/jira/browse/HDFS-1997 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) If the image transfer process receives a client-side error when transferring edit logs, it will throw an exception before it has completely read all of the input from the server-side servlet. Then, the finally clause will throw a new error, since the received length is less than the length given in the header. This masks the client-side exception and makes it look like a network error or a server-side problem. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1998) make refresh-namodenodes.sh refreshing all namenodes
make refresh-namodenodes.sh refreshing all namenodes Key: HDFS-1998 URL: https://issues.apache.org/jira/browse/HDFS-1998 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Tanping Wang Assignee: Tanping Wang Priority: Minor Fix For: 0.23.0 refresh-namenodes.sh is used to refresh name nodes in the cluster to check for updates of include/exclude list. It is used when decommissioning or adding a data node. Currently it only refreshes the name node who serves the defaultFs, if there is defaultFs defined. Fix it by refreshing all the name nodes in the cluster. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1983) Fix path display for copy rm
[ https://issues.apache.org/jira/browse/HDFS-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039414#comment-13039414 ] Hudson commented on HDFS-1983: -- Integrated in Hadoop-Hdfs-trunk-Commit #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/685/]) HDFS-1983. Fix path display for copy and rm commands in TestHDFSCLI and TestDFSShell. Contributed by Daryn Sharp. todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1127712 Files : * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/cli/testHDFSConf.xml * /hadoop/hdfs/trunk/src/test/hdfs/org/apache/hadoop/hdfs/TestDFSShell.java * /hadoop/hdfs/trunk/CHANGES.txt Fix path display for copy rm -- Key: HDFS-1983 URL: https://issues.apache.org/jira/browse/HDFS-1983 Project: Hadoop HDFS Issue Type: Test Components: test Affects Versions: 0.23.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 0.23.0 Attachments: HDFS-1983-2.patch, HDFS-1983.patch This will also fix a few misc broken tests. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1943) fail to start datanode while start-dfs.sh is executed by root user
[ https://issues.apache.org/jira/browse/HDFS-1943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039413#comment-13039413 ] Hudson commented on HDFS-1943: -- Integrated in Hadoop-Hdfs-trunk-Commit #685 (See [https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/685/]) fail to start datanode while start-dfs.sh is executed by root user -- Key: HDFS-1943 URL: https://issues.apache.org/jira/browse/HDFS-1943 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 0.23.0 Reporter: Wei Yongjun Assignee: Wei Yongjun Priority: Blocker Fix For: 0.23.0 Attachments: HDFS-1943.patch When start-dfs.sh is run by root user, we got the following error message: # start-dfs.sh Starting namenodes on [localhost ] localhost: namenode running as process 2556. Stop it first. localhost: starting datanode, logging to /usr/hadoop/hadoop-common-0.23.0-SNAPSHOT/bin/../logs/hadoop-root-datanode-cspf01.out localhost: Unrecognized option: -jvm localhost: Could not create the Java virtual machine. The -jvm options should be passed to jsvc when we starting a secure datanode, but it still passed to java when start-dfs.sh is run by root while secure datanode is disabled. This is a bug of bin/hdfs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1997) Image transfer process misreports client side exceptions
[ https://issues.apache.org/jira/browse/HDFS-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1997: -- Attachment: hdfs-1997.txt Image transfer process misreports client side exceptions Key: HDFS-1997 URL: https://issues.apache.org/jira/browse/HDFS-1997 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 0.22.0, Edit log branch (HDFS-1073), 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0, Edit log branch (HDFS-1073), 0.23.0 Attachments: hdfs-1997.txt If the image transfer process receives a client-side error when transferring edit logs, it will throw an exception before it has completely read all of the input from the server-side servlet. Then, the finally clause will throw a new error, since the received length is less than the length given in the header. This masks the client-side exception and makes it look like a network error or a server-side problem. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1997) Image transfer process misreports client side exceptions
[ https://issues.apache.org/jira/browse/HDFS-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1997: -- Affects Version/s: 0.23.0 0.22.0 Fix Version/s: 0.23.0 0.22.0 This affects trunk as well, so marking as such. Image transfer process misreports client side exceptions Key: HDFS-1997 URL: https://issues.apache.org/jira/browse/HDFS-1997 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 0.22.0, Edit log branch (HDFS-1073), 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0, Edit log branch (HDFS-1073), 0.23.0 Attachments: hdfs-1997.txt If the image transfer process receives a client-side error when transferring edit logs, it will throw an exception before it has completely read all of the input from the server-side servlet. Then, the finally clause will throw a new error, since the received length is less than the length given in the header. This masks the client-side exception and makes it look like a network error or a server-side problem. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1997) Image transfer process misreports client side exceptions
[ https://issues.apache.org/jira/browse/HDFS-1997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1997: -- Status: Patch Available (was: Open) Image transfer process misreports client side exceptions Key: HDFS-1997 URL: https://issues.apache.org/jira/browse/HDFS-1997 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: 0.22.0, Edit log branch (HDFS-1073), 0.23.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: 0.22.0, Edit log branch (HDFS-1073), 0.23.0 Attachments: hdfs-1997.txt If the image transfer process receives a client-side error when transferring edit logs, it will throw an exception before it has completely read all of the input from the server-side servlet. Then, the finally clause will throw a new error, since the received length is less than the length given in the header. This masks the client-side exception and makes it look like a network error or a server-side problem. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1999) Tests use deprecated configs
Tests use deprecated configs Key: HDFS-1999 URL: https://issues.apache.org/jira/browse/HDFS-1999 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0, 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.22.0, 0.23.0 A few of the HDFS tests (not intended to test deprecation) use config keys which are deprecated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2000) Missing deprecation for io.bytes.per.checksum
Missing deprecation for io.bytes.per.checksum - Key: HDFS-2000 URL: https://issues.apache.org/jira/browse/HDFS-2000 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0, 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.22.0, 0.23.0 Hadoop long ago deprecated the configuration io.bytes.per.checksum in favor of dfs.bytes-per-checksum, but when the programmatic deprecation support was added, we didn't add an entry for this pair. This is causing some tests to fail on branch-0.22 since the inclusion of HADOOP-7287, since some tests which were inadvertently using default config values are now having their settings actually picked up. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1994) Fix race conditions when running two rapidly checkpointing 2NNs
[ https://issues.apache.org/jira/browse/HDFS-1994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1994: -- Attachment: hdfs-1994.txt Updated patch that does solution a above. Also includes a new test case to trigger the interleaving and make sure only one is successful. If we decide we also want to do b above, it could be done in a followup. But it's a very rare race that only really shows up in this kind of stress test, and it doesn't cause any corruption, just a missed checkpoint. Fix race conditions when running two rapidly checkpointing 2NNs --- Key: HDFS-1994 URL: https://issues.apache.org/jira/browse/HDFS-1994 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) Attachments: hdfs-1994.txt, hdfs-1994.txt HDFS-1984 added the ability to run two secondary namenodes at the same time. However, there were two races I found when stress testing this (by running two NNs each checkpointing in a tight loop with no sleep): 1) the writing of the seen_txid file was not atomic, so it was at some points reading an empty file 2) it was possible for two checkpointers to try to take a checkpoint at the same transaction ID, which would cause the two image downloads to collide and fail -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2000) Missing deprecation for io.bytes.per.checksum
[
https://issues.apache.org/jira/browse/HDFS-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-2000:
-
Attachment: hdfs-2000-22.0.patch
Here's a patch for 0.22, which should also apply cleanly to trunk. No tests are
included, but I can confirm that the presence of this patch fixes the
{{TestFiRename}} test which had been broken on branch-0.22 since the commit of
HADOOP-7287.
Missing deprecation for io.bytes.per.checksum
-
Key: HDFS-2000
URL: https://issues.apache.org/jira/browse/HDFS-2000
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.22.0, 0.23.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Fix For: 0.22.0, 0.23.0
Attachments: hdfs-2000-22.0.patch
Hadoop long ago deprecated the configuration io.bytes.per.checksum in favor
of dfs.bytes-per-checksum, but when the programmatic deprecation support
was added, we didn't add an entry for this pair.
This is causing some tests to fail on branch-0.22 since the inclusion of
HADOOP-7287, since some tests which were inadvertently using default config
values are now having their settings actually picked up.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2001) HDFS-1073: Kill previous.checkpoint, lastcheckpoint.tmp directories
HDFS-1073: Kill previous.checkpoint, lastcheckpoint.tmp directories --- Key: HDFS-2001 URL: https://issues.apache.org/jira/browse/HDFS-2001 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Todd Lipcon Assignee: Todd Lipcon -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2000) Missing deprecation for io.bytes.per.checksum
[ https://issues.apache.org/jira/browse/HDFS-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2000: - Status: Patch Available (was: Open) Missing deprecation for io.bytes.per.checksum - Key: HDFS-2000 URL: https://issues.apache.org/jira/browse/HDFS-2000 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0, 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.22.0, 0.23.0 Attachments: hdfs-2000-22.0.patch Hadoop long ago deprecated the configuration io.bytes.per.checksum in favor of dfs.bytes-per-checksum, but when the programmatic deprecation support was added, we didn't add an entry for this pair. This is causing some tests to fail on branch-0.22 since the inclusion of HADOOP-7287, since some tests which were inadvertently using default config values are now having their settings actually picked up. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1963) HDFS rpm integration project
[ https://issues.apache.org/jira/browse/HDFS-1963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Yang updated HDFS-1963: Attachment: HDFS-1963-4.patch Make sure webapps directory is setup in $HADOOP_PREFIX/share/hadoop/hdfs/webapps, and hdfs script is setup accordingly. HDFS rpm integration project Key: HDFS-1963 URL: https://issues.apache.org/jira/browse/HDFS-1963 Project: Hadoop HDFS Issue Type: New Feature Components: build Environment: Java 6, RHEL 5.5 Reporter: Eric Yang Assignee: Eric Yang Attachments: HDFS-1963-1.patch, HDFS-1963-2.patch, HDFS-1963-3.patch, HDFS-1963-4.patch, HDFS-1963.patch This jira is corresponding to HADOOP-6255 and associated directory layout change. The patch for creating HDFS rpm packaging should be posted here for patch test build to verify against hdfs svn trunk. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2001) HDFS-1073: Kill previous.checkpoint, lastcheckpoint.tmp directories
[ https://issues.apache.org/jira/browse/HDFS-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2001: -- Component/s: name-node Description: In the old design for edits/image storage, the secondary namenode does a complicated dance of moving current/ to lastcheckpoint.tmp, checkpointing into current/, then moving lastcheckpoint.tmp back to previous.checkpoint. The idea here was so that there would always be one directory with a valid set of storage files. In the HDFS-1073 design, this complicated dance isn't necessary. If a checkpoint fails, we can just rm that single fsimage_N.ckpt file and still be left with a valid storage directory. So, we can just let the 2NN keep a single current/ dir around for all checkpoints and eliminate the complexity of the dance. Affects Version/s: Edit log branch (HDFS-1073) Fix Version/s: Edit log branch (HDFS-1073) HDFS-1073: Kill previous.checkpoint, lastcheckpoint.tmp directories --- Key: HDFS-2001 URL: https://issues.apache.org/jira/browse/HDFS-2001 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) In the old design for edits/image storage, the secondary namenode does a complicated dance of moving current/ to lastcheckpoint.tmp, checkpointing into current/, then moving lastcheckpoint.tmp back to previous.checkpoint. The idea here was so that there would always be one directory with a valid set of storage files. In the HDFS-1073 design, this complicated dance isn't necessary. If a checkpoint fails, we can just rm that single fsimage_N.ckpt file and still be left with a valid storage directory. So, we can just let the 2NN keep a single current/ dir around for all checkpoints and eliminate the complexity of the dance. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1996) ivy: hdfs test jar should be independent to common test jar
[
https://issues.apache.org/jira/browse/HDFS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-1996:
Attachment: HDFS-1996.patch
Make sure hdfs test profile contains junit jar file because it is a direct
dependency.
ivy: hdfs test jar should be independent to common test jar
---
Key: HDFS-1996
URL: https://issues.apache.org/jira/browse/HDFS-1996
Project: Hadoop HDFS
Issue Type: Improvement
Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Eric Yang
Attachments: HDFS-1996.patch
hdfs tests and common tests may require different libraries, e.g. common
tests need ftpserver for testing {{FTPFileSystem}} but hdfs does not.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1996) ivy: hdfs test jar should be independent to common test jar
[
https://issues.apache.org/jira/browse/HDFS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-1996:
Status: Patch Available (was: Open)
Declare junit as direct dependency for hdfs test ivy profile.
ivy: hdfs test jar should be independent to common test jar
---
Key: HDFS-1996
URL: https://issues.apache.org/jira/browse/HDFS-1996
Project: Hadoop HDFS
Issue Type: Improvement
Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Eric Yang
Attachments: HDFS-1996.patch
hdfs tests and common tests may require different libraries, e.g. common
tests need ftpserver for testing {{FTPFileSystem}} but hdfs does not.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1996) ivy: hdfs test jar should be independent to common test jar
[
https://issues.apache.org/jira/browse/HDFS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039434#comment-13039434
]
Tsz Wo (Nicholas), SZE commented on HDFS-1996:
--
Could you also change mockito from common-master to test-master?
ivy: hdfs test jar should be independent to common test jar
---
Key: HDFS-1996
URL: https://issues.apache.org/jira/browse/HDFS-1996
Project: Hadoop HDFS
Issue Type: Improvement
Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Eric Yang
Attachments: HDFS-1996.patch
hdfs tests and common tests may require different libraries, e.g. common
tests need ftpserver for testing {{FTPFileSystem}} but hdfs does not.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2002) Incorrect computation of needed blocks in getTurnOffTip()
Incorrect computation of needed blocks in getTurnOffTip()
-
Key: HDFS-2002
URL: https://issues.apache.org/jira/browse/HDFS-2002
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Fix For: 0.22.0
{{SafeModeInfo.getTurnOffTip()}} under-reports the number of blocks needed to
reach the safemode threshold.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2002) Incorrect computation of needed blocks in getTurnOffTip()
[
https://issues.apache.org/jira/browse/HDFS-2002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039447#comment-13039447
]
Konstantin Shvachko commented on HDFS-2002:
---
In case there are 3 blocks in the system with the default threshold 0.9990, and
with no replicas reported yet the safemode tip says:
_*{{Safe mode is ON. The reported blocks 0 needs additional 2 blocks to reach
the threshold 0.9990 of total blocks 3. Safe mode will be turned off
automatically.}}*_
While it should report one more _*{{needs additional 3 blocks}}*_.
And when the threshold is set larger than one, say 1.5, the numbers don't make
any sense at all.
Incorrect computation of needed blocks in getTurnOffTip()
-
Key: HDFS-2002
URL: https://issues.apache.org/jira/browse/HDFS-2002
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.22.0
Reporter: Konstantin Shvachko
Fix For: 0.22.0
{{SafeModeInfo.getTurnOffTip()}} under-reports the number of blocks needed to
reach the safemode threshold.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1996) ivy: hdfs test jar should be independent to common test jar
[
https://issues.apache.org/jira/browse/HDFS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated HDFS-1996:
Attachment: HDFS-1996-1.patch
Change mockit to test-master. Thanks Nicholas. :)
ivy: hdfs test jar should be independent to common test jar
---
Key: HDFS-1996
URL: https://issues.apache.org/jira/browse/HDFS-1996
Project: Hadoop HDFS
Issue Type: Improvement
Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Eric Yang
Attachments: HDFS-1996-1.patch, HDFS-1996.patch
hdfs tests and common tests may require different libraries, e.g. common
tests need ftpserver for testing {{FTPFileSystem}} but hdfs does not.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1875) MiniDFSCluster hard-codes dfs.datanode.address to localhost
[ https://issues.apache.org/jira/browse/HDFS-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039455#comment-13039455 ] Matt Foley commented on HDFS-1875: -- Hi Eric, looks generally good. A few comments: # MiniDFSCluster ** startDataNodes() - the @param checkDataNodeAddrConfig comment has been added to the javadocs for the stubbed-out original method signature. It needs to be associated with the new longer-args-list method signature. # TestDFSAddressConfig ** Configuration.unset() method is only available in trunk, not yahoo-merge, at this time. This was added to trunk in HADOOP-7001 on Nov 24, 2010, but seems to not be in yahoo-merge. Have you asked Suresh to merge HADOOP-7001 to yahoo-merge? # DataNodeCluster ** In the class javadocs and/or the USAGE string, should document the change in semantics - that the datanode addresses specified in config will be used unless not set. Also, with this setup, if you then run the Client on the same server as the DataNodeCluster, can the Client communicate successfully? Or does it try to use 127.0.0.1 and fail to connect? Have you searched for other unit test classes that use a Client and a DataNodeCluster? Do they all still work? Thanks. MiniDFSCluster hard-codes dfs.datanode.address to localhost --- Key: HDFS-1875 URL: https://issues.apache.org/jira/browse/HDFS-1875 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 0.23.0 Attachments: HDFS-1875.patch When creating RPC addresses that represent the communication sockets for each simulated DataNode, the MiniDFSCluster class hard-codes the address of the dfs.datanode.address port to be 127.0.0.1:0 The DataNodeCluster test tool uses the MiniDFSCluster class to create a selected number of simulated datanodes on a single host. In the DataNodeCluster setup, the NameNode is not simulated but is started as a separate daemon. The problem is that if the write requrests into the simulated datanodes are originated on a host that is not the same host running the simulated datanodes, the connections are refused. This is because the RPC sockets that are started by MiniDFSCluster are for localhost (127.0.0.1) and are not accessible from outside that same machine. It is proposed that the MiniDFSCluster.setupDatanodeAddress() method be overloaded in order to accommodate an environment where the NameNode is on one host, the client is on another host, and the simulated DataNodes are on yet another host (or even multiple hosts simulating multiple DataNodes each). The overloaded API would add a parameter that would be used as the basis for creating the RPS sockets. By default, it would remain 127.0.0.1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1996) ivy: hdfs test jar should be independent to common test jar
[
https://issues.apache.org/jira/browse/HDFS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039459#comment-13039459
]
Tsz Wo (Nicholas), SZE commented on HDFS-1996:
--
+1 patch looks good.
ivy: hdfs test jar should be independent to common test jar
---
Key: HDFS-1996
URL: https://issues.apache.org/jira/browse/HDFS-1996
Project: Hadoop HDFS
Issue Type: Improvement
Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Eric Yang
Attachments: HDFS-1996-1.patch, HDFS-1996.patch
hdfs tests and common tests may require different libraries, e.g. common
tests need ftpserver for testing {{FTPFileSystem}} but hdfs does not.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1623) High Availability Framework for HDFS NN
[ https://issues.apache.org/jira/browse/HDFS-1623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-1623: --- Attachment: NameNode HA_v2_1.pdf Update doc: v_2_1 Some minor updates I missed. Added section numbers High Availability Framework for HDFS NN --- Key: HDFS-1623 URL: https://issues.apache.org/jira/browse/HDFS-1623 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia Assignee: Sanjay Radia Attachments: HDFS-High-Availability.pdf, NameNode HA_v2.pdf, NameNode HA_v2_1.pdf, Namenode HA Framework.pdf -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1934) Fix NullPointerException when certain File APIs return null
[
https://issues.apache.org/jira/browse/HDFS-1934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharath Mundlapudi updated HDFS-1934:
-
Affects Version/s: (was: 0.20.205.0)
Fix Version/s: (was: 0.20.205.0)
Fix NullPointerException when certain File APIs return null
---
Key: HDFS-1934
URL: https://issues.apache.org/jira/browse/HDFS-1934
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 0.23.0
Reporter: Bharath Mundlapudi
Assignee: Bharath Mundlapudi
Fix For: 0.23.0
While testing Disk Fail Inplace, We encountered the NPE from this part of the
code.
File[] files = dir.listFiles();
for (File f : files) {
...
}
This is kinda of an API issue. When a disk is bad (or name is not a
directory), this API (listFiles, list) return null rather than throwing an
exception. This 'for loop' throws a NPE exception. And same applies to
dir.list() API.
Fix all the places where null condition was not checked.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1875) MiniDFSCluster hard-codes dfs.datanode.address to localhost
[ https://issues.apache.org/jira/browse/HDFS-1875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039467#comment-13039467 ] Matt Foley commented on HDFS-1875: -- Assumed context for the last question above is a box where datanode config addresses are specified in the environment, e.g. by hdfs-site.xml. MiniDFSCluster hard-codes dfs.datanode.address to localhost --- Key: HDFS-1875 URL: https://issues.apache.org/jira/browse/HDFS-1875 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Eric Payne Assignee: Eric Payne Fix For: 0.23.0 Attachments: HDFS-1875.patch When creating RPC addresses that represent the communication sockets for each simulated DataNode, the MiniDFSCluster class hard-codes the address of the dfs.datanode.address port to be 127.0.0.1:0 The DataNodeCluster test tool uses the MiniDFSCluster class to create a selected number of simulated datanodes on a single host. In the DataNodeCluster setup, the NameNode is not simulated but is started as a separate daemon. The problem is that if the write requrests into the simulated datanodes are originated on a host that is not the same host running the simulated datanodes, the connections are refused. This is because the RPC sockets that are started by MiniDFSCluster are for localhost (127.0.0.1) and are not accessible from outside that same machine. It is proposed that the MiniDFSCluster.setupDatanodeAddress() method be overloaded in order to accommodate an environment where the NameNode is on one host, the client is on another host, and the simulated DataNodes are on yet another host (or even multiple hosts simulating multiple DataNodes each). The overloaded API would add a parameter that would be used as the basis for creating the RPS sockets. By default, it would remain 127.0.0.1 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2001) HDFS-1073: Kill previous.checkpoint, lastcheckpoint.tmp directories
[ https://issues.apache.org/jira/browse/HDFS-2001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-2001: -- Attachment: hdfs-2001.txt This patch goes on top of HDFS-1991 and HDFS-1994. It removes the previouscheckpoint/lastcheckpoint directories and updates TestDFSStorageStateRecovery to match that. I currently commented out a couple of the test cases in TestSaveNamespace which need to still be updated and improved on this branch. I also added a new test case for the scenario when the 2NN fails two checkpoints in a row. Previous to this patch, there was a bug with that situation, but this patch addresses it. With this patch, I'm able to run a stress test with two 2NNs checkpointing as fast as they can for a while without issues. I'll let it loop for a few hours tonight. HDFS-1073: Kill previous.checkpoint, lastcheckpoint.tmp directories --- Key: HDFS-2001 URL: https://issues.apache.org/jira/browse/HDFS-2001 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) Attachments: hdfs-2001.txt In the old design for edits/image storage, the secondary namenode does a complicated dance of moving current/ to lastcheckpoint.tmp, checkpointing into current/, then moving lastcheckpoint.tmp back to previous.checkpoint. The idea here was so that there would always be one directory with a valid set of storage files. In the HDFS-1073 design, this complicated dance isn't necessary. If a checkpoint fails, we can just rm that single fsimage_N.ckpt file and still be left with a valid storage directory. So, we can just let the 2NN keep a single current/ dir around for all checkpoints and eliminate the complexity of the dance. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1592) Datanode startup doesn't honor volumes.tolerated
[ https://issues.apache.org/jira/browse/HDFS-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039473#comment-13039473 ] Jitendra Nath Pandey commented on HDFS-1592: Eli, Do you have any more concerns? I intend to commit this patch by tomorrow. Datanode startup doesn't honor volumes.tolerated - Key: HDFS-1592 URL: https://issues.apache.org/jira/browse/HDFS-1592 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.204.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Fix For: 0.20.204.0, 0.23.0 Attachments: HDFS-1592-1.patch, HDFS-1592-2.patch, HDFS-1592-3.patch, HDFS-1592-4.patch, HDFS-1592-5.patch, HDFS-1592-rel20.patch Datanode startup doesn't honor volumes.tolerated for hadoop 20 version. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-420) Fuse-dfs should cache fs handles
[ https://issues.apache.org/jira/browse/HDFS-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-420: - Fix Version/s: (was: 0.20.3) 0.23.0 Issue Type: Improvement (was: Bug) Summary: Fuse-dfs should cache fs handles (was: fuse_dfs is unable to connect to the dfs after a copying a large number of files into the dfs over fuse) Latest patch looks good. I agree we should cache fs handles in fuse-dfs. In the updated patch attached I refactored out functions for creating the fs handle table, finding and inserting references into it. I also implemented doDisconnect, it works (I bet the segv you saw was from trying to free an entry in the table when the ref-count drops to zero) however the performance is much worse. I did some measurements comparing hdfsConnectAsUserNewInstance vs hdfsConnectAsUser and they perform similarly. The initial call is always slow because it creates a JVM, but after that it looks like the bulk of the time is spent loading configuration, the FileSystem cache in the Java client doesn't make a difference (even when the client and NN are a hop away). So it turns out the big performance gain comes from caching the fs handle in fuse-dfs vs getting a handle from libHdfs. Since eg an ls can result in many fuse-dfs operations it really pays to cache the handle. The updated patch also contains the following additional changes: * Improved the fix for HDFS-1757. * Modifed doConnectAsUser to return the fs rather than use a IN/OUT arg. * Made some the ERRORs in fuse_options.c INFOs since they're not error cases * Removed the dead code in fuse_impls_access.c. This is HDFS-428. * Removed the non-PERMS code since this no longer compiles, is not used. * Removed the auto-connect in dfs_init since connection testing is already done in main. * Minor update fuse_dfs_wrapper.sh to reflect the new ivy dir names. * Removed doConnect, it's only used once and we don't want to encourage more uses. I verified TestFuseDFS passes. Also did some stress testing of a fuse-dfs mount using multiple clients/users on a pseudo-cluster running from a build of trunk. Mind taking a look at the latest code? Fuse-dfs should cache fs handles Key: HDFS-420 URL: https://issues.apache.org/jira/browse/HDFS-420 Project: Hadoop HDFS Issue Type: Improvement Components: contrib/fuse-dfs Affects Versions: 0.20.2 Environment: Fedora core 10, x86_64, 2.6.27.7-134.fc10.x86_64 #1 SMP (AMD 64), gcc 4.3.2, java 1.6.0 (IcedTea6 1.4 (fedora-7.b12.fc10-x86_64) Runtime Environment (build 1.6.0_0-b12) OpenJDK 64-Bit Server VM (build 10.0-b19, mixed mode) Reporter: Dima Brodsky Assignee: Brian Bockelman Fix For: 0.23.0 Attachments: fuse_dfs_020_memleaks.patch, fuse_dfs_020_memleaks_v3.patch, fuse_dfs_020_memleaks_v8.patch, hdfs-420-1.patch I run the following test: 1. Run hadoop DFS in single node mode 2. start up fuse_dfs 3. copy my source tree, about 250 megs, into the DFS cp -av * /mnt/hdfs/ in /var/log/messages I keep seeing: Dec 22 09:02:08 bodum fuse_dfs: ERROR: hdfs trying to utime /bar/backend-trunk2/src/machinery/hadoop/output/2008/11/19 to 1229385138/1229963739 and then eventually Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not
[jira] [Updated] (HDFS-420) Fuse-dfs should cache fs handles
[ https://issues.apache.org/jira/browse/HDFS-420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-420: - Attachment: hdfs-420-1.patch Fuse-dfs should cache fs handles Key: HDFS-420 URL: https://issues.apache.org/jira/browse/HDFS-420 Project: Hadoop HDFS Issue Type: Improvement Components: contrib/fuse-dfs Affects Versions: 0.20.2 Environment: Fedora core 10, x86_64, 2.6.27.7-134.fc10.x86_64 #1 SMP (AMD 64), gcc 4.3.2, java 1.6.0 (IcedTea6 1.4 (fedora-7.b12.fc10-x86_64) Runtime Environment (build 1.6.0_0-b12) OpenJDK 64-Bit Server VM (build 10.0-b19, mixed mode) Reporter: Dima Brodsky Assignee: Brian Bockelman Fix For: 0.23.0 Attachments: fuse_dfs_020_memleaks.patch, fuse_dfs_020_memleaks_v3.patch, fuse_dfs_020_memleaks_v8.patch, hdfs-420-1.patch I run the following test: 1. Run hadoop DFS in single node mode 2. start up fuse_dfs 3. copy my source tree, about 250 megs, into the DFS cp -av * /mnt/hdfs/ in /var/log/messages I keep seeing: Dec 22 09:02:08 bodum fuse_dfs: ERROR: hdfs trying to utime /bar/backend-trunk2/src/machinery/hadoop/output/2008/11/19 to 1229385138/1229963739 and then eventually Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1333 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1209 Dec 22 09:03:49 bodum fuse_dfs: ERROR: could not connect to dfs fuse_dfs.c:1037 and the file system hangs. hadoop is still running and I don't see any errors in it's logs. I have to unmount the dfs and restart fuse_dfs and then everything is fine again. At some point I see the following messages in the /var/log/messages: ERROR: dfs problem - could not close file_handle(139677114350528) for /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8339-93825052368848-1229278807.log fuse_dfs.c:1464 Dec 22 09:04:49 bodum fuse_dfs: ERROR: dfs problem - could not close file_handle(139676770220176) for /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8140-93825025883216-1229278759.log fuse_dfs.c:1464 Dec 22 09:05:13 bodum fuse_dfs: ERROR: dfs problem - could not close file_handle(139677114812832) for /bar/backend-trunk2/src/machinery/hadoop/input/2008/12/14/actionrecordlog-8138-93825070138960-1229251587.log fuse_dfs.c:1464 Is this a known issue? Am I just flooding the system too much. All of this is being performed on a single, dual core, machine. Thanks! ttyl Dima -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039478#comment-13039478 ] Sanjay Radia commented on HDFS-1580: Please add some text doc to motivate the Journal#getNumberOfTransactions(long sinceTxnId) . From what I understand this method gives the # of transactions since tX in a particular storage dir. Would it be better instead to ask getHighestTxid()? Add interface for generic Write Ahead Logging mechanisms Key: HDFS-1580 URL: https://issues.apache.org/jira/browse/HDFS-1580 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ivan Kelly Fix For: Edit log branch (HDFS-1073) Attachments: EditlogInterface.1.pdf, EditlogInterface.2.pdf, HDFS-1580+1521.diff, HDFS-1580.diff, HDFS-1580.diff, HDFS-1580.diff, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1996) ivy: hdfs test jar should be independent to common test jar
[
https://issues.apache.org/jira/browse/HDFS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039484#comment-13039484
]
Hadoop QA commented on HDFS-1996:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12480486/HDFS-1996-1.patch
against trunk revision 1127712.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestDFSStorageStateRecovery
+1 contrib tests. The patch passed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/625//testReport/
Findbugs warnings:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/625//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://builds.apache.org/hudson/job/PreCommit-HDFS-Build/625//console
This message is automatically generated.
ivy: hdfs test jar should be independent to common test jar
---
Key: HDFS-1996
URL: https://issues.apache.org/jira/browse/HDFS-1996
Project: Hadoop HDFS
Issue Type: Improvement
Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Eric Yang
Attachments: HDFS-1996-1.patch, HDFS-1996.patch
hdfs tests and common tests may require different libraries, e.g. common
tests need ftpserver for testing {{FTPFileSystem}} but hdfs does not.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1996) ivy: hdfs test jar should be independent to common test jar
[
https://issues.apache.org/jira/browse/HDFS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-1996:
-
Resolution: Fixed
Fix Version/s: 0.23.0
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
I have committed this. Thanks, Eric!
ivy: hdfs test jar should be independent to common test jar
---
Key: HDFS-1996
URL: https://issues.apache.org/jira/browse/HDFS-1996
Project: Hadoop HDFS
Issue Type: Improvement
Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Eric Yang
Fix For: 0.23.0
Attachments: HDFS-1996-1.patch, HDFS-1996.patch
hdfs tests and common tests may require different libraries, e.g. common
tests need ftpserver for testing {{FTPFileSystem}} but hdfs does not.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1996) ivy: hdfs test jar should be independent to common test jar
[
https://issues.apache.org/jira/browse/HDFS-1996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039490#comment-13039490
]
Hudson commented on HDFS-1996:
--
Integrated in Hadoop-Hdfs-trunk-Commit #686 (See
[https://builds.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/686/])
HDFS-1996. ivy: hdfs test jar should be independent to common test jar.
Contributed by Eric Yang
szetszwo :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1127759
Files :
* /hadoop/hdfs/trunk/CHANGES.txt
* /hadoop/hdfs/trunk/ivy.xml
ivy: hdfs test jar should be independent to common test jar
---
Key: HDFS-1996
URL: https://issues.apache.org/jira/browse/HDFS-1996
Project: Hadoop HDFS
Issue Type: Improvement
Components: build
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Eric Yang
Fix For: 0.23.0
Attachments: HDFS-1996-1.patch, HDFS-1996.patch
hdfs tests and common tests may require different libraries, e.g. common
tests need ftpserver for testing {{FTPFileSystem}} but hdfs does not.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1592) Datanode startup doesn't honor volumes.tolerated
[ https://issues.apache.org/jira/browse/HDFS-1592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039509#comment-13039509 ] Eli Collins commented on HDFS-1592: --- +1 lgtm. Thanks Bharath Nit: due an exception should due to exception Datanode startup doesn't honor volumes.tolerated - Key: HDFS-1592 URL: https://issues.apache.org/jira/browse/HDFS-1592 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.204.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Fix For: 0.20.204.0, 0.23.0 Attachments: HDFS-1592-1.patch, HDFS-1592-2.patch, HDFS-1592-3.patch, HDFS-1592-4.patch, HDFS-1592-5.patch, HDFS-1592-rel20.patch Datanode startup doesn't honor volumes.tolerated for hadoop 20 version. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2000) Missing deprecation for io.bytes.per.checksum
[ https://issues.apache.org/jira/browse/HDFS-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2000: -- Fix Version/s: (was: 0.23.0) Hadoop Flags: [Reviewed] +1 pending Hudson Missing deprecation for io.bytes.per.checksum - Key: HDFS-2000 URL: https://issues.apache.org/jira/browse/HDFS-2000 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0, 0.23.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 0.22.0 Attachments: hdfs-2000-22.0.patch Hadoop long ago deprecated the configuration io.bytes.per.checksum in favor of dfs.bytes-per-checksum, but when the programmatic deprecation support was added, we didn't add an entry for this pair. This is causing some tests to fail on branch-0.22 since the inclusion of HADOOP-7287, since some tests which were inadvertently using default config values are now having their settings actually picked up. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1991) HDFS-1073: Some refactoring of 2NN to easier share code with BN and CN
[ https://issues.apache.org/jira/browse/HDFS-1991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13039519#comment-13039519 ] Eli Collins commented on HDFS-1991: --- +1 lgtm HDFS-1073: Some refactoring of 2NN to easier share code with BN and CN -- Key: HDFS-1991 URL: https://issues.apache.org/jira/browse/HDFS-1991 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Affects Versions: Edit log branch (HDFS-1073) Reporter: Todd Lipcon Assignee: Todd Lipcon Fix For: Edit log branch (HDFS-1073) Attachments: hdfs-1991.txt Currently the Checkpointer class shares a lot of very similar code with the secondary namenode. This JIRA is to do a little cleanup in SecondaryNameNode that will make it easier to share code with the BackupNode and CheckpointNode. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
