[jira] [Updated] (HDFS-4581) DataNode#checkDiskError should not be called on network errors
[ https://issues.apache.org/jira/browse/HDFS-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-4581: -- Fix Version/s: (was: 3.0.0) 1.3.0 Applying Fix Version 1.3.0 for the branch-1 commit done here; seems to have been missed post commit. > DataNode#checkDiskError should not be called on network errors > -- > > Key: HDFS-4581 > URL: https://issues.apache.org/jira/browse/HDFS-4581 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 1.1.1, 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Rohit Kochar >Assignee: Rohit Kochar > Fix For: 0.23.7, 2.0.4-alpha, 1.3.0 > > Attachments: HDFS-4581-branch1.patch, HDFS-4581-branch1.patch, > HDFS-4581-branch1.patch, HDFS-4581-branch1.patch, HDFS-4581.patch, > HDFS-4581.patch, HDFS-4581.patch, HDFS-4581.patch, HDFS-4581.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4581) DataNode#checkDiskError should not be called on network errors
[ https://issues.apache.org/jira/browse/HDFS-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-4581: -- Target Version/s: (was: 1.2.0, 3.0.0) > DataNode#checkDiskError should not be called on network errors > -- > > Key: HDFS-4581 > URL: https://issues.apache.org/jira/browse/HDFS-4581 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 1.1.1, 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Rohit Kochar >Assignee: Rohit Kochar > Fix For: 0.23.7, 2.0.4-alpha, 1.3.0 > > Attachments: HDFS-4581-branch1.patch, HDFS-4581-branch1.patch, > HDFS-4581-branch1.patch, HDFS-4581-branch1.patch, HDFS-4581.patch, > HDFS-4581.patch, HDFS-4581.patch, HDFS-4581.patch, HDFS-4581.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4622) Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1
[ https://issues.apache.org/jira/browse/HDFS-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harsh J updated HDFS-4622: -- Fix Version/s: (was: 1.2.1) 1.3.0 > Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1 > --- > > Key: HDFS-4622 > URL: https://issues.apache.org/jira/browse/HDFS-4622 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Trivial > Fix For: 1.3.0 > > Attachments: HDFS-4622.b1.001.patch > > > HDFS-4222 moves checkSuperuserPrivilege out of the namespace lock scope but > missed the rollEditLog method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4622) Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1
[ https://issues.apache.org/jira/browse/HDFS-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13634978#comment-13634978 ] Harsh J commented on HDFS-4622: --- Reset the Fix Version to point to the right one. 1.2.1 was not a valid point, as 1.2.0 itself is unreleased and this patch didn't go to the 1.2 branch. If it did, please reset to 1.2.0. > Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1 > --- > > Key: HDFS-4622 > URL: https://issues.apache.org/jira/browse/HDFS-4622 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 1.2.0 >Reporter: Jing Zhao >Assignee: Jing Zhao >Priority: Trivial > Fix For: 1.3.0 > > Attachments: HDFS-4622.b1.001.patch > > > HDFS-4222 moves checkSuperuserPrivilege out of the namespace lock scope but > missed the rollEditLog method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4710) Turning off HDFS short-circuit checksums unexpectedly slows down Hive
Gopal V created HDFS-4710:
-
Summary: Turning off HDFS short-circuit checksums unexpectedly
slows down Hive
Key: HDFS-4710
URL: https://issues.apache.org/jira/browse/HDFS-4710
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.0.4-alpha
Environment: Centos (EC2) + short-circuit reads on
Reporter: Gopal V
Priority: Minor
When short-circuit reads are on, HDFS client slows down when checksums are
turned off.
With checksums on, the query takes 45.341 seconds and with it turned off, it
takes 56.345 seconds. This is slower than the speeds observed when
short-circuiting is turned off.
The issue seems to be that FSDataInputStream.readByte() calls are directly
transferred to the disk fd when the checksums are turned off.
Even though all the columns are integers, the data being read will be read via
DataInputStream which does
{code}
public final int readInt() throws IOException {
int ch1 = in.read();
int ch2 = in.read();
int ch3 = in.read();
int ch4 = in.read();
{code}
To confirm, an strace of the Yarn container shows
{code}
26690 read(154, "B", 1) = 1
26690 read(154, "\250", 1) = 1
26690 read(154, ".", 1) = 1
26690 read(154, "\24", 1) = 1
{code}
To emulate this without the entirety of Hive code, I have written a simpler
test app
https://github.com/t3rmin4t0r/shortcircuit-reader
The jar will read a file in -bs sized buffers. Running it with 1 byte
blocks gives similar results to the Hive test run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4686) Fix quota computation for rename with snapshots
[ https://issues.apache.org/jira/browse/HDFS-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4686: Attachment: HDFS-4686.001.patch Initial patch for review. Still need to fix quota computation for (snapshot deletion + rename). > Fix quota computation for rename with snapshots > --- > > Key: HDFS-4686 > URL: https://issues.apache.org/jira/browse/HDFS-4686 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4686.001.patch > > > Currently after a rename operation within/from a snapshottable directory, a > reference node is created in both src and dst subtree, pointing to the > original renamed inode. With this change the original quota computation may > count the quota usage of the renamed subtree multiple times. This jira tries > to fix this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4695) TestEditLog leaks open file handles between tests
[ https://issues.apache.org/jira/browse/HDFS-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635081#comment-13635081 ] Hudson commented on HDFS-4695: -- Integrated in Hadoop-Yarn-trunk #187 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/187/]) HDFS-4695. TestEditLog leaks open file handles between tests. Contributed by Ivan Mitic. (Revision 1469015) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469015 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java > TestEditLog leaks open file handles between tests > - > > Key: HDFS-4695 > URL: https://issues.apache.org/jira/browse/HDFS-4695 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0-alpha >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 2.0.5-beta > > Attachments: HDFS-4695.2.patch, HDFS-4695.patch > > > The test leaks open file handles causing subsequent test cases to fail on > Windows (common cross-platform issue we've seen multiple times so far). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4695) TestEditLog leaks open file handles between tests
[ https://issues.apache.org/jira/browse/HDFS-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635137#comment-13635137 ] Hudson commented on HDFS-4695: -- Integrated in Hadoop-Hdfs-trunk #1376 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1376/]) HDFS-4695. TestEditLog leaks open file handles between tests. Contributed by Ivan Mitic. (Revision 1469015) Result = FAILURE suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469015 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java > TestEditLog leaks open file handles between tests > - > > Key: HDFS-4695 > URL: https://issues.apache.org/jira/browse/HDFS-4695 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0-alpha >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 2.0.5-beta > > Attachments: HDFS-4695.2.patch, HDFS-4695.patch > > > The test leaks open file handles causing subsequent test cases to fail on > Windows (common cross-platform issue we've seen multiple times so far). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4707) Fix FilterFileSystem and findbugs warning in Snapshot branch
[ https://issues.apache.org/jira/browse/HDFS-4707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635139#comment-13635139 ] Hudson commented on HDFS-4707: -- Integrated in Hadoop-Hdfs-Snapshots-Branch-build #161 (See [https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/161/]) HDFS-4707. Add snapshot methods to FilterFileSystem and fix findbugs warnings. (Revision 1469119) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469119 Files : * /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FilterFileSystem.java * /hadoop/common/branches/HDFS-2802/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFilterFileSystem.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-2802.txt * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageSerialization.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/Snapshot.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotFSImageFormat.java > Fix FilterFileSystem and findbugs warning in Snapshot branch > > > Key: HDFS-4707 > URL: https://issues.apache.org/jira/browse/HDFS-4707 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: Snapshot (HDFS-2802) > > Attachments: h4707_20130417.patch > > > The snapshot methods are not added to FilterFileSystem. > Findbugs warnings: > - SIC Should snapshot.Snapshot$Root be a _static_ inner class? > - WMI Method FSImageFormat$Saver.saveImage(ByteBuffer, > INodeDirectory, DataOutputStream, Snapshot, boolean) makes inefficient use of > keySet iterator instead of entrySet iterator > - BC Unchecked/unconfirmed cast from > FSImageSerialization.writeINodeFile(INodeFile, DataOutput, boolean) > - BC Unchecked/unconfirmed cast from > snapshot.SnapshotFSImageFormat.loadDirectoryDiffList(INodeDirectory, > DataInput, FSImageFormat$Loader) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4706) disallowSnapshot does not work for root
[ https://issues.apache.org/jira/browse/HDFS-4706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635140#comment-13635140 ] Hudson commented on HDFS-4706: -- Integrated in Hadoop-Hdfs-Snapshots-Branch-build #161 (See [https://builds.apache.org/job/Hadoop-Hdfs-Snapshots-Branch-build/161/]) HDFS-4706. Do not replace root inode for disallowSnapshot. (Revision 1469122) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469122 Files : * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-2802.txt * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/SnapshottableDirectoryStatus.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/INodeDirectorySnapshottable.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotManager.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/SnapshotStats.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestNestedSnapshots.java * /hadoop/common/branches/HDFS-2802/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotMetrics.java > disallowSnapshot does not work for root > --- > > Key: HDFS-4706 > URL: https://issues.apache.org/jira/browse/HDFS-4706 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: Snapshot (HDFS-2802) > > Attachments: h4706_20130417.patch > > > disallowSnapshot replaces a snapshottable directory to a normal directory. > However, we cannot replace root. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4711) Can not change replication factor of file during moving or deleting.
Vladimir Barinov created HDFS-4711: -- Summary: Can not change replication factor of file during moving or deleting. Key: HDFS-4711 URL: https://issues.apache.org/jira/browse/HDFS-4711 Project: Hadoop HDFS Issue Type: Bug Reporter: Vladimir Barinov Priority: Minor I don't know is it a feature or a bug. According to hdfs dfs -help we can use key -D to set specific options for action; When we copying or uploading file to hdfs we can override replication factor with -D dfs.replication=N. That's works well. But it doesn't work for moving or removing(to trash) file. Step to reproduce: Uploading file hdfs dfs -put somefile /tmp/somefile Copying with changing replication: hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4711) Can not change replication factor of file during moving or deleting.
[ https://issues.apache.org/jira/browse/HDFS-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Barinov updated HDFS-4711: --- Description: I don't know is it a feature or a bug. According to hdfs dfs -help we can use key -D to set specific options for action; When we copying or uploading file to hdfs we can override replication factor with -D dfs.replication=N. That's works well. But it doesn't work for moving or removing(to trash) file. Steps to reproduce: Uploading file hdfs dfs -put somefile /tmp/somefile Copying with changing replication: hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 was: I don't know is it a feature or a bug. According to hdfs dfs -help we can use key -D to set specific options for action; When we copying or uploading file to hdfs we can override replication factor with -D dfs.replication=N. That's works well. But it doesn't work for moving or removing(to trash) file. Step to reproduce: Uploading file hdfs dfs -put somefile /tmp/somefile Copying with changing replication: hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 > Can not change replication factor of file during moving or deleting. > > > Key: HDFS-4711 > URL: https://issues.apache.org/jira/browse/HDFS-4711 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Vladimir Barinov >Priority: Minor > > I don't know is it a feature or a bug. > According to hdfs dfs -help we can use key -D to set specific options for > action; > When we copying or uploading file to hdfs we can override replication factor > with -D dfs.replication=N. That's works well. > But it doesn't work for moving or removing(to trash) file. > Steps to reproduce: > Uploading file > hdfs dfs -put somefile /tmp/somefile > Copying with changing replication: > hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4711) Can not change replication factor of file during moving or deleting.
[ https://issues.apache.org/jira/browse/HDFS-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Barinov updated HDFS-4711: --- Affects Version/s: 2.0.0-alpha > Can not change replication factor of file during moving or deleting. > > > Key: HDFS-4711 > URL: https://issues.apache.org/jira/browse/HDFS-4711 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Vladimir Barinov >Priority: Minor > > I don't know is it a feature or a bug. > According to hdfs dfs -help we can use key -D to set specific options for > action; > When we copying or uploading file to hdfs we can override replication factor > with -D dfs.replication=N. That's works well. > But it doesn't work for moving or removing(to trash) file. > Steps to reproduce: > Uploading file > hdfs dfs -put somefile /tmp/somefile > Copying with changing replication: > hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4711) Can not change replication factor of file during moving or deleting.
[ https://issues.apache.org/jira/browse/HDFS-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Barinov updated HDFS-4711: --- Description: I don't know is it a feature or a bug. According to hdfs dfs -help we can use key -D to set specific options for action; When we copying or uploading file to hdfs we can override replication factor with -D dfs.replication=N. That's works well. But it doesn't work for moving or removing(to trash) file. Steps to reproduce: Uploading file hdfs dfs -put somefile /tmp/somefile Copying with changing replication: hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 hadoop version: Hadoop 2.0.0-cdh4.1.2 was: I don't know is it a feature or a bug. According to hdfs dfs -help we can use key -D to set specific options for action; When we copying or uploading file to hdfs we can override replication factor with -D dfs.replication=N. That's works well. But it doesn't work for moving or removing(to trash) file. Steps to reproduce: Uploading file hdfs dfs -put somefile /tmp/somefile Copying with changing replication: hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 > Can not change replication factor of file during moving or deleting. > > > Key: HDFS-4711 > URL: https://issues.apache.org/jira/browse/HDFS-4711 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Vladimir Barinov >Priority: Minor > > I don't know is it a feature or a bug. > According to hdfs dfs -help we can use key -D to set specific options for > action; > When we copying or uploading file to hdfs we can override replication factor > with -D dfs.replication=N. That's works well. > But it doesn't work for moving or removing(to trash) file. > Steps to reproduce: > Uploading file > hdfs dfs -put somefile /tmp/somefile > Copying with changing replication: > hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 > hadoop version: > Hadoop 2.0.0-cdh4.1.2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4712) New libhdfs method hdfsGetDataNodes
andrea manzi created HDFS-4712: -- Summary: New libhdfs method hdfsGetDataNodes Key: HDFS-4712 URL: https://issues.apache.org/jira/browse/HDFS-4712 Project: Hadoop HDFS Issue Type: New Feature Components: libhdfs Reporter: andrea manzi we have implemented a possible extension to libhdfs to retrieve information about the available datanodes ( there was a mail on the hadoop-hdsf-dev mailing list initially abut this : http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201204.mbox/%3CCANhO- s0mvororrxpjnjbql6brkj4c7l+u816xkdc+2r0whj...@mail.gmail.com%3E) i would like to know how to proceed to create a patch, cause on the wiki http://wiki.apache.org/hadoop/HowToContribute i can see info about JAVA patches but nothing related to extensions in C. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4695) TestEditLog leaks open file handles between tests
[ https://issues.apache.org/jira/browse/HDFS-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635180#comment-13635180 ] Hudson commented on HDFS-4695: -- Integrated in Hadoop-Mapreduce-trunk #1403 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1403/]) HDFS-4695. TestEditLog leaks open file handles between tests. Contributed by Ivan Mitic. (Revision 1469015) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469015 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java > TestEditLog leaks open file handles between tests > - > > Key: HDFS-4695 > URL: https://issues.apache.org/jira/browse/HDFS-4695 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0-alpha >Reporter: Ivan Mitic >Assignee: Ivan Mitic > Fix For: 2.0.5-beta > > Attachments: HDFS-4695.2.patch, HDFS-4695.patch > > > The test leaks open file handles causing subsequent test cases to fail on > Windows (common cross-platform issue we've seen multiple times so far). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4709) TestDFSClientRetries#testGetFileChecksum
[ https://issues.apache.org/jira/browse/HDFS-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HDFS-4709: - Summary: TestDFSClientRetries#testGetFileChecksum (was: TestDFSClientRetries#testGetFileChecksum fails using IBM java 6) > TestDFSClientRetries#testGetFileChecksum > > > Key: HDFS-4709 > URL: https://issues.apache.org/jira/browse/HDFS-4709 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.3-alpha >Reporter: Tian Hong Wang >Assignee: Tian Hong Wang > Labels: patch > Fix For: 2.0.3-alpha > > Attachments: HDFS-4709.patch > > > testGetFileChecksum(org.apache.hadoop.hdfs.TestDFSClientRetries) Time > elapsed: 3993 sec <<< ERROR! > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /testGetFileChecksum could only be replicated to 0 nodes instead of > minReplication (=1). There are 3 datanode(s) running and 3 node(s) are > excluded in this operation. > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1339) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2186) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:491) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:351) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40744) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1735) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1731) > at > java.security.AccessController.doPrivileged(AccessController.java:310) > at javax.security.auth.Subject.doAs(Subject.java:573) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1729) > at org.apache.hadoop.ipc.Client.call(Client.java:1235) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) > at $Proxy10.addBlock(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) > at java.lang.reflect.Method.invoke(Method.java:611) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) > at $Proxy10.addBlock(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:311) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1156) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1009) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4709) TestDFSClientRetries#testGetFileChecksum fails
[ https://issues.apache.org/jira/browse/HDFS-4709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tian Hong Wang updated HDFS-4709: - Summary: TestDFSClientRetries#testGetFileChecksum fails (was: TestDFSClientRetries#testGetFileChecksum) > TestDFSClientRetries#testGetFileChecksum fails > -- > > Key: HDFS-4709 > URL: https://issues.apache.org/jira/browse/HDFS-4709 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.3-alpha >Reporter: Tian Hong Wang >Assignee: Tian Hong Wang > Labels: patch > Fix For: 2.0.3-alpha > > Attachments: HDFS-4709.patch > > > testGetFileChecksum(org.apache.hadoop.hdfs.TestDFSClientRetries) Time > elapsed: 3993 sec <<< ERROR! > org.apache.hadoop.ipc.RemoteException(java.io.IOException): File > /testGetFileChecksum could only be replicated to 0 nodes instead of > minReplication (=1). There are 3 datanode(s) running and 3 node(s) are > excluded in this operation. > at > org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1339) > at > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2186) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:491) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:351) > at > org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:40744) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:454) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1735) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1731) > at > java.security.AccessController.doPrivileged(AccessController.java:310) > at javax.security.auth.Subject.doAs(Subject.java:573) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1729) > at org.apache.hadoop.ipc.Client.call(Client.java:1235) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) > at $Proxy10.addBlock(Unknown Source) > at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) > at java.lang.reflect.Method.invoke(Method.java:611) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:164) > at > org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:83) > at $Proxy10.addBlock(Unknown Source) > at > org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:311) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1156) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1009) > at > org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:464) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4699) TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
[
https://issues.apache.org/jira/browse/HDFS-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4699:
Status: Open (was: Patch Available)
> TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
> ---
>
> Key: HDFS-4699
> URL: https://issues.apache.org/jira/browse/HDFS-4699
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4699.1.patch
>
>
> I have seen {{TestPipelinesFailover#testPipelineRecoveryStress}} fail
> sporadically due to timeout during {{loopRecoverLease}}, which waits for up
> to 30 seconds before timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4699) TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
[
https://issues.apache.org/jira/browse/HDFS-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4699:
Target Version/s: 3.0.0, 0.23.7, 2.0.4-alpha, 1.3.0 (was: 3.0.0)
Affects Version/s: 1.3.0
2.0.4-alpha
0.23.7
> TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
> ---
>
> Key: HDFS-4699
> URL: https://issues.apache.org/jira/browse/HDFS-4699
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha, 1.3.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4699.1.patch
>
>
> I have seen {{TestPipelinesFailover#testPipelineRecoveryStress}} fail
> sporadically due to timeout during {{loopRecoverLease}}, which waits for up
> to 30 seconds before timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4699) TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
[
https://issues.apache.org/jira/browse/HDFS-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4699:
Attachment: HDFS-4699.branch-1.1.patch
HDFS-4699.branch-0.23.1.patch
> TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
> ---
>
> Key: HDFS-4699
> URL: https://issues.apache.org/jira/browse/HDFS-4699
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha, 1.3.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4699.1.patch, HDFS-4699.branch-0.23.1.patch,
> HDFS-4699.branch-1.1.patch
>
>
> I have seen {{TestPipelinesFailover#testPipelineRecoveryStress}} fail
> sporadically due to timeout during {{loopRecoverLease}}, which waits for up
> to 30 seconds before timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4699) TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
[
https://issues.apache.org/jira/browse/HDFS-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4699:
Attachment: (was: HDFS-4699.1.patch)
> TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
> ---
>
> Key: HDFS-4699
> URL: https://issues.apache.org/jira/browse/HDFS-4699
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha, 1.3.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4699.1.patch, HDFS-4699.branch-0.23.1.patch,
> HDFS-4699.branch-1.1.patch
>
>
> I have seen {{TestPipelinesFailover#testPipelineRecoveryStress}} fail
> sporadically due to timeout during {{loopRecoverLease}}, which waits for up
> to 30 seconds before timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4699) TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
[
https://issues.apache.org/jira/browse/HDFS-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4699:
Attachment: HDFS-4699.1.patch
> TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
> ---
>
> Key: HDFS-4699
> URL: https://issues.apache.org/jira/browse/HDFS-4699
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha, 1.3.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4699.1.patch, HDFS-4699.branch-0.23.1.patch,
> HDFS-4699.branch-1.1.patch
>
>
> I have seen {{TestPipelinesFailover#testPipelineRecoveryStress}} fail
> sporadically due to timeout during {{loopRecoverLease}}, which waits for up
> to 30 seconds before timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4699) TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
[
https://issues.apache.org/jira/browse/HDFS-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4699:
Status: Patch Available (was: Open)
> TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
> ---
>
> Key: HDFS-4699
> URL: https://issues.apache.org/jira/browse/HDFS-4699
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha, 1.3.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4699.1.patch, HDFS-4699.branch-0.23.1.patch,
> HDFS-4699.branch-1.1.patch
>
>
> I have seen {{TestPipelinesFailover#testPipelineRecoveryStress}} fail
> sporadically due to timeout during {{loopRecoverLease}}, which waits for up
> to 30 seconds before timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4699) TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
[
https://issues.apache.org/jira/browse/HDFS-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635290#comment-13635290
]
Chris Nauroth commented on HDFS-4699:
-
I noticed that HDFS-4581 also put the network error filtering logic into
branch-1 and branch-0.23, so I've added patches for those branches too. I
uploaded the trunk patch again, just so that Jenkins sees it as the newest file.
> TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
> ---
>
> Key: HDFS-4699
> URL: https://issues.apache.org/jira/browse/HDFS-4699
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha, 1.3.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4699.1.patch, HDFS-4699.branch-0.23.1.patch,
> HDFS-4699.branch-1.1.patch
>
>
> I have seen {{TestPipelinesFailover#testPipelineRecoveryStress}} fail
> sporadically due to timeout during {{loopRecoverLease}}, which waits for up
> to 30 seconds before timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4581) DataNode#checkDiskError should not be called on network errors
[ https://issues.apache.org/jira/browse/HDFS-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635292#comment-13635292 ] Chris Nauroth commented on HDFS-4581: - In HDFS-4699, I have added more cases to the network error filtering logic, based on reviewing logs from tests of rapid NN failovers. > DataNode#checkDiskError should not be called on network errors > -- > > Key: HDFS-4581 > URL: https://issues.apache.org/jira/browse/HDFS-4581 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 1.1.1, 3.0.0, 0.23.7, 2.0.4-alpha >Reporter: Rohit Kochar >Assignee: Rohit Kochar > Fix For: 0.23.7, 2.0.4-alpha, 1.3.0 > > Attachments: HDFS-4581-branch1.patch, HDFS-4581-branch1.patch, > HDFS-4581-branch1.patch, HDFS-4581-branch1.patch, HDFS-4581.patch, > HDFS-4581.patch, HDFS-4581.patch, HDFS-4581.patch, HDFS-4581.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4713) Wrong server principal is used for rpc calls to namenode if HA is enabled
Kihwal Lee created HDFS-4713: Summary: Wrong server principal is used for rpc calls to namenode if HA is enabled Key: HDFS-4713 URL: https://issues.apache.org/jira/browse/HDFS-4713 Project: Hadoop HDFS Issue Type: Bug Components: ha, namenode Affects Versions: 2.0.4-alpha Reporter: Kihwal Lee Priority: Blocker When various components are connecting to a namenode in a HA-enabled environment, a wrong server principal may be picked up. This result in SASL failure, since the client-side used a wrong service ticket for the connection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4713) Wrong server principal is used for rpc calls to namenode if HA is enabled
[ https://issues.apache.org/jira/browse/HDFS-4713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635435#comment-13635435 ] Kihwal Lee commented on HDFS-4713: -- For standby bootstrapping, HDFS-3284 made it work but HDFS-3438 broke it again. In general the proxy object for a protocol should be created with the config that has the correct server principle set. For example, standby namenode talks to active namenode via NamenodeProtocol, where the server principal key is defined as DFS_NAMENODE_USER_NAME_KEY, "dfs.namenode.kerberos.principal". The standby bootstrapping and checkpointing fail because the namenode proxy object has a conf with DFS_NAMENODE_USER_NAME_KEY set to itself. The RPC address is correctly set, but the wrong server principle is used. When I modified the code to create the proxy with "other NN config", everything worked. I haven't checked thoroughly, but ConfiguredFailoverProxyProvider may have a similar issue. > Wrong server principal is used for rpc calls to namenode if HA is enabled > - > > Key: HDFS-4713 > URL: https://issues.apache.org/jira/browse/HDFS-4713 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, namenode >Affects Versions: 2.0.4-alpha >Reporter: Kihwal Lee >Priority: Blocker > > When various components are connecting to a namenode in a HA-enabled > environment, a wrong server principal may be picked up. This result in SASL > failure, since the client-side used a wrong service ticket for the connection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4713) Wrong server principal is used for rpc calls to namenode if HA is enabled
[ https://issues.apache.org/jira/browse/HDFS-4713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-4713: - Affects Version/s: (was: 2.0.4-alpha) 2.0.5-beta > Wrong server principal is used for rpc calls to namenode if HA is enabled > - > > Key: HDFS-4713 > URL: https://issues.apache.org/jira/browse/HDFS-4713 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, namenode >Affects Versions: 2.0.5-beta >Reporter: Kihwal Lee >Priority: Blocker > > When various components are connecting to a namenode in a HA-enabled > environment, a wrong server principal may be picked up. This result in SASL > failure, since the client-side used a wrong service ticket for the connection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4699) TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
[
https://issues.apache.org/jira/browse/HDFS-4699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635450#comment-13635450
]
Hadoop QA commented on HDFS-4699:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579356/HDFS-4699.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4274//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4274//console
This message is automatically generated.
> TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
> ---
>
> Key: HDFS-4699
> URL: https://issues.apache.org/jira/browse/HDFS-4699
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Affects Versions: 3.0.0, 0.23.7, 2.0.4-alpha, 1.3.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4699.1.patch, HDFS-4699.branch-0.23.1.patch,
> HDFS-4699.branch-1.1.patch
>
>
> I have seen {{TestPipelinesFailover#testPipelineRecoveryStress}} fail
> sporadically due to timeout during {{loopRecoverLease}}, which waits for up
> to 30 seconds before timing out.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4710) Turning off HDFS short-circuit checksums unexpectedly slows down Hive
[
https://issues.apache.org/jira/browse/HDFS-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635476#comment-13635476
]
Colin Patrick McCabe commented on HDFS-4710:
Hi Gopal,
Good find.
I suppose the fix here is to change {{BlockReaderLocal}} and
{{BlockReaderLocalLegacy}} to honor
{{dfs.client.read.shortcircuit.buffer.size}} when
{{dfs.client.read.shortcircuit.skip.checksum}} is turned on.
> Turning off HDFS short-circuit checksums unexpectedly slows down Hive
> -
>
> Key: HDFS-4710
> URL: https://issues.apache.org/jira/browse/HDFS-4710
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.0.4-alpha
> Environment: Centos (EC2) + short-circuit reads on
>Reporter: Gopal V
>Priority: Minor
> Labels: perfomance
>
> When short-circuit reads are on, HDFS client slows down when checksums are
> turned off.
> With checksums on, the query takes 45.341 seconds and with it turned off, it
> takes 56.345 seconds. This is slower than the speeds observed when
> short-circuiting is turned off.
> The issue seems to be that FSDataInputStream.readByte() calls are directly
> transferred to the disk fd when the checksums are turned off.
> Even though all the columns are integers, the data being read will be read
> via DataInputStream which does
> {code}
> public final int readInt() throws IOException {
> int ch1 = in.read();
> int ch2 = in.read();
> int ch3 = in.read();
> int ch4 = in.read();
> {code}
> To confirm, an strace of the Yarn container shows
> {code}
> 26690 read(154, "B", 1) = 1
> 26690 read(154, "\250", 1) = 1
> 26690 read(154, ".", 1) = 1
> 26690 read(154, "\24", 1) = 1
> {code}
> To emulate this without the entirety of Hive code, I have written a simpler
> test app
> https://github.com/t3rmin4t0r/shortcircuit-reader
> The jar will read a file in -bs sized buffers. Running it with 1 byte
> blocks gives similar results to the Hive test run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4714) Support logging short messages in Namenode IPC server for configurable list of exception classes.
Kihwal Lee created HDFS-4714: Summary: Support logging short messages in Namenode IPC server for configurable list of exception classes. Key: HDFS-4714 URL: https://issues.apache.org/jira/browse/HDFS-4714 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0, 0.23.7, 2.0.5-beta Reporter: Kihwal Lee Namenode can slow down significantly if a rogue client/job issues massive number of requests that will fail. E.g. permission denied, quota overage, etc. The major contributing factor in slow down is the long namenode log message, which includes full stack trace. Previously similar issues involving safe mode and standby node have been addressed and we can extend it to suppress logging stack traces for configured list of exception classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4710) Turning off HDFS short-circuit checksums unexpectedly slows down Hive
[
https://issues.apache.org/jira/browse/HDFS-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635501#comment-13635501
]
Gopal V commented on HDFS-4710:
---
It is not really a one-size-fits-all fix, I guess.
For large reads, a direct read would be the right option to avoid having
another copy.
But for small repeated reads, it does make sense to bounce buffer.size chunks
through a buffer to avoid syscall overhead.
> Turning off HDFS short-circuit checksums unexpectedly slows down Hive
> -
>
> Key: HDFS-4710
> URL: https://issues.apache.org/jira/browse/HDFS-4710
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.0.4-alpha
> Environment: Centos (EC2) + short-circuit reads on
>Reporter: Gopal V
>Priority: Minor
> Labels: perfomance
>
> When short-circuit reads are on, HDFS client slows down when checksums are
> turned off.
> With checksums on, the query takes 45.341 seconds and with it turned off, it
> takes 56.345 seconds. This is slower than the speeds observed when
> short-circuiting is turned off.
> The issue seems to be that FSDataInputStream.readByte() calls are directly
> transferred to the disk fd when the checksums are turned off.
> Even though all the columns are integers, the data being read will be read
> via DataInputStream which does
> {code}
> public final int readInt() throws IOException {
> int ch1 = in.read();
> int ch2 = in.read();
> int ch3 = in.read();
> int ch4 = in.read();
> {code}
> To confirm, an strace of the Yarn container shows
> {code}
> 26690 read(154, "B", 1) = 1
> 26690 read(154, "\250", 1) = 1
> 26690 read(154, ".", 1) = 1
> 26690 read(154, "\24", 1) = 1
> {code}
> To emulate this without the entirety of Hive code, I have written a simpler
> test app
> https://github.com/t3rmin4t0r/shortcircuit-reader
> The jar will read a file in -bs sized buffers. Running it with 1 byte
> blocks gives similar results to the Hive test run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4712) New libhdfs method hdfsGetDataNodes
[
https://issues.apache.org/jira/browse/HDFS-4712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635506#comment-13635506
]
Colin Patrick McCabe commented on HDFS-4712:
Hi Andrea,
The procedure for creating C patches is much the same as that for Java. Post
your patch here in the form of a diff, get Jenkins results and feedback, and
then get a +1 from a committer.
It sounds like what you want is some way to expose
{{NameNodeRpcServer#getDatanodeReport}}, which ultimately calls
{{FSNamesystem#datanodeReport}}.
If you return dynamically allocated memory, please be sure to also offer a
function in libhdfs to free that memory. You should not assume that the
application and the library use the same memory allocator.
Also, as M. C. Srivas commented, please be aware that the list of DataNodes can
change at any point. It would help if you would describe what you're trying to
do.
> New libhdfs method hdfsGetDataNodes
> ---
>
> Key: HDFS-4712
> URL: https://issues.apache.org/jira/browse/HDFS-4712
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: libhdfs
>Reporter: andrea manzi
>
> we have implemented a possible extension to libhdfs to retrieve information
> about the available datanodes ( there was a mail on the hadoop-hdsf-dev
> mailing list initially abut this :
> http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201204.mbox/%3CCANhO-
> s0mvororrxpjnjbql6brkj4c7l+u816xkdc+2r0whj...@mail.gmail.com%3E)
> i would like to know how to proceed to create a patch, cause on the wiki
> http://wiki.apache.org/hadoop/HowToContribute i can see info about JAVA
> patches but nothing related to extensions in C.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4711) Can not change replication factor of file during moving or deleting.
[
https://issues.apache.org/jira/browse/HDFS-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635513#comment-13635513
]
Colin Patrick McCabe commented on HDFS-4711:
What's wrong with using {{hadoop fs -setrep}}?
The move command is not supposed to alter replication status...
> Can not change replication factor of file during moving or deleting.
>
>
> Key: HDFS-4711
> URL: https://issues.apache.org/jira/browse/HDFS-4711
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.0.0-alpha
>Reporter: Vladimir Barinov
>Priority: Minor
>
> I don't know is it a feature or a bug.
> According to hdfs dfs -help we can use key -D to set specific options for
> action;
> When we copying or uploading file to hdfs we can override replication factor
> with -D dfs.replication=N. That's works well.
> But it doesn't work for moving or removing(to trash) file.
> Steps to reproduce:
> Uploading file
> hdfs dfs -put somefile /tmp/somefile
> Copying with changing replication:
> hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2
> hadoop version:
> Hadoop 2.0.0-cdh4.1.2
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4710) Turning off HDFS short-circuit checksums unexpectedly slows down Hive
[
https://issues.apache.org/jira/browse/HDFS-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635514#comment-13635514
]
Colin Patrick McCabe commented on HDFS-4710:
If we honor {{dfs.client.read.shortcircuit.buffer.size}}, the client can decide
whether to use a bounce buffer or not, and if so, how big.
> Turning off HDFS short-circuit checksums unexpectedly slows down Hive
> -
>
> Key: HDFS-4710
> URL: https://issues.apache.org/jira/browse/HDFS-4710
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 2.0.4-alpha
> Environment: Centos (EC2) + short-circuit reads on
>Reporter: Gopal V
>Priority: Minor
> Labels: perfomance
>
> When short-circuit reads are on, HDFS client slows down when checksums are
> turned off.
> With checksums on, the query takes 45.341 seconds and with it turned off, it
> takes 56.345 seconds. This is slower than the speeds observed when
> short-circuiting is turned off.
> The issue seems to be that FSDataInputStream.readByte() calls are directly
> transferred to the disk fd when the checksums are turned off.
> Even though all the columns are integers, the data being read will be read
> via DataInputStream which does
> {code}
> public final int readInt() throws IOException {
> int ch1 = in.read();
> int ch2 = in.read();
> int ch3 = in.read();
> int ch4 = in.read();
> {code}
> To confirm, an strace of the Yarn container shows
> {code}
> 26690 read(154, "B", 1) = 1
> 26690 read(154, "\250", 1) = 1
> 26690 read(154, ".", 1) = 1
> 26690 read(154, "\24", 1) = 1
> {code}
> To emulate this without the entirety of Hive code, I have written a simpler
> test app
> https://github.com/t3rmin4t0r/shortcircuit-reader
> The jar will read a file in -bs sized buffers. Running it with 1 byte
> blocks gives similar results to the Hive test run.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4711) Can not change replication factor of file during moving or deleting.
[ https://issues.apache.org/jira/browse/HDFS-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635530#comment-13635530 ] Vladimir Barinov commented on HDFS-4711: -setrep works well but it was confusing when move does not supported it and usage tells nothing about it. Probably I should change ticket to "new feature" or "improvement"? > Can not change replication factor of file during moving or deleting. > > > Key: HDFS-4711 > URL: https://issues.apache.org/jira/browse/HDFS-4711 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Vladimir Barinov >Priority: Minor > > I don't know is it a feature or a bug. > According to hdfs dfs -help we can use key -D to set specific options for > action; > When we copying or uploading file to hdfs we can override replication factor > with -D dfs.replication=N. That's works well. > But it doesn't work for moving or removing(to trash) file. > Steps to reproduce: > Uploading file > hdfs dfs -put somefile /tmp/somefile > Copying with changing replication: > hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 > hadoop version: > Hadoop 2.0.0-cdh4.1.2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HDFS-4549: -- Attachment: HDFS-4549.2.patch I've backported HDFS-3577, HDFS-3318, and HDFS-3788 as best as possible. However, I'm seeing performance problems above 2GB. It looks like when the MessageBodyWriter reports a length greater than 2GB, Jersey is reverting to chunked transfer, causing the previously seen performance problems with Jetty. I tested this on a 2.0.3 deployment and got the same results (chunked transfer above a certain size). [~szetszwo], can you expand on the motivation for using MessageBodyWriter and OpenEntity instead of manually setting the header values? Would it be problematic to make all branches use the method of my first patch? It seems that Jersey is taking the MessageBodyWriter information only as a suggestion. > WebHDFS hits a Jetty performance issue > -- > > Key: HDFS-4549 > URL: https://issues.apache.org/jira/browse/HDFS-4549 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.1.2 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch > > > WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked > transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not > observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HDFS-4549: -- Attachment: example4549.txt > WebHDFS hits a Jetty performance issue > -- > > Key: HDFS-4549 > URL: https://issues.apache.org/jira/browse/HDFS-4549 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.1.2 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch > > > WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked > transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not > observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-4711) Can not change replication factor of file during moving or deleting.
[ https://issues.apache.org/jira/browse/HDFS-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe reassigned HDFS-4711: -- Assignee: Colin Patrick McCabe > Can not change replication factor of file during moving or deleting. > > > Key: HDFS-4711 > URL: https://issues.apache.org/jira/browse/HDFS-4711 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Vladimir Barinov >Assignee: Colin Patrick McCabe >Priority: Minor > > I don't know is it a feature or a bug. > According to hdfs dfs -help we can use key -D to set specific options for > action; > When we copying or uploading file to hdfs we can override replication factor > with -D dfs.replication=N. That's works well. > But it doesn't work for moving or removing(to trash) file. > Steps to reproduce: > Uploading file > hdfs dfs -put somefile /tmp/somefile > Copying with changing replication: > hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 > hadoop version: > Hadoop 2.0.0-cdh4.1.2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635618#comment-13635618 ] Alejandro Abdelnur commented on HDFS-4549: -- I think we should leave Jersey to decide what to use, chunk encoding or not. If we always force the content-length, we can run into issues in clients (I believe Java HTTP does -or used to do this-) that try to allocate a buffer in memory for the full content-length. By using chunk encoding, you are forcing the client to fallback on partial-caching/flushing. Is the Jetty version use by Hadoop 1 that has issues with chunk encoding? Also, if we don't see the problem in trunk/branch-2, why change it there? not broken, don't fix it. > WebHDFS hits a Jetty performance issue > -- > > Key: HDFS-4549 > URL: https://issues.apache.org/jira/browse/HDFS-4549 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.1.2 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch > > > WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked > transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not > observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4711) Can not change replication factor of file during moving or deleting.
[ https://issues.apache.org/jira/browse/HDFS-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635619#comment-13635619 ] Colin Patrick McCabe commented on HDFS-4711: I don't think this is a good idea. Consider moving a large directory with hundreds of thousands of files. Are you going to check each file to see if its replication is equal to the current dfs.replication setting? And if it's not, are you going to do setrep on it? This will also create a lot of problems for current users, since they'll find that mv unexpectedly changes their replication settings. Remember that dfs.replication is usually set to a value due to the configuration XML files, even if the user didn't set it explicitly. > Can not change replication factor of file during moving or deleting. > > > Key: HDFS-4711 > URL: https://issues.apache.org/jira/browse/HDFS-4711 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Vladimir Barinov >Assignee: Colin Patrick McCabe >Priority: Minor > > I don't know is it a feature or a bug. > According to hdfs dfs -help we can use key -D to set specific options for > action; > When we copying or uploading file to hdfs we can override replication factor > with -D dfs.replication=N. That's works well. > But it doesn't work for moving or removing(to trash) file. > Steps to reproduce: > Uploading file > hdfs dfs -put somefile /tmp/somefile > Copying with changing replication: > hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 > hadoop version: > Hadoop 2.0.0-cdh4.1.2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635637#comment-13635637 ] Tsz Wo (Nicholas), SZE commented on HDFS-4549: -- @Mark, the motivation for using MessageBodyWriter is that manually setting the Content-Length header may cause problems such as Jersey may add the another Content-Length header. It seems that MessageBodyWriter is a standard way in Jersey. BTW, do you see any performance problem in trunk/2.0.3? > WebHDFS hits a Jetty performance issue > -- > > Key: HDFS-4549 > URL: https://issues.apache.org/jira/browse/HDFS-4549 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.1.2 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch > > > WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked > transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not > observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4711) Can not change replication factor of file during moving or deleting.
[ https://issues.apache.org/jira/browse/HDFS-4711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Barinov resolved HDFS-4711. Resolution: Won't Fix > Can not change replication factor of file during moving or deleting. > > > Key: HDFS-4711 > URL: https://issues.apache.org/jira/browse/HDFS-4711 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0-alpha >Reporter: Vladimir Barinov >Assignee: Colin Patrick McCabe >Priority: Minor > > I don't know is it a feature or a bug. > According to hdfs dfs -help we can use key -D to set specific options for > action; > When we copying or uploading file to hdfs we can override replication factor > with -D dfs.replication=N. That's works well. > But it doesn't work for moving or removing(to trash) file. > Steps to reproduce: > Uploading file > hdfs dfs -put somefile /tmp/somefile > Copying with changing replication: > hdfs dfs -D dfs.replication=1 -mv /tmp/somefile /tmp/somefile2 > hadoop version: > Hadoop 2.0.0-cdh4.1.2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4504) DFSOutputStream#close doesn't always release resources (such as leases)
[
https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635672#comment-13635672
]
Colin Patrick McCabe commented on HDFS-4504:
bq. RPC retry should be done in RPC level. Please don't add retry to DFSClient.
OK. I will remove this in the next patch.
I suppose lease recovery will take care of un-completed files eventually.
> DFSOutputStream#close doesn't always release resources (such as leases)
> ---
>
> Key: HDFS-4504
> URL: https://issues.apache.org/jira/browse/HDFS-4504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-4504.001.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One
> example is if there is a pipeline error and then pipeline recovery fails.
> Unfortunately, in this case, some of the resources used by the
> {{DFSOutputStream}} are leaked. One particularly important resource is file
> leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many
> blocks to a file, but then fail to close it. Unfortunately, the
> {{LeaseRenewerThread}} inside the client will continue to renew the lease for
> the "undead" file. Future attempts to close the file will just rethrow the
> previous exception, and no progress can be made by the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[
https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635687#comment-13635687
]
Mark Wagner commented on HDFS-4549:
---
bq. I think we should leave Jersey to decide what to use, chunk encoding or not.
I agree that that would usually be best, however this is causing significant
performance problems, so I think it needs to be handled somehow. I just took a
look at the memory usage when pulling a 1GB file (which does use
content-length) and didn't see any memory usage increase. I'll look into this
further though.
{quote}
Is the Jetty version use by Hadoop 1 that has issues with chunk encoding?
Also, if we don't see the problem in trunk/branch-2, why change it there? not
broken, don't fix it.
{quote}
The issue has been observed in 6.1.26, which is used by branch-1, branch-2, and
trunk. This problem was first seen (chunked encoding being used, leading to
performance loss) in branch-1 (1.0.4 to be precise) for files of any size. I
didn't see it on branch-2/trunk because I didn't test larger files. When I was
backporting though, I noticed that it was still slow above 2GB, and it turns
out the same is true for branch-2/trunk (I've attached logs for a demonstration
of this issue using 2.0.3-alpha).
> WebHDFS hits a Jetty performance issue
> --
>
> Key: HDFS-4549
> URL: https://issues.apache.org/jira/browse/HDFS-4549
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 1.1.2
>Reporter: Mark Wagner
>Assignee: Mark Wagner
> Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch
>
>
> WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked
> transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not
> observed this on trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635695#comment-13635695 ] Mark Wagner commented on HDFS-4549: --- [~szetszwo], that makes sense. Yes, I see performance issues above 2GB on 2.0.3. > WebHDFS hits a Jetty performance issue > -- > > Key: HDFS-4549 > URL: https://issues.apache.org/jira/browse/HDFS-4549 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.1.2 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch > > > WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked > transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not > observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635710#comment-13635710 ] Tsz Wo (Nicholas), SZE commented on HDFS-4549: -- @Mark, Since there are also performance problem in trunk, let's first backport the patch to branch-1 and then fix all the branch the same way. If manually setting content-length works well, we may set it when the file size is >= 2GB. Let me create another JIRA for backporting and continue the performance improvement here. > WebHDFS hits a Jetty performance issue > -- > > Key: HDFS-4549 > URL: https://issues.apache.org/jira/browse/HDFS-4549 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.1.2 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch > > > WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked > transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not > observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4715) Backport HDFS-3577 and other related WebHDFS to branch-1
Tsz Wo (Nicholas), SZE created HDFS-4715: Summary: Backport HDFS-3577 and other related WebHDFS to branch-1 Key: HDFS-4715 URL: https://issues.apache.org/jira/browse/HDFS-4715 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Reporter: Tsz Wo (Nicholas), SZE Assignee: Mark Wagner The related JIRAs are HDFS-3577, HDFS-3318, and HDFS-3788. Backporting them can fix some WebHDFS performance issues in branch-1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4715) Backport HDFS-3577 and other related WebHDFS issue to branch-1
[ https://issues.apache.org/jira/browse/HDFS-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4715: - Summary: Backport HDFS-3577 and other related WebHDFS issue to branch-1 (was: Backport HDFS-3577 and other related WebHDFS to branch-1) > Backport HDFS-3577 and other related WebHDFS issue to branch-1 > -- > > Key: HDFS-4715 > URL: https://issues.apache.org/jira/browse/HDFS-4715 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Mark Wagner > > The related JIRAs are HDFS-3577, HDFS-3318, and HDFS-3788. Backporting them > can fix some WebHDFS performance issues in branch-1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4549) WebHDFS hits a Jetty performance issue
[ https://issues.apache.org/jira/browse/HDFS-4549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635725#comment-13635725 ] Alejandro Abdelnur commented on HDFS-4549: -- Before changing to remove chunk encoding, can we try the following? http://stackoverflow.com/questions/9031311/slow-transfers-in-jetty-with-chunked-transfer-encoding-at-certain-buffer-size > WebHDFS hits a Jetty performance issue > -- > > Key: HDFS-4549 > URL: https://issues.apache.org/jira/browse/HDFS-4549 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 1.1.2 >Reporter: Mark Wagner >Assignee: Mark Wagner > Attachments: example4549.txt, HDFS-4549.1.patch, HDFS-4549.2.patch > > > WebHDFS on branch-1 is hitting a Jetty issue for me when it does chunked > transfers. This is the same Jetty issue as MAPREDUCE-4399. I have not > observed this on trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4504) DFSOutputStream#close doesn't always release resources (such as leases)
[
https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-4504:
---
Attachment: HDFS-4504.002.patch
* remove manual retry (allow RPC retry to do this job)
* remove debug messages
* TestBlockRecovery: set lower RPC timeout so that we give up on close prior to
the test timeout.
* If both {{completeFile}} and the block data flush fail, throw a
{{MultipleIOException}} with information about both failures.
> DFSOutputStream#close doesn't always release resources (such as leases)
> ---
>
> Key: HDFS-4504
> URL: https://issues.apache.org/jira/browse/HDFS-4504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One
> example is if there is a pipeline error and then pipeline recovery fails.
> Unfortunately, in this case, some of the resources used by the
> {{DFSOutputStream}} are leaked. One particularly important resource is file
> leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many
> blocks to a file, but then fail to close it. Unfortunately, the
> {{LeaseRenewerThread}} inside the client will continue to renew the lease for
> the "undead" file. Future attempts to close the file will just rethrow the
> previous exception, and no progress can be made by the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4715) Backport HDFS-3577 and other related WebHDFS issue to branch-1
[
https://issues.apache.org/jira/browse/HDFS-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635733#comment-13635733
]
Tsz Wo (Nicholas), SZE commented on HDFS-4715:
--
The patch (HDFS-4549.2.patch) looks different from the current trunk code. For
backporting, we would try to backport the code exactly so that it is easier to
maintain the code. Could you make the backport patch the same as the original
patches? Some other comments:
- The import org.apache.log4j.helpers.BoundedFIFO is not used.
- the parameter in update(..) should be the read length but not the read value.
{code}
+return update(getInputStream().read())
{code}
- the filelength in udpate(..) may be null
> Backport HDFS-3577 and other related WebHDFS issue to branch-1
> --
>
> Key: HDFS-4715
> URL: https://issues.apache.org/jira/browse/HDFS-4715
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Mark Wagner
>
> The related JIRAs are HDFS-3577, HDFS-3318, and HDFS-3788. Backporting them
> can fix some WebHDFS performance issues in branch-1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4716) TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
Chris Nauroth created HDFS-4716:
---
Summary:
TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
Key: HDFS-4716
URL: https://issues.apache.org/jira/browse/HDFS-4716
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode, test
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
{{TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs}} fails on
Windows due to an incorrectly initialized name dir in the {{Configuration}}
used by the test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4717) Change the parameter type of the snapshot methods in HdfsAdmin to Path
Tsz Wo (Nicholas), SZE created HDFS-4717: Summary: Change the parameter type of the snapshot methods in HdfsAdmin to Path Key: HDFS-4717 URL: https://issues.apache.org/jira/browse/HDFS-4717 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE In HdfsAdmin, the path parameter type in allowSnapshot(String path) and allowSnapshot(String path) should be Path but not String. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4717) Change the parameter type of the snapshot methods in HdfsAdmin to Path
[ https://issues.apache.org/jira/browse/HDFS-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4717: - Description: In HdfsAdmin, the path parameter type in allowSnapshot(String path) and disallowSnapshot(String path) should be Path but not String. (was: In HdfsAdmin, the path parameter type in allowSnapshot(String path) and allowSnapshot(String path) should be Path but not String.) > Change the parameter type of the snapshot methods in HdfsAdmin to Path > -- > > Key: HDFS-4717 > URL: https://issues.apache.org/jira/browse/HDFS-4717 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > > In HdfsAdmin, the path parameter type in allowSnapshot(String path) and > disallowSnapshot(String path) should be Path but not String. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4716) TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-4716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4716:
Attachment: HDFS-4716.1.patch
The test was not setting dfs.namenode.name.dir. It was using the default. On
Windows, this would end up being an invalid URI containing '\' characters after
configuration performed property substitution, i.e. file://C:\foo\bar/dfs/name.
This patch is a one-line fix to initialize dfs.namenode.name.dir to a correct
value under test.build.data.
> TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
> -
>
> Key: HDFS-4716
> URL: https://issues.apache.org/jira/browse/HDFS-4716
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, test
>Affects Versions: 3.0.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4716.1.patch
>
>
> {{TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs}} fails on
> Windows due to an incorrectly initialized name dir in the {{Configuration}}
> used by the test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4716) TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-4716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4716:
Status: Patch Available (was: Open)
> TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
> -
>
> Key: HDFS-4716
> URL: https://issues.apache.org/jira/browse/HDFS-4716
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, test
>Affects Versions: 3.0.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4716.1.patch
>
>
> {{TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs}} fails on
> Windows due to an incorrectly initialized name dir in the {{Configuration}}
> used by the test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4716) TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-4716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4716:
Status: Open (was: Patch Available)
> TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
> -
>
> Key: HDFS-4716
> URL: https://issues.apache.org/jira/browse/HDFS-4716
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, test
>Affects Versions: 3.0.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4716.1.patch
>
>
> {{TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs}} fails on
> Windows due to an incorrectly initialized name dir in the {{Configuration}}
> used by the test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4705) TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
[
https://issues.apache.org/jira/browse/HDFS-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4705:
Attachment: HDFS-4705.1.patch
Hi, Ivan. I had a patch in progress for this one. I accidentally filed
HDFS-4716, but I just resolved it as duplicate. I'm attaching my current
patch, which is a one-liner fix for just {{TestAllowFormat}} (not the other
ones that you mentioned in the comments).
> TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
> -
>
> Key: HDFS-4705
> URL: https://issues.apache.org/jira/browse/HDFS-4705
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
>Priority: Minor
> Attachments: HDFS-4705.1.patch
>
>
> Test fails on Windows with the below exception:
> {code}
> testFormatShouldBeIgnoredForNonFileBasedDirs(org.apache.hadoop.hdfs.server.namenode.TestAllowFormat)
> Time elapsed: 49 sec <<< ERROR!
> java.io.IOException: No image directories available!
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:905)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:758)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:259)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAllowFormat.testFormatShouldBeIgnoredForNonFileBasedDirs(TestAllowFormat.java:181)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4705) TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
[
https://issues.apache.org/jira/browse/HDFS-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-4705:
Status: Patch Available (was: Open)
> TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
> -
>
> Key: HDFS-4705
> URL: https://issues.apache.org/jira/browse/HDFS-4705
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
>Priority: Minor
> Attachments: HDFS-4705.1.patch
>
>
> Test fails on Windows with the below exception:
> {code}
> testFormatShouldBeIgnoredForNonFileBasedDirs(org.apache.hadoop.hdfs.server.namenode.TestAllowFormat)
> Time elapsed: 49 sec <<< ERROR!
> java.io.IOException: No image directories available!
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:905)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:758)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:259)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAllowFormat.testFormatShouldBeIgnoredForNonFileBasedDirs(TestAllowFormat.java:181)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4713) Wrong server principal is used for rpc calls to namenode if HA is enabled
[ https://issues.apache.org/jira/browse/HDFS-4713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635767#comment-13635767 ] Kihwal Lee commented on HDFS-4713: -- It seems client side is okay. > Wrong server principal is used for rpc calls to namenode if HA is enabled > - > > Key: HDFS-4713 > URL: https://issues.apache.org/jira/browse/HDFS-4713 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, namenode >Affects Versions: 2.0.5-beta >Reporter: Kihwal Lee >Priority: Blocker > > When various components are connecting to a namenode in a HA-enabled > environment, a wrong server principal may be picked up. This result in SASL > failure, since the client-side used a wrong service ticket for the connection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4716) TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-4716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HDFS-4716.
-
Resolution: Duplicate
> TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
> -
>
> Key: HDFS-4716
> URL: https://issues.apache.org/jira/browse/HDFS-4716
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, test
>Affects Versions: 3.0.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4716.1.patch
>
>
> {{TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs}} fails on
> Windows due to an incorrectly initialized name dir in the {{Configuration}}
> used by the test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4705) TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
[
https://issues.apache.org/jira/browse/HDFS-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635778#comment-13635778
]
Chris Nauroth commented on HDFS-4705:
-
I can see 2 different approaches to fixing these tests:
# Fix each individual test to initialize dfs.namenode.name.dir properly.
# Add special case logic in {{Configuration#substituteVars}} saying that if the
value starts with "file:", then pass any substitution values through
{{File#toURI}} and {{URI#getPath}}. On Windows, this would have the effect of
converting '\' to '/' and yield a valid URI.
My initial patch takes the first approach. I expect I can quickly prep a patch
for the second approach too so we can compare and contrast.
BTW, I apologize if I stepped on your toes here. Somehow, I missed that you
had already filed this jira.
> TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
> -
>
> Key: HDFS-4705
> URL: https://issues.apache.org/jira/browse/HDFS-4705
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
>Priority: Minor
> Attachments: HDFS-4705.1.patch
>
>
> Test fails on Windows with the below exception:
> {code}
> testFormatShouldBeIgnoredForNonFileBasedDirs(org.apache.hadoop.hdfs.server.namenode.TestAllowFormat)
> Time elapsed: 49 sec <<< ERROR!
> java.io.IOException: No image directories available!
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:905)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:758)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:259)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAllowFormat.testFormatShouldBeIgnoredForNonFileBasedDirs(TestAllowFormat.java:181)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4715) Backport HDFS-3577 and other related WebHDFS issue to branch-1
[ https://issues.apache.org/jira/browse/HDFS-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HDFS-4715: -- Attachment: HDFS-4751.1.patch Addressed your points, and got things as close to trunk as possible. > Backport HDFS-3577 and other related WebHDFS issue to branch-1 > -- > > Key: HDFS-4715 > URL: https://issues.apache.org/jira/browse/HDFS-4715 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Mark Wagner > Attachments: HDFS-4751.1.patch > > > The related JIRAs are HDFS-3577, HDFS-3318, and HDFS-3788. Backporting them > can fix some WebHDFS performance issues in branch-1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4715) Backport HDFS-3577 and other related WebHDFS issue to branch-1
[ https://issues.apache.org/jira/browse/HDFS-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Wagner updated HDFS-4715: -- Status: Patch Available (was: Open) > Backport HDFS-3577 and other related WebHDFS issue to branch-1 > -- > > Key: HDFS-4715 > URL: https://issues.apache.org/jira/browse/HDFS-4715 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Mark Wagner > Attachments: HDFS-4751.1.patch > > > The related JIRAs are HDFS-3577, HDFS-3318, and HDFS-3788. Backporting them > can fix some WebHDFS performance issues in branch-1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4650) Add unit tests for rename with snapshots
[ https://issues.apache.org/jira/browse/HDFS-4650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-4650: Assignee: Jing Zhao (was: Arpit Agarwal) > Add unit tests for rename with snapshots > > > Key: HDFS-4650 > URL: https://issues.apache.org/jira/browse/HDFS-4650 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > > Add more unit tests and update current unit tests to cover different cases > for rename with existence of snapshottable directories and snapshots. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4686) Fix quota computation for rename with snapshots
[ https://issues.apache.org/jira/browse/HDFS-4686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4686: Attachment: HDFS-4686.001.patch Update the patch. Now the patch can pass all the unit tests with quota check enabled. Things to do next: 1. Fix INode#computeContentSummary 2. Add javadoc and clean the code > Fix quota computation for rename with snapshots > --- > > Key: HDFS-4686 > URL: https://issues.apache.org/jira/browse/HDFS-4686 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Jing Zhao > Attachments: HDFS-4686.001.patch, HDFS-4686.001.patch > > > Currently after a rename operation within/from a snapshottable directory, a > reference node is created in both src and dst subtree, pointing to the > original renamed inode. With this change the original quota computation may > count the quota usage of the renamed subtree multiple times. This jira tries > to fix this issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4504) DFSOutputStream#close doesn't always release resources (such as leases)
[
https://issues.apache.org/jira/browse/HDFS-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635834#comment-13635834
]
Hadoop QA commented on HDFS-4504:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579431/HDFS-4504.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4275//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4275//console
This message is automatically generated.
> DFSOutputStream#close doesn't always release resources (such as leases)
> ---
>
> Key: HDFS-4504
> URL: https://issues.apache.org/jira/browse/HDFS-4504
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
> Attachments: HDFS-4504.001.patch, HDFS-4504.002.patch
>
>
> {{DFSOutputStream#close}} can throw an {{IOException}} in some cases. One
> example is if there is a pipeline error and then pipeline recovery fails.
> Unfortunately, in this case, some of the resources used by the
> {{DFSOutputStream}} are leaked. One particularly important resource is file
> leases.
> So it's possible for a long-lived HDFS client, such as Flume, to write many
> blocks to a file, but then fail to close it. Unfortunately, the
> {{LeaseRenewerThread}} inside the client will continue to renew the lease for
> the "undead" file. Future attempts to close the file will just rethrow the
> previous exception, and no progress can be made by the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4715) Backport HDFS-3577 and other related WebHDFS issue to branch-1
[
https://issues.apache.org/jira/browse/HDFS-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635839#comment-13635839
]
Hadoop QA commented on HDFS-4715:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579452/HDFS-4751.1.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4278//console
This message is automatically generated.
> Backport HDFS-3577 and other related WebHDFS issue to branch-1
> --
>
> Key: HDFS-4715
> URL: https://issues.apache.org/jira/browse/HDFS-4715
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Mark Wagner
> Attachments: HDFS-4751.1.patch
>
>
> The related JIRAs are HDFS-3577, HDFS-3318, and HDFS-3788. Backporting them
> can fix some WebHDFS performance issues in branch-1.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4705) TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
[
https://issues.apache.org/jira/browse/HDFS-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635857#comment-13635857
]
Ivan Mitic commented on HDFS-4705:
--
Hey Chris, no worries at all, and thanks for attaching the patch!
I started with a similar approach to yours and then saw the other test
failures, so I thought it is worth to spent some time seeing if it makes sense
to fix them all at ones.
I was thinking along the lines of changing {{Util#fileAsURI}} such that it
converts the given File to Path and then from Path to the URI. However, on top
of this we'd also have to change the default value for
{{dfs.namenode.name.dir}} as "file://$hadoop.tmp.dir/dfs/name" is actually not
a valid local URI (there should 3 forward slashes after "file:", IOW:
"file:///$hadoop.tmp.dir/dfs/name"). I haven't tested this out, so it might be
that there are some problems here.
You're welcome to take this up if you want :)
> TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
> -
>
> Key: HDFS-4705
> URL: https://issues.apache.org/jira/browse/HDFS-4705
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
>Priority: Minor
> Attachments: HDFS-4705.1.patch
>
>
> Test fails on Windows with the below exception:
> {code}
> testFormatShouldBeIgnoredForNonFileBasedDirs(org.apache.hadoop.hdfs.server.namenode.TestAllowFormat)
> Time elapsed: 49 sec <<< ERROR!
> java.io.IOException: No image directories available!
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:905)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:758)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:259)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAllowFormat.testFormatShouldBeIgnoredForNonFileBasedDirs(TestAllowFormat.java:181)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4705) TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
[
https://issues.apache.org/jira/browse/HDFS-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635859#comment-13635859
]
Ivan Mitic commented on HDFS-4705:
--
PS. I think the approach from the current patch is actually a fine way to
address the problem, I just thought it is worth to check if there are better
ways.
> TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
> -
>
> Key: HDFS-4705
> URL: https://issues.apache.org/jira/browse/HDFS-4705
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
>Priority: Minor
> Attachments: HDFS-4705.1.patch
>
>
> Test fails on Windows with the below exception:
> {code}
> testFormatShouldBeIgnoredForNonFileBasedDirs(org.apache.hadoop.hdfs.server.namenode.TestAllowFormat)
> Time elapsed: 49 sec <<< ERROR!
> java.io.IOException: No image directories available!
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:905)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:758)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:259)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAllowFormat.testFormatShouldBeIgnoredForNonFileBasedDirs(TestAllowFormat.java:181)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4705) TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
[
https://issues.apache.org/jira/browse/HDFS-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635865#comment-13635865
]
Ivan Mitic commented on HDFS-4705:
--
bq. However, on top of this we'd also have to change the default value for
dfs.namenode.name.dir as "file://$hadoop.tmp.dir/dfs/name" is actually not a
valid local URI (there should 3 forward slashes after "file:", IOW:
"file:///$hadoop.tmp.dir/dfs/name").
Correcting myself here. The above URI is actually valid since $hadoop.tmp.dir
contains one forward slash at the beginning. The rest of what I said below
should still make sense.
> TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
> -
>
> Key: HDFS-4705
> URL: https://issues.apache.org/jira/browse/HDFS-4705
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
>Priority: Minor
> Attachments: HDFS-4705.1.patch
>
>
> Test fails on Windows with the below exception:
> {code}
> testFormatShouldBeIgnoredForNonFileBasedDirs(org.apache.hadoop.hdfs.server.namenode.TestAllowFormat)
> Time elapsed: 49 sec <<< ERROR!
> java.io.IOException: No image directories available!
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:905)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:758)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:259)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAllowFormat.testFormatShouldBeIgnoredForNonFileBasedDirs(TestAllowFormat.java:181)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4716) TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
[
https://issues.apache.org/jira/browse/HDFS-4716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635869#comment-13635869
]
Hadoop QA commented on HDFS-4716:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579438/HDFS-4716.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4276//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4276//console
This message is automatically generated.
> TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs fails on Windows
> -
>
> Key: HDFS-4716
> URL: https://issues.apache.org/jira/browse/HDFS-4716
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, test
>Affects Versions: 3.0.0
>Reporter: Chris Nauroth
>Assignee: Chris Nauroth
> Attachments: HDFS-4716.1.patch
>
>
> {{TestAllowFormat#testFormatShouldBeIgnoredForNonFileBasedDirs}} fails on
> Windows due to an incorrectly initialized name dir in the {{Configuration}}
> used by the test.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4705) TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
[
https://issues.apache.org/jira/browse/HDFS-4705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635876#comment-13635876
]
Hadoop QA commented on HDFS-4705:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12579441/HDFS-4705.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/4277//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4277//console
This message is automatically generated.
> TestAllowFormat fails on Windows because of invalid dfs.namenode.name.dir
> -
>
> Key: HDFS-4705
> URL: https://issues.apache.org/jira/browse/HDFS-4705
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Ivan Mitic
>Assignee: Ivan Mitic
>Priority: Minor
> Attachments: HDFS-4705.1.patch
>
>
> Test fails on Windows with the below exception:
> {code}
> testFormatShouldBeIgnoredForNonFileBasedDirs(org.apache.hadoop.hdfs.server.namenode.TestAllowFormat)
> Time elapsed: 49 sec <<< ERROR!
> java.io.IOException: No image directories available!
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:912)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImageInAllDirs(FSImage.java:905)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:151)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:758)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:259)
> at
> org.apache.hadoop.hdfs.server.namenode.TestAllowFormat.testFormatShouldBeIgnoredForNonFileBasedDirs(TestAllowFormat.java:181)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4434) Provide a mapping from INodeId to INode
[ https://issues.apache.org/jira/browse/HDFS-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635882#comment-13635882 ] Hudson commented on HDFS-4434: -- Integrated in Hadoop-trunk-Commit #3632 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/3632/]) HDFS-4434. Provide a mapping from INodeId to INode. Contributed by Suresh Srinivas. (Revision 1469644) Result = SUCCESS suresh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1469644 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlocksMap.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormat.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INodeId.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java > Provide a mapping from INodeId to INode > --- > > Key: HDFS-4434 > URL: https://issues.apache.org/jira/browse/HDFS-4434 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Suresh Srinivas > Attachments: HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch > > > This JIRA is to provide a way to access the INode via its id. The proposed > solution is to have an in-memory mapping from INodeId to INode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4434) Provide a mapping from INodeId to INode
[ https://issues.apache.org/jira/browse/HDFS-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-4434: -- Resolution: Fixed Fix Version/s: 3.0.0 Release Note: This change adds support for referencing files and directories based on fileID/inodeID using a path /.reserved/.inodes/. With this change creating a file or directory /.reserved is not longer allowed. Before upgrading to a release with this change, files /.reserved needs to be renamed to another name. Hadoop Flags: Incompatible change,Reviewed Status: Resolved (was: Patch Available) > Provide a mapping from INodeId to INode > --- > > Key: HDFS-4434 > URL: https://issues.apache.org/jira/browse/HDFS-4434 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 3.0.0 >Reporter: Brandon Li >Assignee: Suresh Srinivas > Fix For: 3.0.0 > > Attachments: HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, > HDFS-4434.patch > > > This JIRA is to provide a way to access the INode via its id. The proposed > solution is to have an in-memory mapping from INodeId to INode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li resolved HDFS-4489. -- Resolution: Fixed Close this JIRA since all its sub-issues have been resolved. > Use InodeID as as an identifier of a file in HDFS protocols and APIs > > > Key: HDFS-4489 > URL: https://issues.apache.org/jira/browse/HDFS-4489 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Brandon Li >Assignee: Brandon Li > > The benefit of using InodeID to uniquely identify a file can be multiple > folds. Here are a few of them: > 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, > HDFS-4437. > 2. modification checks in tools like distcp. Since a file could have been > replaced or renamed to, the file name and size combination is no t reliable, > but the combination of file id and size is unique. > 3. id based protocol support (e.g., NFS) > 4. to make the pluggable block placement policy use fileid instead of > filename (HDFS-385). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-4717) Change the parameter type of the snapshot methods in HdfsAdmin to Path
[ https://issues.apache.org/jira/browse/HDFS-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-4717: - Attachment: h4717_20130418.patch h4717_20130418.patch: change the type. > Change the parameter type of the snapshot methods in HdfsAdmin to Path > -- > > Key: HDFS-4717 > URL: https://issues.apache.org/jira/browse/HDFS-4717 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h4717_20130418.patch > > > In HdfsAdmin, the path parameter type in allowSnapshot(String path) and > disallowSnapshot(String path) should be Path but not String. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-4717) Change the parameter type of the snapshot methods in HdfsAdmin to Path
[ https://issues.apache.org/jira/browse/HDFS-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13635933#comment-13635933 ] Jing Zhao commented on HDFS-4717: - The patch looks good to me. +1 for the patch. > Change the parameter type of the snapshot methods in HdfsAdmin to Path > -- > > Key: HDFS-4717 > URL: https://issues.apache.org/jira/browse/HDFS-4717 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h4717_20130418.patch > > > In HdfsAdmin, the path parameter type in allowSnapshot(String path) and > disallowSnapshot(String path) should be Path but not String. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4717) Change the parameter type of the snapshot methods in HdfsAdmin to Path
[ https://issues.apache.org/jira/browse/HDFS-4717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-4717. -- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) Hadoop Flags: Reviewed Thanks Jing for reviewing the patch. I have committed this. > Change the parameter type of the snapshot methods in HdfsAdmin to Path > -- > > Key: HDFS-4717 > URL: https://issues.apache.org/jira/browse/HDFS-4717 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: Snapshot (HDFS-2802) > > Attachments: h4717_20130418.patch > > > In HdfsAdmin, the path parameter type in allowSnapshot(String path) and > disallowSnapshot(String path) should be Path but not String. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
