[jira] [Updated] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1799: - Attachment: HDFS-1799-all.diff Added HDFS-1799-all.diff which is all 3 patches in one, so that Hudson can run it. I would suggest applying the patches in sequence though to keep the changes separate. > Refactor log rolling and filename management out of FSEditLog > - > > Key: HDFS-1799 > URL: https://issues.apache.org/jira/browse/HDFS-1799 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: 0001-Added-state-management-to-FSEditLog.patch, > 0002-Standardised-error-pattern.patch, > 0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch, > HDFS-1799-all.diff, hdfs-1799.txt, hdfs-1799.txt > > > This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA > focuses on pluggability, this task is simply the minimum needed for HDFS-1073: > - Refactor the filename-specific code for rolling, diverting, and reverting > log streams out of FSEditLog into a new class > - Clean up the related code in FSEditLog a bit > Notably, this JIRA is going to temporarily break the BackupNode. I plan to > circle back on the BackupNode later on this branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017341#comment-13017341 ] Hadoop QA commented on HDFS-1799: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475802/HDFS-1799-all.diff against trunk revision 1090114. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/332//console This message is automatically generated. > Refactor log rolling and filename management out of FSEditLog > - > > Key: HDFS-1799 > URL: https://issues.apache.org/jira/browse/HDFS-1799 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: 0001-Added-state-management-to-FSEditLog.patch, > 0002-Standardised-error-pattern.patch, > 0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch, > HDFS-1799-all.diff, hdfs-1799.txt, hdfs-1799.txt > > > This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA > focuses on pluggability, this task is simply the minimum needed for HDFS-1073: > - Refactor the filename-specific code for rolling, diverting, and reverting > log streams out of FSEditLog into a new class > - Clean up the related code in FSEditLog a bit > Notably, this JIRA is going to temporarily break the BackupNode. I plan to > circle back on the BackupNode later on this branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1799: - Status: Patch Available (was: Open) > Refactor log rolling and filename management out of FSEditLog > - > > Key: HDFS-1799 > URL: https://issues.apache.org/jira/browse/HDFS-1799 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: 0001-Added-state-management-to-FSEditLog.patch, > 0002-Standardised-error-pattern.patch, > 0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch, > HDFS-1799-all.diff, hdfs-1799.txt, hdfs-1799.txt > > > This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA > focuses on pluggability, this task is simply the minimum needed for HDFS-1073: > - Refactor the filename-specific code for rolling, diverting, and reverting > log streams out of FSEditLog into a new class > - Clean up the related code in FSEditLog a bit > Notably, this JIRA is going to temporarily break the BackupNode. I plan to > circle back on the BackupNode later on this branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017343#comment-13017343 ] Ivan Kelly commented on HDFS-1799: -- This is trying to build against trunk, which is strange. > Refactor log rolling and filename management out of FSEditLog > - > > Key: HDFS-1799 > URL: https://issues.apache.org/jira/browse/HDFS-1799 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: 0001-Added-state-management-to-FSEditLog.patch, > 0002-Standardised-error-pattern.patch, > 0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch, > HDFS-1799-all.diff, hdfs-1799.txt, hdfs-1799.txt > > > This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA > focuses on pluggability, this task is simply the minimum needed for HDFS-1073: > - Refactor the filename-specific code for rolling, diverting, and reverting > log streams out of FSEditLog into a new class > - Clean up the related code in FSEditLog a bit > Notably, this JIRA is going to temporarily break the BackupNode. I plan to > circle back on the BackupNode later on this branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1799) Refactor log rolling and filename management out of FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-1799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1799: - Status: Open (was: Patch Available) > Refactor log rolling and filename management out of FSEditLog > - > > Key: HDFS-1799 > URL: https://issues.apache.org/jira/browse/HDFS-1799 > Project: Hadoop HDFS > Issue Type: Sub-task >Affects Versions: Edit log branch (HDFS-1073) >Reporter: Todd Lipcon >Assignee: Todd Lipcon > Fix For: Edit log branch (HDFS-1073) > > Attachments: 0001-Added-state-management-to-FSEditLog.patch, > 0002-Standardised-error-pattern.patch, > 0003-Add-JournalFactory-and-move-divert-revert-out-of-FSE.patch, > HDFS-1799-all.diff, hdfs-1799.txt, hdfs-1799.txt > > > This is somewhat similar to HDFS-1580, but less ambitious. While that JIRA > focuses on pluggability, this task is simply the minimum needed for HDFS-1073: > - Refactor the filename-specific code for rolling, diverting, and reverting > log streams out of FSEditLog into a new class > - Clean up the related code in FSEditLog a bit > Notably, this JIRA is going to temporarily break the BackupNode. I plan to > circle back on the BackupNode later on this branch. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1820) FTPFileSystem attempts to close the outputstream even when it is not initialised
FTPFileSystem attempts to close the outputstream even when it is not initialised
Key: HDFS-1820
URL: https://issues.apache.org/jira/browse/HDFS-1820
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 0.20.1
Environment: occurs on all platforms
Reporter: Sudharsan Sampath
FTPFileSystem's create method attempts to close the outputstream even when it
is not initialized causing a null pointer exception. In our case the apache
commons FTPClient was not able to create the destination file due to
permissions issue. The FtpClient promptly reported a 553 : Permissions issue
but it was overlooked in FTPFileSystem create method.
The following code fails
if (!FTPReply.isPositivePreliminary(client.getReplyCode())) {
// The ftpClient is an inconsistent state. Must close the stream
// which in turn will logout and disconnect from FTP server
fos.close();
throw new IOException("Unable to create file: " + file + ", Aborting");
}
as 'fos' is null. As a result the proper error message "Unable to create file
XXX" is not reported but rather a null pointer exception.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-664) Add a way to efficiently replace a disk in a live datanode
[ https://issues.apache.org/jira/browse/HDFS-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Valderrama updated HDFS-664: - Attachment: HDFS-664.patch This is a proof-of-concept patch against 0.20.3-rc2. You can add a volume to a live datanode by running the command "hadoop dnshell -registerVolume /path/to/volume" on the machine hosting the datanode. > Add a way to efficiently replace a disk in a live datanode > -- > > Key: HDFS-664 > URL: https://issues.apache.org/jira/browse/HDFS-664 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Affects Versions: 0.22.0 >Reporter: Steve Loughran > Attachments: HDFS-664.patch > > > In clusters where the datanode disks are hot swappable, you need to be able > to swap out a disk on a live datanode without taking down the datanode. You > don't want to decommission the whole node as that is overkill. on a system > with 4 1TB HDDs, giving 3 TB of datanode storage, a decommissioning and > restart will consume up to 6 TB of bandwidth. If a single disk were swapped > in then there would only be 1TB of data to recover over the network. More > importantly, if that data could be moved to free space on the same machine, > the recommissioning could take place at disk rates, not network speeds. > # Maybe have a way of decommissioning a single disk on the DN; the files > could be moved to space on the other disks or the other machines in the rack. > # There may not be time to use that option, in which case pulling out the > disk would be done with no warning, a new disk inserted. > # The DN needs to see that a disk has been replaced (or react to some ops > request telling it this), and start using the new disk again -pushing back > data, rebuilding the balance. > To complicate the process, assume there is a live TT on the system, running > jobs against the data. The TT would probably need to be paused while the work > takes place, any ongoing work handled somehow. Halting the TT and then > restarting it after the replacement disk went in is probably simplest. > The more disks you add to a node, the more this scenario becomes a need. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1814) HDFS portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[ https://issues.apache.org/jira/browse/HDFS-1814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017458#comment-13017458 ] Daryn Sharp commented on HDFS-1814: --- I suggested passing the config because it can accept a config. Although it doesn't appear to be used currently, it might someday... Perhaps someone else can comment. I was just lamenting the current test situation, so I'm fine with a separate bug for the tests. > HDFS portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent > --- > > Key: HDFS-1814 > URL: https://issues.apache.org/jira/browse/HDFS-1814 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs client, name-node >Affects Versions: 0.23.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: hdfs-1814.0.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1821) FileContext.createSymlink with kerberos enabled sets wrong owner
FileContext.createSymlink with kerberos enabled sets wrong owner Key: HDFS-1821 URL: https://issues.apache.org/jira/browse/HDFS-1821 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.22.0 Environment: Kerberos enabled on cluster Reporter: John George Assignee: John George TEST SETUP Using attached sample hdfs java program that illustrates the issue. Using cluster with Kerberos enabled on cluster # Compile instructions $ javac Symlink.java -cp `hadoop classpath` $ jar -cfe Symlink.jar Symlink Symlink.class # create test file for symlink to use 1. hadoop fs -touchz /user/username/filetest # Create symlink using file context 2. hadoop jar Symlink.jar ln /user/username/filetest /user/username/testsymlink # Verify owner of test file 3. hadoop jar Symlink.jar ls /user/username/ -rw-r--r-- jeagles hdfs /user/jeagles/filetest -rwxrwxrwx jeag...@xx..x.xxx hdfs /user/username/testsymlink RESULTS 1. Owner shows 'usern...@xx..x.xxx' for symlink, expecting 'username'. 2. Symlink is corrupted and can't removed, since it was created with an invalid user Sample program to create Symlink FileContext fc = FileContext.getFileContext(getConf()); fc.createSymlink(target, symlink, false); --- -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Attachment: trunkLocalNameImage7.patch Thank Matt for reviewing the patch and performing the test. TrunkLocalNameImage7.patch addressed his comment. > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch, > trunkLocalNameImage3.patch, trunkLocalNameImage4.patch, > trunkLocalNameImage5.patch, trunkLocalNameImage6.patch, > trunkLocalNameImage7.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1630) Checksum fsedits
[ https://issues.apache.org/jira/browse/HDFS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1630: Status: Open (was: Patch Available) > Checksum fsedits > > > Key: HDFS-1630 > URL: https://issues.apache.org/jira/browse/HDFS-1630 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: editsChecksum.patch, editsChecksum1.patch > > > HDFS-903 calculates a MD5 checksum to a saved image, so that we could verify > the integrity of the image at the loading time. > The other half of the story is how to verify fsedits. Similarly we could use > the checksum approach. But since a fsedit file is growing constantly, a > checksum per file does not work. I am thinking to add a checksum per > transaction. Is it doable or too expensive? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1630) Checksum fsedits
[ https://issues.apache.org/jira/browse/HDFS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1630: Status: Patch Available (was: Open) > Checksum fsedits > > > Key: HDFS-1630 > URL: https://issues.apache.org/jira/browse/HDFS-1630 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: editsChecksum.patch, editsChecksum1.patch > > > HDFS-903 calculates a MD5 checksum to a saved image, so that we could verify > the integrity of the image at the loading time. > The other half of the story is how to verify fsedits. Similarly we could use > the checksum approach. But since a fsedit file is growing constantly, a > checksum per file does not work. I am thinking to add a checksum per > transaction. Is it doable or too expensive? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1630) Checksum fsedits
[ https://issues.apache.org/jira/browse/HDFS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1630: Attachment: editsChecksum1.patch This fixed the failed test. > Checksum fsedits > > > Key: HDFS-1630 > URL: https://issues.apache.org/jira/browse/HDFS-1630 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: editsChecksum.patch, editsChecksum1.patch > > > HDFS-903 calculates a MD5 checksum to a saved image, so that we could verify > the integrity of the image at the loading time. > The other half of the story is how to verify fsedits. Similarly we could use > the checksum approach. But since a fsedit file is growing constantly, a > checksum per file does not work. I am thinking to add a checksum per > transaction. Is it doable or too expensive? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Attachment: trunkLocalNameImage7.patch > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch, > trunkLocalNameImage3.patch, trunkLocalNameImage4.patch, > trunkLocalNameImage5.patch, trunkLocalNameImage6.patch, > trunkLocalNameImage7.patch, trunkLocalNameImage7.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Status: Open (was: Patch Available) > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch, > trunkLocalNameImage3.patch, trunkLocalNameImage4.patch, > trunkLocalNameImage5.patch, trunkLocalNameImage6.patch, > trunkLocalNameImage7.patch, trunkLocalNameImage7.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Status: Patch Available (was: Open) > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch, > trunkLocalNameImage3.patch, trunkLocalNameImage4.patch, > trunkLocalNameImage5.patch, trunkLocalNameImage6.patch, > trunkLocalNameImage7.patch, trunkLocalNameImage7.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Attachment: (was: trunkLocalNameImage7.patch) > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch, > trunkLocalNameImage3.patch, trunkLocalNameImage4.patch, > trunkLocalNameImage5.patch, trunkLocalNameImage6.patch, > trunkLocalNameImage7.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Attachment: (was: trunkLocalNameImage7.patch) > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch, > trunkLocalNameImage3.patch, trunkLocalNameImage4.patch, > trunkLocalNameImage5.patch, trunkLocalNameImage6.patch, > trunkLocalNameImage7.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hairong Kuang updated HDFS-1070: Attachment: trunkLocalNameImage7.patch > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch, > trunkLocalNameImage3.patch, trunkLocalNameImage4.patch, > trunkLocalNameImage5.patch, trunkLocalNameImage6.patch, > trunkLocalNameImage7.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1817) Split TestFiDataTransferProtocol.java into two files
[
https://issues.apache.org/jira/browse/HDFS-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017555#comment-13017555
]
Tanping Wang commented on HDFS-1817:
+1. Patch looks good to me.
> Split TestFiDataTransferProtocol.java into two files
>
>
> Key: HDFS-1817
> URL: https://issues.apache.org/jira/browse/HDFS-1817
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Trivial
> Fix For: 0.23.0
>
> Attachments: h1817_20110407.patch
>
>
> {{TestFiDataTransferProtocol}} has tests from pipeline_Fi_01 to _16 and
> pipeline_Fi_39 to _51. It is natural to split them into two files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1818) TestHDFSCLI is failing on trunk
[ https://issues.apache.org/jira/browse/HDFS-1818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017564#comment-13017564 ] Tsz Wo (Nicholas), SZE commented on HDFS-1818: -- Thanks Aaron and Todd for fixing this. You guys are so efficient! > TestHDFSCLI is failing on trunk > --- > > Key: HDFS-1818 > URL: https://issues.apache.org/jira/browse/HDFS-1818 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 0.23.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 0.23.0 > > Attachments: hdfs-1818.0.txt > > > The commit of HADOOP-7202 changed the output of a few FsShell commands. Since > HDFS tests rely on the precise format of this output, TestHDFSCLI is now > failing. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1817) Split TestFiDataTransferProtocol.java into two files
[
https://issues.apache.org/jira/browse/HDFS-1817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-1817:
-
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Thanks Tanping for the review.
I have committed this.
> Split TestFiDataTransferProtocol.java into two files
>
>
> Key: HDFS-1817
> URL: https://issues.apache.org/jira/browse/HDFS-1817
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
>Priority: Trivial
> Fix For: 0.23.0
>
> Attachments: h1817_20110407.patch
>
>
> {{TestFiDataTransferProtocol}} has tests from pipeline_Fi_01 to _16 and
> pipeline_Fi_39 to _51. It is natural to split them into two files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1630) Checksum fsedits
[ https://issues.apache.org/jira/browse/HDFS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017578#comment-13017578 ] Hadoop QA commented on HDFS-1630: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475826/editsChecksum1.patch against trunk revision 1090114. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/333//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/333//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/333//console This message is automatically generated. > Checksum fsedits > > > Key: HDFS-1630 > URL: https://issues.apache.org/jira/browse/HDFS-1630 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: editsChecksum.patch, editsChecksum1.patch > > > HDFS-903 calculates a MD5 checksum to a saved image, so that we could verify > the integrity of the image at the loading time. > The other half of the story is how to verify fsedits. Similarly we could use > the checksum approach. But since a fsedit file is growing constantly, a > checksum per file does not work. I am thinking to add a checksum per > transaction. Is it doable or too expensive? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1630) Checksum fsedits
[ https://issues.apache.org/jira/browse/HDFS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017592#comment-13017592 ] dhruba borthakur commented on HDFS-1630: +1, code looks good to me. > Checksum fsedits > > > Key: HDFS-1630 > URL: https://issues.apache.org/jira/browse/HDFS-1630 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: editsChecksum.patch, editsChecksum1.patch > > > HDFS-903 calculates a MD5 checksum to a saved image, so that we could verify > the integrity of the image at the loading time. > The other half of the story is how to verify fsedits. Similarly we could use > the checksum approach. But since a fsedit file is growing constantly, a > checksum per file does not work. I am thinking to add a checksum per > transaction. Is it doable or too expensive? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1813) Hdfs Federation: Authentication using BlockToken in RPC to datanode fails.
[ https://issues.apache.org/jira/browse/HDFS-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017597#comment-13017597 ] Suresh Srinivas commented on HDFS-1813: --- +1 for the patch. > Hdfs Federation: Authentication using BlockToken in RPC to datanode fails. > -- > > Key: HDFS-1813 > URL: https://issues.apache.org/jira/browse/HDFS-1813 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: Federation Branch > > Attachments: HDFS-1813.1.patch, HDFS-1813.2.patch, HDFS-1813.3.patch, > HDFS-1813.4.patch > > > Several issues are causing this problem > 1. The last LocatedBlock returned by getBlockLocations doesn't have > BlockToken. > 2. The blockPoolTokenSecretManager is not initialized before created rpc > server in datanode. > 3. The getReplicaVisibleLength API in datanode expects WRITE permission in > the block token, but the block tokens are generated with read permission only > in getBlockLocations at namenode. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-744) Support hsync in HDFS
[ https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017598#comment-13017598 ] Hairong Kuang commented on HDFS-744: @Linyin, I am ok if we simply sync every packet to disk if the sync is set as a first pass. This should work when sync calls are very frequent. If this turns out to be a performance problem for hbase, we could file a separate jira to optimize it by syncing to disk only when flush is called or at block boundary. @Nicholas, what your suggested seems good to me. Could we do it in a separate jira? > Support hsync in HDFS > - > > Key: HDFS-744 > URL: https://issues.apache.org/jira/browse/HDFS-744 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hairong Kuang > > HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, > the real expected semantics should be "flushes out to all replicas and all > replicas have done posix fsync equivalent - ie the OS has flushed it to the > disk device (but the disk may have it in its cache)." This jira aims to > implement the expected behaviour. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline
[
https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-1606:
-
Attachment: h1606_20110408.patch
I suspect that the failures of {{TestFiDataTransferProtocol}} were due to
resource not cleaning up completely by {{MiniDFSCluster}}. Moved some the
tests around in HDFS-1817. See if it works.
h1606_20110408.patch: updated with trunk
> Provide a stronger data guarantee in the write pipeline
> ---
>
> Key: HDFS-1606
> URL: https://issues.apache.org/jira/browse/HDFS-1606
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, hdfs client, name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.23.0
>
> Attachments: h1606_20110210.patch, h1606_20110211.patch,
> h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch,
> h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch,
> h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch,
> h1606_20110407c.patch, h1606_20110408.patch
>
>
> In the current design, if there is a datanode/network failure in the write
> pipeline, DFSClient will try to remove the failed datanode from the pipeline
> and then continue writing with the remaining datanodes. As a result, the
> number of datanodes in the pipeline is decreased. Unfortunately, it is
> possible that DFSClient may incorrectly remove a healthy datanode but leave
> the failed datanode in the pipeline because failure detection may be
> inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline
> in order to provide a stronger data guarantee.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1822) Editlog opcodes overlap between 20 security and later releases
Editlog opcodes overlap between 20 security and later releases -- Key: HDFS-1822 URL: https://issues.apache.org/jira/browse/HDFS-1822 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0, 0.22.0, 0.23.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Blocker Fix For: 0.22.0, 0.23.0 Same opcode are used for different operations between 0.20.security, 0.22 and 0.23. This results in failure to load editlogs on later release, especially during upgrades. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline
[ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017624#comment-13017624 ] Hadoop QA commented on HDFS-1606: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475833/h1606_20110408.patch against trunk revision 1090357. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 31 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/334//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/334//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/334//console This message is automatically generated. > Provide a stronger data guarantee in the write pipeline > --- > > Key: HDFS-1606 > URL: https://issues.apache.org/jira/browse/HDFS-1606 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node, hdfs client, name-node >Affects Versions: 0.23.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.23.0 > > Attachments: h1606_20110210.patch, h1606_20110211.patch, > h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, > h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, > h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, > h1606_20110407c.patch, h1606_20110408.patch > > > In the current design, if there is a datanode/network failure in the write > pipeline, DFSClient will try to remove the failed datanode from the pipeline > and then continue writing with the remaining datanodes. As a result, the > number of datanodes in the pipeline is decreased. Unfortunately, it is > possible that DFSClient may incorrectly remove a healthy datanode but leave > the failed datanode in the pipeline because failure detection may be > inaccurate under erroneous conditions. > We propose to have a new mechanism for adding new datanodes to the pipeline > in order to provide a stronger data guarantee. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-1813) Hdfs Federation: Authentication using BlockToken in RPC to datanode fails.
[ https://issues.apache.org/jira/browse/HDFS-1813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey resolved HDFS-1813. Resolution: Fixed Committed the latest patch. > Hdfs Federation: Authentication using BlockToken in RPC to datanode fails. > -- > > Key: HDFS-1813 > URL: https://issues.apache.org/jira/browse/HDFS-1813 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Jitendra Nath Pandey >Assignee: Jitendra Nath Pandey > Fix For: Federation Branch > > Attachments: HDFS-1813.1.patch, HDFS-1813.2.patch, HDFS-1813.3.patch, > HDFS-1813.4.patch > > > Several issues are causing this problem > 1. The last LocatedBlock returned by getBlockLocations doesn't have > BlockToken. > 2. The blockPoolTokenSecretManager is not initialized before created rpc > server in datanode. > 3. The getReplicaVisibleLength API in datanode expects WRITE permission in > the block token, but the block tokens are generated with read permission only > in getBlockLocations at namenode. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1822) Editlog opcodes overlap between 20 security and later releases
[ https://issues.apache.org/jira/browse/HDFS-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017628#comment-13017628 ] Suresh Srinivas commented on HDFS-1822: --- 20.security (LAYOUT_VERSION = -19) adds the following opcodes: private static final byte OP_GET_DELEGATION_TOKEN = 15; //new delegation token private static final byte OP_RENEW_DELEGATION_TOKEN = 16; //renew delegation token private static final byte OP_CANCEL_DELEGATION_TOKEN = 17; //cancel delegation token private static final byte OP_UPDATE_MASTER_KEY = 18; //update master key 21 (layout version = -24) adds the following opcodes: private static final byte OP_RENAME = 15; // new rename private static final byte OP_CONCAT_DELETE = 16; // concat files. private static final byte OP_SYMLINK = 17; // a symbolic link private static final byte OP_GET_DELEGATION_TOKEN = 18; //new delegation token private static final byte OP_RENEW_DELEGATION_TOKEN = 19; //renew delegation token private static final byte OP_CANCEL_DELEGATION_TOKEN = 20; //cancel delegation token private static final byte OP_UPDATE_MASTER_KEY = 21; //update master key 22 (layout version = -27) and trunk (layout version > -27) adds the following opcodes: public static final byte OP_RENAME = 15; // new rename public static final byte OP_CONCAT_DELETE = 16; // concat files. public static final byte OP_SYMLINK = 17; // a symbolic link public static final byte OP_GET_DELEGATION_TOKEN = 18; //new delegation token public static final byte OP_RENEW_DELEGATION_TOKEN = 19; //renew delegation token public static final byte OP_CANCEL_DELEGATION_TOKEN = 20; //cancel delegation token public static final byte OP_UPDATE_MASTER_KEY = 21; //update master key Conflicts in the opcodes: # Opcode 15 means OP_GET_DELEGATION_TOKEN on 20.s and OP_RENAME on later releases # Opcode 16 means OP_RENEW_DELEGATION_TOKEN on 20.s and OP_CONCAT_DELETE on later releases # Opcode 17 means OP_CANCEL_DELEGATION_TOKEN on 20.s and OP_SYMLINK on later releases # Opcode 18 means OP_UPDATE_MASTER_KEY on 20.s and OP_GET_DELEGATION_TOKEN on later releases We need to support the following upgrades: # 20.s to 22 or later releases #* The opcode conflict here makes consuming editlogs impossible # Need to support upgrade from 21 to 22 or later releases I am proposing handling these conflicts as follows, while consuming editlogs: # If layout version is > -24 then it is 20 version, use the definition as shown in 20.security # If layout version is <= -24 use the definition from 21 onwards. This is messy way of doing it. But I do not see any way around it. In future: - We need to make sure, any op code added is added in the trunk first, before adding it in older releases. Trunk should be the source of truth, ensuring the opcodes are chosen uniquely across different releases. > Editlog opcodes overlap between 20 security and later releases > -- > > Key: HDFS-1822 > URL: https://issues.apache.org/jira/browse/HDFS-1822 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0, 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > > Same opcode are used for different operations between 0.20.security, 0.22 and > 0.23. This results in failure to load editlogs on later release, especially > during upgrades. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-744) Support hsync in HDFS
[ https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017633#comment-13017633 ] Tsz Wo (Nicholas), SZE commented on HDFS-744: - > ... Could we do it in a separate jira? Sure. > Support hsync in HDFS > - > > Key: HDFS-744 > URL: https://issues.apache.org/jira/browse/HDFS-744 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Hairong Kuang > > HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, > the real expected semantics should be "flushes out to all replicas and all > replicas have done posix fsync equivalent - ie the OS has flushed it to the > disk device (but the disk may have it in its cache)." This jira aims to > implement the expected behaviour. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1821) FileContext.createSymlink with kerberos enabled sets wrong owner
[ https://issues.apache.org/jira/browse/HDFS-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-1821: -- Attachment: HDFS-1821.patch > FileContext.createSymlink with kerberos enabled sets wrong owner > > > Key: HDFS-1821 > URL: https://issues.apache.org/jira/browse/HDFS-1821 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 > Environment: Kerberos enabled on cluster >Reporter: John George >Assignee: John George > Attachments: HDFS-1821.patch > > > TEST SETUP > Using attached sample hdfs java program that illustrates the issue. > Using cluster with Kerberos enabled on cluster > # Compile instructions > $ javac Symlink.java -cp `hadoop classpath` > $ jar -cfe Symlink.jar Symlink Symlink.class > # create test file for symlink to use > 1. hadoop fs -touchz /user/username/filetest > # Create symlink using file context > 2. hadoop jar Symlink.jar ln /user/username/filetest > /user/username/testsymlink > # Verify owner of test file > 3. hadoop jar Symlink.jar ls /user/username/ > -rw-r--r-- jeagles hdfs /user/jeagles/filetest > -rwxrwxrwx jeag...@xx..x.xxx hdfs /user/username/testsymlink > RESULTS > 1. Owner shows 'usern...@xx..x.xxx' for symlink, expecting 'username'. > 2. Symlink is corrupted and can't removed, since it was created with an > invalid user > > Sample program to create Symlink > FileContext fc = FileContext.getFileContext(getConf()); > fc.createSymlink(target, symlink, false); > --- -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1821) FileContext.createSymlink with kerberos enabled sets wrong owner
[ https://issues.apache.org/jira/browse/HDFS-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-1821: -- Status: Patch Available (was: Open) > FileContext.createSymlink with kerberos enabled sets wrong owner > > > Key: HDFS-1821 > URL: https://issues.apache.org/jira/browse/HDFS-1821 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 > Environment: Kerberos enabled on cluster >Reporter: John George >Assignee: John George > Attachments: HDFS-1821.patch > > > TEST SETUP > Using attached sample hdfs java program that illustrates the issue. > Using cluster with Kerberos enabled on cluster > # Compile instructions > $ javac Symlink.java -cp `hadoop classpath` > $ jar -cfe Symlink.jar Symlink Symlink.class > # create test file for symlink to use > 1. hadoop fs -touchz /user/username/filetest > # Create symlink using file context > 2. hadoop jar Symlink.jar ln /user/username/filetest > /user/username/testsymlink > # Verify owner of test file > 3. hadoop jar Symlink.jar ls /user/username/ > -rw-r--r-- jeagles hdfs /user/jeagles/filetest > -rwxrwxrwx jeag...@xx..x.xxx hdfs /user/username/testsymlink > RESULTS > 1. Owner shows 'usern...@xx..x.xxx' for symlink, expecting 'username'. > 2. Symlink is corrupted and can't removed, since it was created with an > invalid user > > Sample program to create Symlink > FileContext fc = FileContext.getFileContext(getConf()); > fc.createSymlink(target, symlink, false); > --- -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1821) FileContext.createSymlink with kerberos enabled sets wrong owner
[ https://issues.apache.org/jira/browse/HDFS-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017639#comment-13017639 ] John George commented on HDFS-1821: --- [exec] BUILD SUCCESSFUL [exec] Total time: 37 seconds [exec] [exec] [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] > FileContext.createSymlink with kerberos enabled sets wrong owner > > > Key: HDFS-1821 > URL: https://issues.apache.org/jira/browse/HDFS-1821 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 > Environment: Kerberos enabled on cluster >Reporter: John George >Assignee: John George > Attachments: HDFS-1821.patch > > > TEST SETUP > Using attached sample hdfs java program that illustrates the issue. > Using cluster with Kerberos enabled on cluster > # Compile instructions > $ javac Symlink.java -cp `hadoop classpath` > $ jar -cfe Symlink.jar Symlink Symlink.class > # create test file for symlink to use > 1. hadoop fs -touchz /user/username/filetest > # Create symlink using file context > 2. hadoop jar Symlink.jar ln /user/username/filetest > /user/username/testsymlink > # Verify owner of test file > 3. hadoop jar Symlink.jar ls /user/username/ > -rw-r--r-- jeagles hdfs /user/jeagles/filetest > -rwxrwxrwx jeag...@xx..x.xxx hdfs /user/username/testsymlink > RESULTS > 1. Owner shows 'usern...@xx..x.xxx' for symlink, expecting 'username'. > 2. Symlink is corrupted and can't removed, since it was created with an > invalid user > > Sample program to create Symlink > FileContext fc = FileContext.getFileContext(getConf()); > fc.createSymlink(target, symlink, false); > --- -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1821) FileContext.createSymlink with kerberos enabled sets wrong owner
[ https://issues.apache.org/jira/browse/HDFS-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-1821: -- Description: TEST SETUP Using attached sample hdfs java program that illustrates the issue. Using cluster with Kerberos enabled on cluster # Compile instructions $ javac Symlink.java -cp `hadoop classpath` $ jar -cfe Symlink.jar Symlink Symlink.class # create test file for symlink to use 1. hadoop fs -touchz /user/username/filetest # Create symlink using file context 2. hadoop jar Symlink.jar ln /user/username/filetest /user/username/testsymlink # Verify owner of test file 3. hadoop jar Symlink.jar ls /user/username/ -rw-r--r-- username hdfs /user/jeagles/filetest -rwxrwxrwx usern...@xx..x.xxx hdfs /user/username/testsymlink RESULTS 1. Owner shows 'usern...@xx..x.xxx' for symlink, expecting 'username'. 2. Symlink is corrupted and can't removed, since it was created with an invalid user Sample program to create Symlink FileContext fc = FileContext.getFileContext(getConf()); fc.createSymlink(target, symlink, false); --- was: TEST SETUP Using attached sample hdfs java program that illustrates the issue. Using cluster with Kerberos enabled on cluster # Compile instructions $ javac Symlink.java -cp `hadoop classpath` $ jar -cfe Symlink.jar Symlink Symlink.class # create test file for symlink to use 1. hadoop fs -touchz /user/username/filetest # Create symlink using file context 2. hadoop jar Symlink.jar ln /user/username/filetest /user/username/testsymlink # Verify owner of test file 3. hadoop jar Symlink.jar ls /user/username/ -rw-r--r-- jeagles hdfs /user/jeagles/filetest -rwxrwxrwx jeag...@xx..x.xxx hdfs /user/username/testsymlink RESULTS 1. Owner shows 'usern...@xx..x.xxx' for symlink, expecting 'username'. 2. Symlink is corrupted and can't removed, since it was created with an invalid user Sample program to create Symlink FileContext fc = FileContext.getFileContext(getConf()); fc.createSymlink(target, symlink, false); --- > FileContext.createSymlink with kerberos enabled sets wrong owner > > > Key: HDFS-1821 > URL: https://issues.apache.org/jira/browse/HDFS-1821 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 > Environment: Kerberos enabled on cluster >Reporter: John George >Assignee: John George > Attachments: HDFS-1821.patch > > > TEST SETUP > Using attached sample hdfs java program that illustrates the issue. > Using cluster with Kerberos enabled on cluster > # Compile instructions > $ javac Symlink.java -cp `hadoop classpath` > $ jar -cfe Symlink.jar Symlink Symlink.class > # create test file for symlink to use > 1. hadoop fs -touchz /user/username/filetest > # Create symlink using file context > 2. hadoop jar Symlink.jar ln /user/username/filetest > /user/username/testsymlink > # Verify owner of test file > 3. hadoop jar Symlink.jar ls /user/username/ > -rw-r--r-- username hdfs /user/jeagles/filetest > -rwxrwxrwx usern...@xx..x.xxx hdfs /user/username/testsymlink > RESULTS > 1. Owner shows 'usern...@xx..x.xxx' for symlink, expecting 'username'. > 2. Symlink is corrupted and can't removed, since it was created with an > invalid user > > Sample program to create Symlink > FileContext fc = FileContext.getFileContext(getConf()); > fc.createSymlink(target, symlink, false); > --- -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1814) HDFS portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[ https://issues.apache.org/jira/browse/HDFS-1814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017642#comment-13017642 ] Aaron T. Myers commented on HDFS-1814: -- Just finished running the full test suite. The only failures were: org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.TestDFSShell All of these are presently failing on trunk. Since TestDFSShell is pertinent to this patch, I manually verified that the failure is for the same reason that it's failing on trunk. > HDFS portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent > --- > > Key: HDFS-1814 > URL: https://issues.apache.org/jira/browse/HDFS-1814 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs client, name-node >Affects Versions: 0.23.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: hdfs-1814.0.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Moved] (HDFS-1823) start-dfs.sh script fails if HADOOP_HOME is not set
[ https://issues.apache.org/jira/browse/HDFS-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White moved HADOOP-7217 to HDFS-1823: - Component/s: (was: scripts) scripts Fix Version/s: (was: 0.22.0) 0.22.0 Affects Version/s: (was: 0.21.0) 0.21.0 Key: HDFS-1823 (was: HADOOP-7217) Project: Hadoop HDFS (was: Hadoop Common) > start-dfs.sh script fails if HADOOP_HOME is not set > --- > > Key: HDFS-1823 > URL: https://issues.apache.org/jira/browse/HDFS-1823 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.21.0 >Reporter: Tom White >Assignee: Tom White > Fix For: 0.22.0 > > > HDFS portion of HADOOP-6953 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1052) HDFS scalability with multiple namenodes
[ https://issues.apache.org/jira/browse/HDFS-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-1052: -- Attachment: HDFS-1052.patch Consolidated patch for integrating from federation to trunk. > HDFS scalability with multiple namenodes > > > Key: HDFS-1052 > URL: https://issues.apache.org/jira/browse/HDFS-1052 > Project: Hadoop HDFS > Issue Type: New Feature > Components: name-node >Affects Versions: 0.22.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Attachments: Block pool proposal.pdf, HDFS-1052.patch, Mulitple > Namespaces5.pdf, high-level-design.pdf > > > HDFS currently uses a single namenode that limits scalability of the cluster. > This jira proposes an architecture to scale the nameservice horizontally > using multiple namenodes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
[
https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
bc Wong updated HDFS-941:
-
Attachment: HDFS-941-6.patch
Thanks for the feedback, Stack. The v6 patch:
* Undid the whitespace changes. My IDE was misconfigured.
* Turned hardcoded values into config keys.
* Fixed javadoc on BlockReader.
* Renamed {{gotEOS}} to {{eos}}.
* Handled null remoteAddr by closing the socket.
Did not work on:
* DN javadoc. I don't think I touched it.
* SocketCache copyright. It's there, I think.
I don't follow this (not sure what you're referring to):
bq. Why would we retry a socket that is throwing an IOE?
> Datanode xceiver protocol should allow reuse of a connection
>
>
> Key: HDFS-941
> URL: https://issues.apache.org/jira/browse/HDFS-941
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: data-node, hdfs client
>Affects Versions: 0.22.0
>Reporter: Todd Lipcon
>Assignee: bc Wong
> Attachments: HDFS-941-1.patch, HDFS-941-2.patch, HDFS-941-3.patch,
> HDFS-941-3.patch, HDFS-941-4.patch, HDFS-941-5.patch, HDFS-941-6.patch,
> hdfs941-1.png
>
>
> Right now each connection into the datanode xceiver only processes one
> operation.
> In the case that an operation leaves the stream in a well-defined state (eg a
> client reads to the end of a block successfully) the same connection could be
> reused for a second operation. This should improve random read performance
> significantly.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline
[ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017675#comment-13017675 ] Jitendra Nath Pandey commented on HDFS-1606: 1. The method findNewDatanode should return all new datanodes in case there are more than one new datanode. 2. The method addDatanode2ExistingPipeline can be split into following methods a) method to check if transfer is needed. b) method to get additional datanodes and determine source and destination c) method that does actual transfer. 3. DataStreamer#hflush : Should we change it to setHflush(boolean val) to clarify its just setting a flag? 4. Does it make sense to add a unit test for default ReplaceDatanodeOnFailure policy? > Provide a stronger data guarantee in the write pipeline > --- > > Key: HDFS-1606 > URL: https://issues.apache.org/jira/browse/HDFS-1606 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node, hdfs client, name-node >Affects Versions: 0.23.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.23.0 > > Attachments: h1606_20110210.patch, h1606_20110211.patch, > h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, > h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, > h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, > h1606_20110407c.patch, h1606_20110408.patch > > > In the current design, if there is a datanode/network failure in the write > pipeline, DFSClient will try to remove the failed datanode from the pipeline > and then continue writing with the remaining datanodes. As a result, the > number of datanodes in the pipeline is decreased. Unfortunately, it is > possible that DFSClient may incorrectly remove a healthy datanode but leave > the failed datanode in the pipeline because failure detection may be > inaccurate under erroneous conditions. > We propose to have a new mechanism for adding new datanodes to the pipeline > in order to provide a stronger data guarantee. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1821) FileContext.createSymlink with kerberos enabled sets wrong owner
[ https://issues.apache.org/jira/browse/HDFS-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017677#comment-13017677 ] Hadoop QA commented on HDFS-1821: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475836/HDFS-1821.patch against trunk revision 1090357. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/335//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/335//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/335//console This message is automatically generated. > FileContext.createSymlink with kerberos enabled sets wrong owner > > > Key: HDFS-1821 > URL: https://issues.apache.org/jira/browse/HDFS-1821 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 > Environment: Kerberos enabled on cluster >Reporter: John George >Assignee: John George > Attachments: HDFS-1821.patch > > > TEST SETUP > Using attached sample hdfs java program that illustrates the issue. > Using cluster with Kerberos enabled on cluster > # Compile instructions > $ javac Symlink.java -cp `hadoop classpath` > $ jar -cfe Symlink.jar Symlink Symlink.class > # create test file for symlink to use > 1. hadoop fs -touchz /user/username/filetest > # Create symlink using file context > 2. hadoop jar Symlink.jar ln /user/username/filetest > /user/username/testsymlink > # Verify owner of test file > 3. hadoop jar Symlink.jar ls /user/username/ > -rw-r--r-- username hdfs /user/jeagles/filetest > -rwxrwxrwx usern...@xx..x.xxx hdfs /user/username/testsymlink > RESULTS > 1. Owner shows 'usern...@xx..x.xxx' for symlink, expecting 'username'. > 2. Symlink is corrupted and can't removed, since it was created with an > invalid user > > Sample program to create Symlink > FileContext fc = FileContext.getFileContext(getConf()); > fc.createSymlink(target, symlink, false); > --- -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1822) Editlog opcodes overlap between 20 security and later releases
[ https://issues.apache.org/jira/browse/HDFS-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017685#comment-13017685 ] dhruba borthakur commented on HDFS-1822: > We need to make sure, any op code added is added in the trunk first, before > adding it in older releases Have we made any apache release off 0.20.security? > Editlog opcodes overlap between 20 security and later releases > -- > > Key: HDFS-1822 > URL: https://issues.apache.org/jira/browse/HDFS-1822 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0, 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > > Same opcode are used for different operations between 0.20.security, 0.22 and > 0.23. This results in failure to load editlogs on later release, especially > during upgrades. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1823) start-dfs.sh script fails if HADOOP_HOME is not set
[ https://issues.apache.org/jira/browse/HDFS-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HDFS-1823: Attachment: HDFS-1823.patch Simple fix. I manually tested this by starting a pseudo-distributed cluster using this patch and without HADOOP_HOME set. Without the patch the message "Hadoop common not found." is printed and the daemons don't start. > start-dfs.sh script fails if HADOOP_HOME is not set > --- > > Key: HDFS-1823 > URL: https://issues.apache.org/jira/browse/HDFS-1823 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.21.0 >Reporter: Tom White >Assignee: Tom White > Fix For: 0.22.0 > > Attachments: HDFS-1823.patch > > > HDFS portion of HADOOP-6953 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1823) start-dfs.sh script fails if HADOOP_HOME is not set
[ https://issues.apache.org/jira/browse/HDFS-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HDFS-1823: Status: Patch Available (was: Open) > start-dfs.sh script fails if HADOOP_HOME is not set > --- > > Key: HDFS-1823 > URL: https://issues.apache.org/jira/browse/HDFS-1823 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.21.0 >Reporter: Tom White >Assignee: Tom White > Fix For: 0.22.0 > > Attachments: HDFS-1823.patch > > > HDFS portion of HADOOP-6953 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1822) Editlog opcodes overlap between 20 security and later releases
[ https://issues.apache.org/jira/browse/HDFS-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017721#comment-13017721 ] Suresh Srinivas commented on HDFS-1822: --- > Have we made any apache release off 0.20.security? Yes 0.20-security.203 release is already available outside. This is deployed in Yahoo and also CDH releases use this change. > Editlog opcodes overlap between 20 security and later releases > -- > > Key: HDFS-1822 > URL: https://issues.apache.org/jira/browse/HDFS-1822 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0, 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > > Same opcode are used for different operations between 0.20.security, 0.22 and > 0.23. This results in failure to load editlogs on later release, especially > during upgrades. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1822) Editlog opcodes overlap between 20 security and later releases
[ https://issues.apache.org/jira/browse/HDFS-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-1822: -- Attachment: HDFS-1822.patch Changes: # Implemented changes proposed in my previous comment # Changed TestEditLog to new junit4 framework, along with the addition of testcase. > Editlog opcodes overlap between 20 security and later releases > -- > > Key: HDFS-1822 > URL: https://issues.apache.org/jira/browse/HDFS-1822 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0, 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > Attachments: HDFS-1822.patch > > > Same opcode are used for different operations between 0.20.security, 0.22 and > 0.23. This results in failure to load editlogs on later release, especially > during upgrades. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1823) start-dfs.sh script fails if HADOOP_HOME is not set
[ https://issues.apache.org/jira/browse/HDFS-1823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017744#comment-13017744 ] Hadoop QA commented on HDFS-1823: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475854/HDFS-1823.patch against trunk revision 1090357. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/337//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/337//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/337//console This message is automatically generated. > start-dfs.sh script fails if HADOOP_HOME is not set > --- > > Key: HDFS-1823 > URL: https://issues.apache.org/jira/browse/HDFS-1823 > Project: Hadoop HDFS > Issue Type: Bug > Components: scripts >Affects Versions: 0.21.0 >Reporter: Tom White >Assignee: Tom White > Fix For: 0.22.0 > > Attachments: HDFS-1823.patch > > > HDFS portion of HADOOP-6953 -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1814) HDFS portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent
[ https://issues.apache.org/jira/browse/HDFS-1814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017771#comment-13017771 ] Aaron T. Myers commented on HDFS-1814: -- I just re-ran TestDFSShell after the commit of HADOOP-7216, which fixes the TestDFSShell failure. > HDFS portion of HADOOP-7214 - Hadoop /usr/bin/groups equivalent > --- > > Key: HDFS-1814 > URL: https://issues.apache.org/jira/browse/HDFS-1814 > Project: Hadoop HDFS > Issue Type: New Feature > Components: hdfs client, name-node >Affects Versions: 0.23.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: hdfs-1814.0.txt > > -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1630) Checksum fsedits
[
https://issues.apache.org/jira/browse/HDFS-1630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017780#comment-13017780
]
Aaron T. Myers commented on HDFS-1630:
--
Code looks solid to me as well.
One nit: In the case of a checksum mismatch, could you please add the expected
and actual checksums which didn't match to the error message?
Also, in two places you check for "{{if (checksum != null)}}". Why would
{{checksum}} ever be null?
> Checksum fsedits
>
>
> Key: HDFS-1630
> URL: https://issues.apache.org/jira/browse/HDFS-1630
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: name-node
>Reporter: Hairong Kuang
>Assignee: Hairong Kuang
> Attachments: editsChecksum.patch, editsChecksum1.patch
>
>
> HDFS-903 calculates a MD5 checksum to a saved image, so that we could verify
> the integrity of the image at the loading time.
> The other half of the story is how to verify fsedits. Similarly we could use
> the checksum approach. But since a fsedit file is growing constantly, a
> checksum per file does not work. I am thinking to add a checksum per
> transaction. Is it doable or too expensive?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1070) Speedup NameNode image loading and saving by storing local file names
[ https://issues.apache.org/jira/browse/HDFS-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017786#comment-13017786 ] Hadoop QA commented on HDFS-1070: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475829/trunkLocalNameImage7.patch against trunk revision 1090357. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestFileAppend4 org.apache.hadoop.hdfs.TestFileCreationNamenodeRestart org.apache.hadoop.hdfs.TestLargeBlock org.apache.hadoop.hdfs.TestRenameWhileOpen org.apache.hadoop.hdfs.TestWriteConfigurationToDFS org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOIVCanReadOldVersions -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/336//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/336//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/336//console This message is automatically generated. > Speedup NameNode image loading and saving by storing local file names > - > > Key: HDFS-1070 > URL: https://issues.apache.org/jira/browse/HDFS-1070 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Attachments: trunkLocalNameImage.patch, trunkLocalNameImage1.patch, > trunkLocalNameImage3.patch, trunkLocalNameImage4.patch, > trunkLocalNameImage5.patch, trunkLocalNameImage6.patch, > trunkLocalNameImage7.patch > > > Currently each inode stores its full path in the fsimage. I'd propose to > store the local name instead. In order for each inode to identify its parent, > all inodes in a directory tree are stored in the image in in-order. This > proposal also requires each directory stores the number of its children in > image. > This proposal would bring a few benefits as pointed below and therefore > speedup the image loading and saving. > # Remove the overhead of converting java-UTF8 encoded local name to > string-represented full path then to UTF8 encoded full path when saving to an > image and vice versa when loading the image. > # Remove the overhead of traversing the full path when inserting the inode to > its parent inode. > # Reduce the number of temporary java objects during the process of image > loading or saving and therefore reduce the GC overhead. > # Reduce the size of an image. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1606) Provide a stronger data guarantee in the write pipeline
[
https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-1606:
-
Attachment: h1606_20110408b.patch
Thanks Jitendra for the review.
h1606_20110408b.patch:
> 1. The method findNewDatanode should return all new datanodes ...
Since it is only an internal method but not protocol or public API, we may
easily change it later when we add the multiple destination feature.
> 2. The method addDatanode2ExistingPipeline can be split ...
I only split actual transfer out. The remaining codes only has 20 lines
excluding comments.
> 3. DataStreamer#hflush : Should we change it to setHflush(boolean val) to
> clarify its just setting a flag?
Changed.
> 4. Does it make sense to add a unit test for default ReplaceDatanodeOnFailure
> policy?
Added {{testDefaultPolicy()}}.
> Provide a stronger data guarantee in the write pipeline
> ---
>
> Key: HDFS-1606
> URL: https://issues.apache.org/jira/browse/HDFS-1606
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: data-node, hdfs client, name-node
>Affects Versions: 0.23.0
>Reporter: Tsz Wo (Nicholas), SZE
>Assignee: Tsz Wo (Nicholas), SZE
> Fix For: 0.23.0
>
> Attachments: h1606_20110210.patch, h1606_20110211.patch,
> h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch,
> h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch,
> h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch,
> h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch
>
>
> In the current design, if there is a datanode/network failure in the write
> pipeline, DFSClient will try to remove the failed datanode from the pipeline
> and then continue writing with the remaining datanodes. As a result, the
> number of datanodes in the pipeline is decreased. Unfortunately, it is
> possible that DFSClient may incorrectly remove a healthy datanode but leave
> the failed datanode in the pipeline because failure detection may be
> inaccurate under erroneous conditions.
> We propose to have a new mechanism for adding new datanodes to the pipeline
> in order to provide a stronger data guarantee.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
delay instantiation of file system object until it is needed (linked to HADOOP-7207) Key: HDFS-1824 URL: https://issues.apache.org/jira/browse/HDFS-1824 Project: Hadoop HDFS Issue Type: Bug Reporter: Boris Shkolnik Assignee: Boris Shkolnik also re-factor the code little bit to avoid checking for instance of DFS in multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-1824) delay instantiation of file system object until it is needed (linked to HADOOP-7207)
[ https://issues.apache.org/jira/browse/HDFS-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1824: - Attachment: HDFS-1824.patch > delay instantiation of file system object until it is needed (linked to > HADOOP-7207) > > > Key: HDFS-1824 > URL: https://issues.apache.org/jira/browse/HDFS-1824 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1824.patch > > > also re-factor the code little bit to avoid checking for instance of DFS in > multiple places. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1606) Provide a stronger data guarantee in the write pipeline
[ https://issues.apache.org/jira/browse/HDFS-1606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017793#comment-13017793 ] Hadoop QA commented on HDFS-1606: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12475874/h1606_20110408b.patch against trunk revision 1090357. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 31 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/338//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/338//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/338//console This message is automatically generated. > Provide a stronger data guarantee in the write pipeline > --- > > Key: HDFS-1606 > URL: https://issues.apache.org/jira/browse/HDFS-1606 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node, hdfs client, name-node >Affects Versions: 0.23.0 >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 0.23.0 > > Attachments: h1606_20110210.patch, h1606_20110211.patch, > h1606_20110217.patch, h1606_20110228.patch, h1606_20110404.patch, > h1606_20110405.patch, h1606_20110405b.patch, h1606_20110406.patch, > h1606_20110406b.patch, h1606_20110407.patch, h1606_20110407b.patch, > h1606_20110407c.patch, h1606_20110408.patch, h1606_20110408b.patch > > > In the current design, if there is a datanode/network failure in the write > pipeline, DFSClient will try to remove the failed datanode from the pipeline > and then continue writing with the remaining datanodes. As a result, the > number of datanodes in the pipeline is decreased. Unfortunately, it is > possible that DFSClient may incorrectly remove a healthy datanode but leave > the failed datanode in the pipeline because failure detection may be > inaccurate under erroneous conditions. > We propose to have a new mechanism for adding new datanodes to the pipeline > in order to provide a stronger data guarantee. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1822) Editlog opcodes overlap between 20 security and later releases
[ https://issues.apache.org/jira/browse/HDFS-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017798#comment-13017798 ] Allen Wittenauer commented on HDFS-1822: bq. Have we made any apache release off 0.20.security? No, *Apache* has not made a release off this branch. The fact that other distributions have is irrelevant. > Editlog opcodes overlap between 20 security and later releases > -- > > Key: HDFS-1822 > URL: https://issues.apache.org/jira/browse/HDFS-1822 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0, 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > Attachments: HDFS-1822.patch > > > Same opcode are used for different operations between 0.20.security, 0.22 and > 0.23. This results in failure to load editlogs on later release, especially > during upgrades. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1822) Editlog opcodes overlap between 20 security and later releases
[ https://issues.apache.org/jira/browse/HDFS-1822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13017805#comment-13017805 ] Suresh Srinivas commented on HDFS-1822: --- > Apache has not made a release off this branch. Sure. Still there are users using 20 branches of Hadoop, even though it is not Apache released! I think this is an important change for the community. > Editlog opcodes overlap between 20 security and later releases > -- > > Key: HDFS-1822 > URL: https://issues.apache.org/jira/browse/HDFS-1822 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.21.0, 0.22.0, 0.23.0 >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas >Priority: Blocker > Fix For: 0.22.0, 0.23.0 > > Attachments: HDFS-1822.patch > > > Same opcode are used for different operations between 0.20.security, 0.22 and > 0.23. This results in failure to load editlogs on later release, especially > during upgrades. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
