[jira] Updated: (HDFS-970) FSImage writing should always fsync before close
[ https://issues.apache.org/jira/browse/HDFS-970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-970: - Attachment: hdfs-970.txt Trivial patch to add fsyncs to the image saving, fstime writing, and TransferFsImage code. There are no tests included, since the only way to test this is to pull a power plug :) There is probably some negative performance impact, depending on size of files, dirty page limits, available RAM, etc, but I think the safety factor is well worth it! FSImage writing should always fsync before close Key: HDFS-970 URL: https://issues.apache.org/jira/browse/HDFS-970 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hdfs-970.txt Without an fsync, it's common that filesystems will delay the writing of metadata to the journal until all of the data blocks have been flushed. If the system crashes while the dirty pages haven't been flushed, the file is left in an indeterminate state. In some FSs (eg ext4) this will result in a 0-length file. In others (eg XFS) it will result in the correct length but any number of data blocks getting zeroed. Calling FileChannel.force before closing the FSImage prevents this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-970) FSImage writing should always fsync before close
[ https://issues.apache.org/jira/browse/HDFS-970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-970: - Status: Patch Available (was: Open) FSImage writing should always fsync before close Key: HDFS-970 URL: https://issues.apache.org/jira/browse/HDFS-970 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hdfs-970.txt Without an fsync, it's common that filesystems will delay the writing of metadata to the journal until all of the data blocks have been flushed. If the system crashes while the dirty pages haven't been flushed, the file is left in an indeterminate state. In some FSs (eg ext4) this will result in a 0-length file. In others (eg XFS) it will result in the correct length but any number of data blocks getting zeroed. Calling FileChannel.force before closing the FSImage prevents this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-556) Provide info on failed volumes in the web ui
[ https://issues.apache.org/jira/browse/HDFS-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866074#action_12866074 ] dhruba borthakur commented on HDFS-556: --- This will be very helpful for system administrators! Provide info on failed volumes in the web ui Key: HDFS-556 URL: https://issues.apache.org/jira/browse/HDFS-556 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jakob Homan Assignee: Eli Collins Attachments: hdfs-556-1.patch HDFS-457 provided better handling of failed volumes but did not provide a corresponding view of this functionality on the web ui, such as a view of which datanodes have failed volumes. This would be a good feature to have. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1141) completeFile does not check lease ownership
[ https://issues.apache.org/jira/browse/HDFS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866075#action_12866075 ] dhruba borthakur commented on HDFS-1141: +1, code looks good. I will commit in a day unless somebody else has any other review comments. completeFile does not check lease ownership --- Key: HDFS-1141 URL: https://issues.apache.org/jira/browse/HDFS-1141 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1141-branch20.txt, hdfs-1141.txt completeFile should check that the caller still owns the lease of the file that it's completing. This is for the 'testCompleteOtherLeaseHoldersFile' case in HDFS-1139. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-988) saveNamespace can corrupt edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866081#action_12866081 ] dhruba borthakur commented on HDFS-988: --- Code looks excellent. can we add a unit test that does the following: * Create a cluster, open a file for write and and put in safemode. verify that getAdditionalBlock, close etc fails when NN is in safemode saveNamespace can corrupt edits log --- Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-988.txt, saveNamespace.txt The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-970) FSImage writing should always fsync before close
[ https://issues.apache.org/jira/browse/HDFS-970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866115#action_12866115 ] Hadoop QA commented on HDFS-970: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444191/hdfs-970.txt against trunk revision 942863. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/353/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/353/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/353/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/353/console This message is automatically generated. FSImage writing should always fsync before close Key: HDFS-970 URL: https://issues.apache.org/jira/browse/HDFS-970 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.20.1, 0.21.0, 0.22.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hdfs-970.txt Without an fsync, it's common that filesystems will delay the writing of metadata to the journal until all of the data blocks have been flushed. If the system crashes while the dirty pages haven't been flushed, the file is left in an indeterminate state. In some FSs (eg ext4) this will result in a 0-length file. In others (eg XFS) it will result in the correct length but any number of data blocks getting zeroed. Calling FileChannel.force before closing the FSImage prevents this issue. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1141) completeFile does not check lease ownership
[
https://issues.apache.org/jira/browse/HDFS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866176#action_12866176
]
Tsz Wo (Nicholas), SZE commented on HDFS-1141:
--
+1 thanks for working on it, Todd.
Minor: Look like that pendingFile won't be null since the checkLease(..) method
below will throw a LeaseExpiredException when file == null.
{code}
//FSNamesystem
private void checkLease(String src, String holder, INode file)
throws IOException {
if (file == null || file.isDirectory()) {
Lease lease = leaseManager.getLease(holder);
throw new LeaseExpiredException(No lease on + src +
...
{code}
completeFile does not check lease ownership
---
Key: HDFS-1141
URL: https://issues.apache.org/jira/browse/HDFS-1141
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.21.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Blocker
Attachments: hdfs-1141-branch20.txt, hdfs-1141.txt
completeFile should check that the caller still owns the lease of the file
that it's completing. This is for the 'testCompleteOtherLeaseHoldersFile'
case in HDFS-1139.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1141) completeFile does not check lease ownership
[ https://issues.apache.org/jira/browse/HDFS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1141: -- Attachment: hdfs-1141.txt Updated patch based on Nicholas's feedback. I also removed CompleteFileStatus entirely, since we were only using the success/wait codes, it's easier to just use boolean. completeFile does not check lease ownership --- Key: HDFS-1141 URL: https://issues.apache.org/jira/browse/HDFS-1141 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1141-branch20.txt, hdfs-1141.txt, hdfs-1141.txt completeFile should check that the caller still owns the lease of the file that it's completing. This is for the 'testCompleteOtherLeaseHoldersFile' case in HDFS-1139. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1141) completeFile does not check lease ownership
[ https://issues.apache.org/jira/browse/HDFS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1141: -- Status: Open (was: Patch Available) completeFile does not check lease ownership --- Key: HDFS-1141 URL: https://issues.apache.org/jira/browse/HDFS-1141 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1141-branch20.txt, hdfs-1141.txt, hdfs-1141.txt completeFile should check that the caller still owns the lease of the file that it's completing. This is for the 'testCompleteOtherLeaseHoldersFile' case in HDFS-1139. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1141) completeFile does not check lease ownership
[ https://issues.apache.org/jira/browse/HDFS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1141: -- Status: Patch Available (was: Open) completeFile does not check lease ownership --- Key: HDFS-1141 URL: https://issues.apache.org/jira/browse/HDFS-1141 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1141-branch20.txt, hdfs-1141.txt, hdfs-1141.txt completeFile should check that the caller still owns the lease of the file that it's completing. This is for the 'testCompleteOtherLeaseHoldersFile' case in HDFS-1139. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-556) Provide info on failed volumes in the web ui
[ https://issues.apache.org/jira/browse/HDFS-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866222#action_12866222 ] Andrew Ryan commented on HDFS-556: -- For sysadmins of clusters of any size, providing this in a web page is interesting but not useful for other scripts to consume. I've already had to write BeautifulSoup parsers to grab other data of interest and it's suboptimal. So I would really like to see a way to get this data as some sort of structured text. Perhaps if there was a command to fetch the names of the nodes with failed drives or something. Also, does this data appear in dfsadmin -report? That's pretty unstructured, but if it shows up on the UI, it should be there too. Provide info on failed volumes in the web ui Key: HDFS-556 URL: https://issues.apache.org/jira/browse/HDFS-556 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jakob Homan Assignee: Eli Collins Attachments: hdfs-556-1.patch HDFS-457 provided better handling of failed volumes but did not provide a corresponding view of this functionality on the web ui, such as a view of which datanodes have failed volumes. This would be a good feature to have. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1140) Speedup INode.getPathComponents
[ https://issues.apache.org/jira/browse/HDFS-1140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866224#action_12866224 ] Hairong Kuang commented on HDFS-1140: - All paths are stored as bytes in memory. In theory, we do not need to convert bytes to string and then to bytes when loading fsimage. But this needs a lot of re-organization of our code. Speedup INode.getPathComponents --- Key: HDFS-1140 URL: https://issues.apache.org/jira/browse/HDFS-1140 Project: Hadoop HDFS Issue Type: Bug Reporter: Dmytro Molkov Assignee: Dmytro Molkov Attachments: HDFS-1140.patch When the namenode is loading the image there is a significant amount of time being spent in the DFSUtil.string2Bytes. We have a very specific workload here. The path that namenode does getPathComponents for shares N - 1 component with the previous path this method was called for (assuming current path has N components). Hence we can improve the image load time by caching the result of previous conversion. We thought of using some simple LRU cache for components, but the reality is, String.getBytes gets optimized during runtime and LRU cache doesn't perform as well, however using just the latest path components and their translation to bytes in two arrays gives quite a performance boost. I could get another 20% off of the time to load the image on our cluster (30 seconds vs 24) and I wrote a simple benchmark that tests performance with and without caching. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-556) Provide info on failed volumes in the web ui
[ https://issues.apache.org/jira/browse/HDFS-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866227#action_12866227 ] dhruba borthakur commented on HDFS-556: --- @andrew: how about if we put this info in the webI as well dfsadmin -report? will that satisfy you? Provide info on failed volumes in the web ui Key: HDFS-556 URL: https://issues.apache.org/jira/browse/HDFS-556 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jakob Homan Assignee: Eli Collins Attachments: hdfs-556-1.patch HDFS-457 provided better handling of failed volumes but did not provide a corresponding view of this functionality on the web ui, such as a view of which datanodes have failed volumes. This would be a good feature to have. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-556) Provide info on failed volumes in the web ui
[ https://issues.apache.org/jira/browse/HDFS-556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866228#action_12866228 ] Eli Collins commented on HDFS-556: -- @Andrew: I added a volumes failed metric in HDFS-811 for this reason. This way you can get the data programatically or monitor via eg Ganglia. Provide info on failed volumes in the web ui Key: HDFS-556 URL: https://issues.apache.org/jira/browse/HDFS-556 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jakob Homan Assignee: Eli Collins Attachments: hdfs-556-1.patch HDFS-457 provided better handling of failed volumes but did not provide a corresponding view of this functionality on the web ui, such as a view of which datanodes have failed volumes. This would be a good feature to have. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1141) completeFile does not check lease ownership
[ https://issues.apache.org/jira/browse/HDFS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866233#action_12866233 ] Tsz Wo (Nicholas), SZE commented on HDFS-1141: -- Not sure what you're referring to ... Oops, I mixed up pendingFile and fileBlocks. I just have checked the codes. fileBlocks also won't possibly be null, given that pendingFile is an INodeFileUnderConstruction. +1 the new patch looks good. completeFile does not check lease ownership --- Key: HDFS-1141 URL: https://issues.apache.org/jira/browse/HDFS-1141 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1141-branch20.txt, hdfs-1141.txt, hdfs-1141.txt completeFile should check that the caller still owns the lease of the file that it's completing. This is for the 'testCompleteOtherLeaseHoldersFile' case in HDFS-1139. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1061) Memory footprint optimization for INodeFile object.
[ https://issues.apache.org/jira/browse/HDFS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1061: -- Hadoop Flags: [Incompatible change, Reviewed] Contrib test failure unrelated. +1. Marking as incompatible change due to more strict bounds checking. Memory footprint optimization for INodeFile object. Key: HDFS-1061 URL: https://issues.apache.org/jira/browse/HDFS-1061 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.22.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1061-1.patch, HDFS-1061-2.patch, HDFS-1061-3.patch, HDFS-1061.patch I am proposing a footprint optimization to merge blockReplication and preferredBlockSize fields into one 'long header' field in INodeFile class. This saves 8 bytes per INodeFile object on a 64 bit JVM. This memory optimization is transparent and changes are very minimal. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1061) Memory footprint optimization for INodeFile object.
[ https://issues.apache.org/jira/browse/HDFS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1061: -- Status: Resolved (was: Patch Available) Resolution: Fixed I've just committed this. Thanks, Bharath! Resolving as fixed. Memory footprint optimization for INodeFile object. Key: HDFS-1061 URL: https://issues.apache.org/jira/browse/HDFS-1061 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.22.0 Reporter: Bharath Mundlapudi Assignee: Bharath Mundlapudi Priority: Minor Fix For: 0.22.0 Attachments: HDFS-1061-1.patch, HDFS-1061-2.patch, HDFS-1061-3.patch, HDFS-1061.patch I am proposing a footprint optimization to merge blockReplication and preferredBlockSize fields into one 'long header' field in INodeFile class. This saves 8 bytes per INodeFile object on a 64 bit JVM. This memory optimization is transparent and changes are very minimal. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1141) completeFile does not check lease ownership
[ https://issues.apache.org/jira/browse/HDFS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866256#action_12866256 ] Hadoop QA commented on HDFS-1141: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444208/hdfs-1141.txt against trunk revision 942863. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/354/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/354/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/354/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/354/console This message is automatically generated. completeFile does not check lease ownership --- Key: HDFS-1141 URL: https://issues.apache.org/jira/browse/HDFS-1141 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1141-branch20.txt, hdfs-1141.txt, hdfs-1141.txt completeFile should check that the caller still owns the lease of the file that it's completing. This is for the 'testCompleteOtherLeaseHoldersFile' case in HDFS-1139. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-218) name node should provide status of dfs.name.dir's
[ https://issues.apache.org/jira/browse/HDFS-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ravi Phulari updated HDFS-218: -- Status: Patch Available (was: Open) Resubmitting patch for hudson queue. name node should provide status of dfs.name.dir's - Key: HDFS-218 URL: https://issues.apache.org/jira/browse/HDFS-218 Project: Hadoop HDFS Issue Type: New Feature Reporter: Allen Wittenauer Assignee: Ravi Phulari Attachments: HDFS-218-trunk.patch, HDFS-218.patch We've had several reports of people letting their dfs.name.dir fill up. To help prevent this, the name node web ui and perhaps dfsadmin -report or another command should give a disk space report of all dfs.name.dir's as well as whether or not the contents of that dir are actually being used, if the copy is good, last 2ndary name node update, and any thing else that might be useful. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1143) Implement Background deletion
Implement Background deletion - Key: HDFS-1143 URL: https://issues.apache.org/jira/browse/HDFS-1143 Project: Hadoop HDFS Issue Type: Improvement Reporter: Dmytro Molkov Assignee: Scott Chen Right now if you try to delete massive number of files from the namenode it will freeze (sometimes for minutes). Most of the time is spent going through the blocks map and invalidating all the blocks. This can probably be improved by having a background GC process. The deletion will basically just remove the inode being deleted and then give the subtree that was just deleted to the background thread running cleanup. This way the namenode becomes available for the clients soon after deletion, and all the heavy operations are done in the background. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1138) Modification times are being overwritten when FSImage loads
[ https://issues.apache.org/jira/browse/HDFS-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866274#action_12866274 ] Eli Collins commented on HDFS-1138: --- Nice catch Dmytro! Patch looks good to me. * I'd remove the new addChild method and make both callers explicitly pass whether they propagate the modification time. * Would update the comment from update modification time of the parent directory to something like Optionally update the parent's modification time, we shouldn't if we're just loading an image. Modification times are being overwritten when FSImage loads --- Key: HDFS-1138 URL: https://issues.apache.org/jira/browse/HDFS-1138 Project: Hadoop HDFS Issue Type: Bug Reporter: Dmytro Molkov Assignee: Dmytro Molkov Attachments: HDFS-1138 A very easy way to spot the bug is to do a second restart in TestRestartDFS and check that the modification time on root is the same as it was before the second restart. The problem is modifying time of the parent if the modification time of the child is greater than parent's in addToParent. So if you have /DIR/File then on creation of a file modification time of the DIR will be set, but on cluster restart, or when secondary is checkpointing and reading the image it will add DIR to / and write the new modification time for / which is the modification time of DIR. This is clearly a bug. I will attach a patch with one more parameter being passed from the loadFSImage that says to not propagate the time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1144) Improve Trash Emptier
Improve Trash Emptier - Key: HDFS-1144 URL: https://issues.apache.org/jira/browse/HDFS-1144 Project: Hadoop HDFS Issue Type: Improvement Reporter: Dmytro Molkov Assignee: Dmytro Molkov There are two inefficiencies in the Trash functionality right now that have caused some problems for us. First if you configured your trash interval to be one day (24 hours) that means that you store 2 days worth of data eventually. The Current and the previous timestamp that will not be deleted until the end of the interval. And another problem is accumulating a lot of data in Trash before the Emptier wakes up. If there are a couple of million files trashed and the Emptier does deletion the NameNode will freeze until everything is removed. (this particular problem hopefully will be addressed with HDFS-1143). My proposal is to have two configuration intervals. One for deleting the trashed data and another for checkpointing. This way for example for intervals of one day and one hour we will only store 25 hours of data instead of 48 right now and the deletions will be happening in smaller chunks every hour of the day instead of a huge deletion at the end of the day now. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1145) When NameNode is shutdown it tries to exit safemode
When NameNode is shutdown it tries to exit safemode --- Key: HDFS-1145 URL: https://issues.apache.org/jira/browse/HDFS-1145 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: dhruba borthakur Suppose the NameNode is in safemode. Then we try to shuut it down by invoking NameNode.stop(). The stop() method interrupts all waiting threads, which in turn, causes the SafeMode monitor to exit and thus triggering replicating/deleting of blocks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1145) When NameNode is shutdown it tries to exit safemode
[ https://issues.apache.org/jira/browse/HDFS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur reassigned HDFS-1145: -- Assignee: dhruba borthakur When NameNode is shutdown it tries to exit safemode --- Key: HDFS-1145 URL: https://issues.apache.org/jira/browse/HDFS-1145 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: NNsafemodeMonitor.txt Suppose the NameNode is in safemode. Then we try to shuut it down by invoking NameNode.stop(). The stop() method interrupts all waiting threads, which in turn, causes the SafeMode monitor to exit and thus triggering replicating/deleting of blocks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1145) When NameNode is shutdown it tries to exit safemode
[ https://issues.apache.org/jira/browse/HDFS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866298#action_12866298 ] Ravi Phulari commented on HDFS-1145: Good catch Dhruba, One minor correction, Could you please correct errors in Log message. bq.NameNode isbeing shutdow, quitting SafeModeMonitor thread. When NameNode is shutdown it tries to exit safemode --- Key: HDFS-1145 URL: https://issues.apache.org/jira/browse/HDFS-1145 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: NNsafemodeMonitor.txt Suppose the NameNode is in safemode. Then we try to shuut it down by invoking NameNode.stop(). The stop() method interrupts all waiting threads, which in turn, causes the SafeMode monitor to exit and thus triggering replicating/deleting of blocks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1138) Modification times are being overwritten when FSImage loads
[ https://issues.apache.org/jira/browse/HDFS-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-1138: Attachment: HDFS-1138.2.patch This one addresses Eli's comments. Modification times are being overwritten when FSImage loads --- Key: HDFS-1138 URL: https://issues.apache.org/jira/browse/HDFS-1138 Project: Hadoop HDFS Issue Type: Bug Reporter: Dmytro Molkov Assignee: Dmytro Molkov Attachments: HDFS-1138, HDFS-1138.2.patch A very easy way to spot the bug is to do a second restart in TestRestartDFS and check that the modification time on root is the same as it was before the second restart. The problem is modifying time of the parent if the modification time of the child is greater than parent's in addToParent. So if you have /DIR/File then on creation of a file modification time of the DIR will be set, but on cluster restart, or when secondary is checkpointing and reading the image it will add DIR to / and write the new modification time for / which is the modification time of DIR. This is clearly a bug. I will attach a patch with one more parameter being passed from the loadFSImage that says to not propagate the time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1145) When NameNode is shutdown it tries to exit safemode
[ https://issues.apache.org/jira/browse/HDFS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-1145: --- Attachment: NNsafemodeMonitor.txt I had attached an older version of this patch. here is the most updated version. When NameNode is shutdown it tries to exit safemode --- Key: HDFS-1145 URL: https://issues.apache.org/jira/browse/HDFS-1145 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: NNsafemodeMonitor.txt, NNsafemodeMonitor.txt Suppose the NameNode is in safemode. Then we try to shuut it down by invoking NameNode.stop(). The stop() method interrupts all waiting threads, which in turn, causes the SafeMode monitor to exit and thus triggering replicating/deleting of blocks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1141) completeFile does not check lease ownership
[ https://issues.apache.org/jira/browse/HDFS-1141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866319#action_12866319 ] Todd Lipcon commented on HDFS-1141: --- The failed tests are unrelated (TestNodeCounts passes locally). I'll file a JIRA for the TestNodeCounts failure separately if there isn't already one, since it's rather worrisome! completeFile does not check lease ownership --- Key: HDFS-1141 URL: https://issues.apache.org/jira/browse/HDFS-1141 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1141-branch20.txt, hdfs-1141.txt, hdfs-1141.txt completeFile should check that the caller still owns the lease of the file that it's completing. This is for the 'testCompleteOtherLeaseHoldersFile' case in HDFS-1139. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-889) Possible race condition in BlocksMap.NodeIterator.
[
https://issues.apache.org/jira/browse/HDFS-889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866322#action_12866322
]
Todd Lipcon commented on HDFS-889:
--
Is this just a test bug? i.e is the contract of BlockManager that its methods
require the FSN lock to be held, and the test is at fault for not doing so? Or
do we have other cases in the NN where we access these iterators w/o
synchronization
Possible race condition in BlocksMap.NodeIterator.
--
Key: HDFS-889
URL: https://issues.apache.org/jira/browse/HDFS-889
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.22.0
Reporter: Steve Loughran
Hudson's test run for HDFS-165 is showing an NPE in
{{org.apache.hadoop.hdfs.server.namenode.TestNodeCount.testNodeCount()}}
One problem could be in {{BlocksMap.NodeIterator}}. It's {{hasNext()}} method
checks the next entry isn't null. But what if between the {{hasNext() call
and the next() operation, the map changes and an entry goes away? In that
situation, the node returned from next() will be null.
This is potentially serious as a quick look through the code shows that the
iterator gets retrieved a lot and everywhere hadoop does so, it assumes the
value is not null. It's also one of those problems that doesn't have a simple
make it go away fix.
Options
# Ignore it, hope it doesn't happen very often and the test failing was a one
off that will never happen in a production datacentre. This is the default.
The iterator is only used in the namenode, so while it does depend on the #
of datanodes, it isn't running in 4000 machines in a big cluster.
# Leave the iterator as is, have all the in-Hadoop code check for a
null-value and break the loop
# Patch the {{NodeIterator}} to be consistent with the {{Iterator}}
specification and throw a {{NoSuchElementException}} if the next value is
null. This does not make the problem go away, but now it is handled by having
every use in-Hadoop catching the exception at the right point and exiting the
loop.
Testing. This should be possible.
# Create a block map
# iterate over a block
# while the iterator is in progress remove the next block in the list. Expect
the next call to next() to fail in whatever way you choose.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1103) Replica recovery doesn't distinguish between flushed-but-corrupted last chunk and unflushed last chunk
[ https://issues.apache.org/jira/browse/HDFS-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866329#action_12866329 ] Todd Lipcon commented on HDFS-1103: --- I worked on this a bit, and think we have a few options: 1) Assume that corruption of the last checksum chunk always represents an invalid checksum and possibly-truncated data, but the bytes themselves are OK in the data file. In this case, we can use the following algorithm: - For each replica, run validateIntegrity to determine the length of bytes that fit the checksum - Determine the maximum valid length across all replicas - For any replica that has at least that number of bytes on disk (regardless of validity), include it in the synchronization 2) Assume that chunks that aren't validated are corrupt in either the checksum or the data In this case, we take the max of the valid lengths, and only include blocks in synchronization when they have at least this many valid bytes. Thus, in the case where we have two replicas, one of which is potentially corrupt at the end, we'd end up only keeping the valid one. We can still lose flushed data in the case of multiple datanode failure, if both of the DNs end up with corrupt last chunks (unlikely but possible) 3) Ignore checksums completely for the last chunk We can entirely throw away the checksums on the last chunk, and augment ReplicaRecoveryInfo to actually include a BytesWritable of the last data chunk (only 512 bytes by default). During the recovery operation, we can simply find the common prefix of data across all of the replicas, and synchronize to that length (recomputing checksum on each node). Since we're only talking about one checksum chunk, and these are known to be small, I don't think it's problematic to shove the bytes in an RPC. Thoughts? Replica recovery doesn't distinguish between flushed-but-corrupted last chunk and unflushed last chunk -- Key: HDFS-1103 URL: https://issues.apache.org/jira/browse/HDFS-1103 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.21.0, 0.22.0 Reporter: Todd Lipcon Priority: Blocker Attachments: hdfs-1103-test.txt When the DN creates a replica under recovery, it calls validateIntegrity, which truncates the last checksum chunk off of a replica if it is found to be invalid. Then when the block recovery process happens, this shortened block wins over a longer replica from another node where there was no corruption. Thus, if just one of the DNs has an invalid last checksum chunk, data that has been sync()ed to other datanodes can be lost. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1143) Implement Background deletion
[ https://issues.apache.org/jira/browse/HDFS-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866330#action_12866330 ] dhruba borthakur commented on HDFS-1143: One option is to remove the inode from the Directory tree, keep the subtree aside and release the global lock. Then traverse the subtree, for each node in the subtree : 1. acquire the global lock 2. remove all the blocks associated with that node from the BlocksMap 3. release the lock. Implement Background deletion - Key: HDFS-1143 URL: https://issues.apache.org/jira/browse/HDFS-1143 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.22.0 Reporter: Dmytro Molkov Assignee: Scott Chen Fix For: 0.22.0 Right now if you try to delete massive number of files from the namenode it will freeze (sometimes for minutes). Most of the time is spent going through the blocks map and invalidating all the blocks. This can probably be improved by having a background GC process. The deletion will basically just remove the inode being deleted and then give the subtree that was just deleted to the background thread running cleanup. This way the namenode becomes available for the clients soon after deletion, and all the heavy operations are done in the background. Thoughts? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-1144) Improve Trash Emptier
[ https://issues.apache.org/jira/browse/HDFS-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov resolved HDFS-1144. - Resolution: Duplicate Since Trash is actually a part of Common I created HADOOP-6761 Improve Trash Emptier - Key: HDFS-1144 URL: https://issues.apache.org/jira/browse/HDFS-1144 Project: Hadoop HDFS Issue Type: Improvement Reporter: Dmytro Molkov Assignee: Dmytro Molkov There are two inefficiencies in the Trash functionality right now that have caused some problems for us. First if you configured your trash interval to be one day (24 hours) that means that you store 2 days worth of data eventually. The Current and the previous timestamp that will not be deleted until the end of the interval. And another problem is accumulating a lot of data in Trash before the Emptier wakes up. If there are a couple of million files trashed and the Emptier does deletion the NameNode will freeze until everything is removed. (this particular problem hopefully will be addressed with HDFS-1143). My proposal is to have two configuration intervals. One for deleting the trashed data and another for checkpointing. This way for example for intervals of one day and one hour we will only store 25 hours of data instead of 48 right now and the deletions will be happening in smaller chunks every hour of the day instead of a huge deletion at the end of the day now. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1142) Lease recovery doesn't reassign lease when triggered by append()
[ https://issues.apache.org/jira/browse/HDFS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1142: -- Attachment: hdfs-1142.txt New patch on top of trunk (since HDFS-1141 got committed) Lease recovery doesn't reassign lease when triggered by append() Key: HDFS-1142 URL: https://issues.apache.org/jira/browse/HDFS-1142 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1142.txt, hdfs-1142.txt If a soft lease has expired and another writer calls append(), it triggers lease recovery but doesn't reassign the lease to a new owner. Therefore, the old writer can continue to allocate new blocks, try to steal back the lease, etc. This is for the testRecoveryOnBlockBoundary case of HDFS-1139 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1142) Lease recovery doesn't reassign lease when triggered by append()
[ https://issues.apache.org/jira/browse/HDFS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1142: -- Status: Patch Available (was: Open) Lease recovery doesn't reassign lease when triggered by append() Key: HDFS-1142 URL: https://issues.apache.org/jira/browse/HDFS-1142 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1142.txt, hdfs-1142.txt If a soft lease has expired and another writer calls append(), it triggers lease recovery but doesn't reassign the lease to a new owner. Therefore, the old writer can continue to allocate new blocks, try to steal back the lease, etc. This is for the testRecoveryOnBlockBoundary case of HDFS-1139 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1138) Modification times are being overwritten when FSImage loads
[ https://issues.apache.org/jira/browse/HDFS-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866347#action_12866347 ] Eli Collins commented on HDFS-1138: --- +1 Patch looks good. Modification times are being overwritten when FSImage loads --- Key: HDFS-1138 URL: https://issues.apache.org/jira/browse/HDFS-1138 Project: Hadoop HDFS Issue Type: Bug Reporter: Dmytro Molkov Assignee: Dmytro Molkov Attachments: HDFS-1138, HDFS-1138.2.patch A very easy way to spot the bug is to do a second restart in TestRestartDFS and check that the modification time on root is the same as it was before the second restart. The problem is modifying time of the parent if the modification time of the child is greater than parent's in addToParent. So if you have /DIR/File then on creation of a file modification time of the DIR will be set, but on cluster restart, or when secondary is checkpointing and reading the image it will add DIR to / and write the new modification time for / which is the modification time of DIR. This is clearly a bug. I will attach a patch with one more parameter being passed from the loadFSImage that says to not propagate the time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-988) saveNamespace can corrupt edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-988: -- Status: Open (was: Patch Available) hi todd, can we get a unit test for this one (as described in the earlier comment)? Thanks. saveNamespace can corrupt edits log --- Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0, 0.22.0 Reporter: dhruba borthakur Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-988.txt, saveNamespace.txt The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-218) name node should provide status of dfs.name.dir's
[ https://issues.apache.org/jira/browse/HDFS-218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866357#action_12866357 ] Hadoop QA commented on HDFS-218: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444167/HDFS-218-trunk.patch against trunk revision 943214. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/355/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/355/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/355/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/355/console This message is automatically generated. name node should provide status of dfs.name.dir's - Key: HDFS-218 URL: https://issues.apache.org/jira/browse/HDFS-218 Project: Hadoop HDFS Issue Type: New Feature Reporter: Allen Wittenauer Assignee: Ravi Phulari Attachments: HDFS-218-trunk.patch, HDFS-218.patch We've had several reports of people letting their dfs.name.dir fill up. To help prevent this, the name node web ui and perhaps dfsadmin -report or another command should give a disk space report of all dfs.name.dir's as well as whether or not the contents of that dir are actually being used, if the copy is good, last 2ndary name node update, and any thing else that might be useful. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1138) Modification times are being overwritten when FSImage loads
[ https://issues.apache.org/jira/browse/HDFS-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-1138: --- Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Fix Version/s: 0.22.0 Resolution: Fixed I just committed this. Thanks Dmytro! Modification times are being overwritten when FSImage loads --- Key: HDFS-1138 URL: https://issues.apache.org/jira/browse/HDFS-1138 Project: Hadoop HDFS Issue Type: Bug Reporter: Dmytro Molkov Assignee: Dmytro Molkov Fix For: 0.22.0 Attachments: HDFS-1138, HDFS-1138.2.patch A very easy way to spot the bug is to do a second restart in TestRestartDFS and check that the modification time on root is the same as it was before the second restart. The problem is modifying time of the parent if the modification time of the child is greater than parent's in addToParent. So if you have /DIR/File then on creation of a file modification time of the DIR will be set, but on cluster restart, or when secondary is checkpointing and reading the image it will add DIR to / and write the new modification time for / which is the modification time of DIR. This is clearly a bug. I will attach a patch with one more parameter being passed from the loadFSImage that says to not propagate the time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1142) Lease recovery doesn't reassign lease when triggered by append()
[ https://issues.apache.org/jira/browse/HDFS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866361#action_12866361 ] dhruba borthakur commented on HDFS-1142: can we re-run hadoopQA tests on the new patch? Lease recovery doesn't reassign lease when triggered by append() Key: HDFS-1142 URL: https://issues.apache.org/jira/browse/HDFS-1142 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1142.txt, hdfs-1142.txt If a soft lease has expired and another writer calls append(), it triggers lease recovery but doesn't reassign the lease to a new owner. Therefore, the old writer can continue to allocate new blocks, try to steal back the lease, etc. This is for the testRecoveryOnBlockBoundary case of HDFS-1139 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1142) Lease recovery doesn't reassign lease when triggered by append()
[ https://issues.apache.org/jira/browse/HDFS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866362#action_12866362 ] Todd Lipcon commented on HDFS-1142: --- Yea, I toggled patch available when I uploaded, should be on the queue. Lease recovery doesn't reassign lease when triggered by append() Key: HDFS-1142 URL: https://issues.apache.org/jira/browse/HDFS-1142 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1142.txt, hdfs-1142.txt If a soft lease has expired and another writer calls append(), it triggers lease recovery but doesn't reassign the lease to a new owner. Therefore, the old writer can continue to allocate new blocks, try to steal back the lease, etc. This is for the testRecoveryOnBlockBoundary case of HDFS-1139 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1053) A client side mount table to give per-application/per-job file system view
[ https://issues.apache.org/jira/browse/HDFS-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866371#action_12866371 ] Milind Bhandarkar commented on HDFS-1053: - Sanjay, do you have a proposal / requirements doc for this ? A client side mount table to give per-application/per-job file system view -- Key: HDFS-1053 URL: https://issues.apache.org/jira/browse/HDFS-1053 Project: Hadoop HDFS Issue Type: New Feature Affects Versions: 0.22.0 Reporter: Sanjay Radia Assignee: Sanjay Radia Fix For: 0.22.0 This jira proposes a client side mount table to allow application-centric (or job-centric) filesystem views. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1138) Modification times are being overwritten when FSImage loads
[ https://issues.apache.org/jira/browse/HDFS-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866375#action_12866375 ] Hadoop QA commented on HDFS-1138: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444237/HDFS-1138.2.patch against trunk revision 943214. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/176/console This message is automatically generated. Modification times are being overwritten when FSImage loads --- Key: HDFS-1138 URL: https://issues.apache.org/jira/browse/HDFS-1138 Project: Hadoop HDFS Issue Type: Bug Reporter: Dmytro Molkov Assignee: Dmytro Molkov Fix For: 0.22.0 Attachments: HDFS-1138, HDFS-1138.2.patch A very easy way to spot the bug is to do a second restart in TestRestartDFS and check that the modification time on root is the same as it was before the second restart. The problem is modifying time of the parent if the modification time of the child is greater than parent's in addToParent. So if you have /DIR/File then on creation of a file modification time of the DIR will be set, but on cluster restart, or when secondary is checkpointing and reading the image it will add DIR to / and write the new modification time for / which is the modification time of DIR. This is clearly a bug. I will attach a patch with one more parameter being passed from the loadFSImage that says to not propagate the time. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1142) Lease recovery doesn't reassign lease when triggered by append()
[ https://issues.apache.org/jira/browse/HDFS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-1142: -- Status: Patch Available (was: Open) Toggling cuz Hudson build screwed up (silly NoClassDefFound thing, not this patch) Lease recovery doesn't reassign lease when triggered by append() Key: HDFS-1142 URL: https://issues.apache.org/jira/browse/HDFS-1142 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1142.txt, hdfs-1142.txt If a soft lease has expired and another writer calls append(), it triggers lease recovery but doesn't reassign the lease to a new owner. Therefore, the old writer can continue to allocate new blocks, try to steal back the lease, etc. This is for the testRecoveryOnBlockBoundary case of HDFS-1139 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-606) ConcurrentModificationException in invalidateCorruptReplicas()
[
https://issues.apache.org/jira/browse/HDFS-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-606:
-
Attachment: hdfs-606-branch20.txt
Ran into this issue stress testing the append work on branch 20. This patch
includes a test case that demonstrates the issue (test case depends on some of
the test utils from other branch 20 append patches)
ConcurrentModificationException in invalidateCorruptReplicas()
--
Key: HDFS-606
URL: https://issues.apache.org/jira/browse/HDFS-606
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.21.0
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.21.0
Attachments: CMEinCorruptReplicas.patch, hdfs-606-branch20.txt
{{BlockManager.invalidateCorruptReplicas()}} iterates over
DatanodeDescriptor-s while removing corrupt replicas from the descriptors.
This causes {{ConcurrentModificationException}} if there is more than one
replicas of the block. I ran into this exception debugging different
scenarios in append, but it should be fixed in the trunk too.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1146) Javadoc for getDelegationTokenSecretManager in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1146: --- Attachment: HDFS-1146-y20.1.patch patch for yahoo-20. Javadoc for getDelegationTokenSecretManager in FSNamesystem --- Key: HDFS-1146 URL: https://issues.apache.org/jira/browse/HDFS-1146 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1146-y20.1.patch Javadoc is missing for public method getDelegationTokenSecretManager in FSNamesystem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1003) authorization checks for inter-server protocol (based on HADOOP-6600)
[ https://issues.apache.org/jira/browse/HDFS-1003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866418#action_12866418 ] Jitendra Nath Pandey commented on HDFS-1003: +1 pending hudson tests. authorization checks for inter-server protocol (based on HADOOP-6600) - Key: HDFS-1003 URL: https://issues.apache.org/jira/browse/HDFS-1003 Project: Hadoop HDFS Issue Type: New Feature Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1003.patch authorization checks for inter-server protocol (based on HADOOP-6600) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1142) Lease recovery doesn't reassign lease when triggered by append()
[ https://issues.apache.org/jira/browse/HDFS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866424#action_12866424 ] Hadoop QA commented on HDFS-1142: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12444247/hdfs-1142.txt against trunk revision 943306. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/357/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/357/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/357/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/357/console This message is automatically generated. Lease recovery doesn't reassign lease when triggered by append() Key: HDFS-1142 URL: https://issues.apache.org/jira/browse/HDFS-1142 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1142.txt, hdfs-1142.txt If a soft lease has expired and another writer calls append(), it triggers lease recovery but doesn't reassign the lease to a new owner. Therefore, the old writer can continue to allocate new blocks, try to steal back the lease, etc. This is for the testRecoveryOnBlockBoundary case of HDFS-1139 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1142) Lease recovery doesn't reassign lease when triggered by append()
[ https://issues.apache.org/jira/browse/HDFS-1142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866436#action_12866436 ] dhruba borthakur commented on HDFS-1142: +1 code looks good to me. can somebody else review this patch too? Lease recovery doesn't reassign lease when triggered by append() Key: HDFS-1142 URL: https://issues.apache.org/jira/browse/HDFS-1142 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-1142.txt, hdfs-1142.txt If a soft lease has expired and another writer calls append(), it triggers lease recovery but doesn't reassign the lease to a new owner. Therefore, the old writer can continue to allocate new blocks, try to steal back the lease, etc. This is for the testRecoveryOnBlockBoundary case of HDFS-1139 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1145) When NameNode is shutdown it tries to exit safemode
[ https://issues.apache.org/jira/browse/HDFS-1145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866439#action_12866439 ] dhruba borthakur commented on HDFS-1145: Thanks Ravi, can you pl review the new patch file? When NameNode is shutdown it tries to exit safemode --- Key: HDFS-1145 URL: https://issues.apache.org/jira/browse/HDFS-1145 Project: Hadoop HDFS Issue Type: Bug Components: name-node Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: NNsafemodeMonitor.txt, NNsafemodeMonitor.txt Suppose the NameNode is in safemode. Then we try to shuut it down by invoking NameNode.stop(). The stop() method interrupts all waiting threads, which in turn, causes the SafeMode monitor to exit and thus triggering replicating/deleting of blocks. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1130) Pass Administrator acl to HTTPServer for common servlet access.
[ https://issues.apache.org/jira/browse/HDFS-1130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866446#action_12866446 ] Amareshwari Sriramadasu commented on HDFS-1130: --- A couple of comments on the patch: * hdfs_permissions_guide.xml and hdfs_default.xml has documentation for dfs.permissions.supergroup. It should be modified/removed. * TestMapredSystemDir sets a value for the configuration dfs.permissions.supergroup. The same should be removed. Other code changes look good to me. Pass Administrator acl to HTTPServer for common servlet access. --- Key: HDFS-1130 URL: https://issues.apache.org/jira/browse/HDFS-1130 Project: Hadoop HDFS Issue Type: Bug Components: security Reporter: Amareshwari Sriramadasu Fix For: 0.22.0 Attachments: hdfs-1130.patch Once HADOOP-6748 is done, HDFS should pass administrator acl when HTTPServer is constructed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-941) Datanode xceiver protocol should allow reuse of a connection
[ https://issues.apache.org/jira/browse/HDFS-941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866456#action_12866456 ] sam rash commented on HDFS-941: --- +1 for the idea of caching sockets, but I have some questions/concerns about the implementation. some comments: 1. avoid making the cache implementation tied to the class ReaderSocketCache. Don't make the cache a static member of the same class. Let the cache be an instantiable object. Let DFSClient store the cache either as an instance or static var (don't force everything to use the same cache instance--better for testing and stubbing out as well) 2. a lot of the logic around re-using is complicated--I think this could be simplified a. not clear why sockets are always in the cache even if not usable: i would think adding only when usable and removing when used would be cleaner? b. if we can keep the cache clean, no need for lazy removal of unusable sockets 3. shouldn't there be a cap on the # of sockets there can be in the cache? -again, should only be usable ones, but a max # put into the cache makes sense. If we have a flurry of reads using tons of sockets to several DNs, no need to keep 100s or more sockets in a cache 4. general concern about potential socket leaks; 5. seems like this needs more thought into the effects of synchronization: the freemap has to be traversed every time to get a socket in a sync block. see above if we can avoid lazy removal by not putting unusable sockets in the cache (unsuable either since they are in use or not usable at all) 6. do we have real performance benchmarks from actual clusters that show a significant benefit? as noted above, the change is fairly complex (caching is in fact hard :) and if we don't see a substantial performance improvement, the risk of bugs may outweigh the benefit that's my 2c anyway -sr Datanode xceiver protocol should allow reuse of a connection Key: HDFS-941 URL: https://issues.apache.org/jira/browse/HDFS-941 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, hdfs client Affects Versions: 0.22.0 Reporter: Todd Lipcon Assignee: bc Wong Attachments: HDFS-941-1.patch, HDFS-941-2.patch, HDFS-941-3.patch, HDFS-941-3.patch, HDFS-941-4.patch Right now each connection into the datanode xceiver only processes one operation. In the case that an operation leaves the stream in a well-defined state (eg a client reads to the end of a block successfully) the same connection could be reused for a second operation. This should improve random read performance significantly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1147) Reduce NN startup time by reducing the processing time of block reports
Reduce NN startup time by reducing the processing time of block reports --- Key: HDFS-1147 URL: https://issues.apache.org/jira/browse/HDFS-1147 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: dhruba borthakur The NameNode restart times are impacted to a large extent by the processing time of block reports. For a cluster with 150 millions blocks, the block report processing in the NN can take upto 20 minutes or so. The NN is open for business only after it has processed the block reports from most of the datanodes. If we can reduce the processing time of a block report, that will directly reduce the restart time of the NN. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-1147) Reduce NN startup time by reducing the processing time of block reports
[ https://issues.apache.org/jira/browse/HDFS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur reassigned HDFS-1147: -- Assignee: dhruba borthakur Reduce NN startup time by reducing the processing time of block reports --- Key: HDFS-1147 URL: https://issues.apache.org/jira/browse/HDFS-1147 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: dhruba borthakur Assignee: dhruba borthakur The NameNode restart times are impacted to a large extent by the processing time of block reports. For a cluster with 150 millions blocks, the block report processing in the NN can take upto 20 minutes or so. The NN is open for business only after it has processed the block reports from most of the datanodes. If we can reduce the processing time of a block report, that will directly reduce the restart time of the NN. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1147) Reduce NN startup time by reducing the processing time of block reports
[ https://issues.apache.org/jira/browse/HDFS-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12866474#action_12866474 ] dhruba borthakur commented on HDFS-1147: One proposal that I have is that the NN can short-circuit the processing time of a blockreport if it knows that this is the very first block report from a datanode. This is the typical case when NN restarts. My experiments indicate that the more than 50% of time is spent in generating the diff (node.reportDiff()) between the incoming block report and what is in the blocksMap. We can reduce this 50% overhead if the NN knows that this is the first ever block report from that datanode: instead of producing a diff, it can directly invoke addStoredBlock() on all the blocks in the incoming block report. This will effectively delay the deletion of blocks that do not belong to the namespace until the next block report, but that might be an acceptable tradeoff. If somebody can explain how this approach can be made to work in the presence of corrupt replicas, that will be great. Reduce NN startup time by reducing the processing time of block reports --- Key: HDFS-1147 URL: https://issues.apache.org/jira/browse/HDFS-1147 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: dhruba borthakur Assignee: dhruba borthakur The NameNode restart times are impacted to a large extent by the processing time of block reports. For a cluster with 150 millions blocks, the block report processing in the NN can take upto 20 minutes or so. The NN is open for business only after it has processed the block reports from most of the datanodes. If we can reduce the processing time of a block report, that will directly reduce the restart time of the NN. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
