[jira] Commented: (HDFS-1181) Move configuration and script files post split
[ https://issues.apache.org/jira/browse/HDFS-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874482#action_12874482 ] Vinod K V commented on HDFS-1181: - +1 for the latest patch. I could successfully bring up a single node HDFS cluster using the common patch and the one at HADOOP-6461. Move configuration and script files post split -- Key: HDFS-1181 URL: https://issues.apache.org/jira/browse/HDFS-1181 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.21.0 Attachments: HDFS-1181.patch, HDFS-1181.patch, HDFS-1181.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1181) Move configuration and script files post project split
[ https://issues.apache.org/jira/browse/HDFS-1181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod K V updated HDFS-1181: Summary: Move configuration and script files post project split (was: Move configuration and script files post split) Hadoop Flags: [Reviewed] Move configuration and script files post project split -- Key: HDFS-1181 URL: https://issues.apache.org/jira/browse/HDFS-1181 Project: Hadoop HDFS Issue Type: Bug Components: scripts Reporter: Tom White Assignee: Tom White Priority: Blocker Fix For: 0.21.0 Attachments: HDFS-1181.patch, HDFS-1181.patch, HDFS-1181.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874514#action_12874514 ] shravankumar commented on HDFS-503: --- Hello sir, 1. what is the meaning for this srcPath prefix=hdfs://dfs1.xxx.com:8000/user/dhruba/ 2. In ADMINISTRATION they mentioned RaidNode Software what it means. 3. In HADOOP_HOME, run ant package to build Hadoop and its contrib packages. This will come when we installed hadoop 0.20.1 or we need download ant package. Implement erasure coding as a layer on HDFS --- Key: HDFS-503 URL: https://issues.apache.org/jira/browse/HDFS-503 Project: Hadoop HDFS Issue Type: New Feature Components: contrib/raid Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: raid1.txt, raid2.txt The goal of this JIRA is to discuss how the cost of raw storage for a HDFS file system can be reduced. Keeping three copies of the same data is very costly, especially when the size of storage is huge. One idea is to reduce the replication factor and do erasure coding of a set of blocks so that the over probability of failure of a block remains the same as before. Many forms of error-correcting codes are available, see http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has described DiskReduce https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. My opinion is to discuss implementation strategies that are not part of base HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874518#action_12874518 ] shravankumar commented on HDFS-503: --- The tags(property,description) used in programming are normal HTML Tags or they have different meaning. Can you send me the document which consist of meanings of these tags. Implement erasure coding as a layer on HDFS --- Key: HDFS-503 URL: https://issues.apache.org/jira/browse/HDFS-503 Project: Hadoop HDFS Issue Type: New Feature Components: contrib/raid Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: raid1.txt, raid2.txt The goal of this JIRA is to discuss how the cost of raw storage for a HDFS file system can be reduced. Keeping three copies of the same data is very costly, especially when the size of storage is huge. One idea is to reduce the replication factor and do erasure coding of a set of blocks so that the over probability of failure of a block remains the same as before. Many forms of error-correcting codes are available, see http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has described DiskReduce https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. My opinion is to discuss implementation strategies that are not part of base HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1161) Make DN minimum valid volumes configurable
[ https://issues.apache.org/jira/browse/HDFS-1161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated HDFS-1161: Status: Resolved (was: Patch Available) Hadoop Flags: [Reviewed] Fix Version/s: (was: 0.22.0) Resolution: Fixed I've just committed this. Thanks Eli! Make DN minimum valid volumes configurable -- Key: HDFS-1161 URL: https://issues.apache.org/jira/browse/HDFS-1161 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.21.0, 0.22.0 Reporter: Eli Collins Assignee: Eli Collins Priority: Blocker Fix For: 0.21.0 Attachments: hdfs-1161-1.patch, hdfs-1161-2.patch, hdfs-1161-3.patch, hdfs-1161-4.patch, hdfs-1161-5.patch, hdfs-1161-6.patch The minimum number of non-faulty volumes to keep the DN active is hard-coded to 1. It would be useful to allow users to configure this value so the DN can be taken offline when eg half of its disks fail, otherwise it doesn't get reported until it's down to it's final disk and suffering degraded performance. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-142) In 0.20, move blocks being written into a blocksBeingWritten directory
[ https://issues.apache.org/jira/browse/HDFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-142: -- Affects Version/s: 0.20-append In 0.20, move blocks being written into a blocksBeingWritten directory -- Key: HDFS-142 URL: https://issues.apache.org/jira/browse/HDFS-142 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20-append Reporter: Raghu Angadi Assignee: dhruba borthakur Priority: Blocker Fix For: 0.20-append Attachments: appendFile-recheck-lease.txt, appendQuestions.txt, deleteTmp.patch, deleteTmp2.patch, deleteTmp5_20.txt, deleteTmp5_20.txt, deleteTmp_0.18.patch, dont-recover-rwr-when-rbw-available.txt, handleTmp1.patch, hdfs-142-commitBlockSynchronization-unknown-datanode.txt, HDFS-142-deaddn-fix.patch, HDFS-142-finalize-fix.txt, hdfs-142-minidfs-fix-from-409.txt, HDFS-142-multiple-blocks-datanode-exception.patch, hdfs-142-recovery-reassignment-and-bbw-cleanup.txt, hdfs-142-testcases.txt, hdfs-142-testleaserecovery-fix.txt, HDFS-142_20.patch, recentInvalidateSets-assertion-fix.txt, testfileappend4-deaddn.txt, validateBlockMetaData-synchronized.txt Before 0.18, when Datanode restarts, it deletes files under data-dir/tmp directory since these files are not valid anymore. But in 0.18 it moves these files to normal directory incorrectly making them valid blocks. One of the following would work : - remove the tmp files during upgrade, or - if the files under /tmp are in pre-18 format (i.e. no generation), delete them. Currently effect of this bug is that, these files end up failing block verification and eventually get deleted. But cause incorrect over-replication at the namenode before that. Also it looks like our policy regd treating files under tmp needs to be defined better. Right now there are probably one or two more bugs with it. Dhruba, please file them if you rememeber. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-142) In 0.20, move blocks being written into a blocksBeingWritten directory
[ https://issues.apache.org/jira/browse/HDFS-142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-142: -- Fix Version/s: 0.20-append In 0.20, move blocks being written into a blocksBeingWritten directory -- Key: HDFS-142 URL: https://issues.apache.org/jira/browse/HDFS-142 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20-append Reporter: Raghu Angadi Assignee: dhruba borthakur Priority: Blocker Fix For: 0.20-append Attachments: appendFile-recheck-lease.txt, appendQuestions.txt, deleteTmp.patch, deleteTmp2.patch, deleteTmp5_20.txt, deleteTmp5_20.txt, deleteTmp_0.18.patch, dont-recover-rwr-when-rbw-available.txt, handleTmp1.patch, hdfs-142-commitBlockSynchronization-unknown-datanode.txt, HDFS-142-deaddn-fix.patch, HDFS-142-finalize-fix.txt, hdfs-142-minidfs-fix-from-409.txt, HDFS-142-multiple-blocks-datanode-exception.patch, hdfs-142-recovery-reassignment-and-bbw-cleanup.txt, hdfs-142-testcases.txt, hdfs-142-testleaserecovery-fix.txt, HDFS-142_20.patch, recentInvalidateSets-assertion-fix.txt, testfileappend4-deaddn.txt, validateBlockMetaData-synchronized.txt Before 0.18, when Datanode restarts, it deletes files under data-dir/tmp directory since these files are not valid anymore. But in 0.18 it moves these files to normal directory incorrectly making them valid blocks. One of the following would work : - remove the tmp files during upgrade, or - if the files under /tmp are in pre-18 format (i.e. no generation), delete them. Currently effect of this bug is that, these files end up failing block verification and eventually get deleted. But cause incorrect over-replication at the namenode before that. Also it looks like our policy regd treating files under tmp needs to be defined better. Right now there are probably one or two more bugs with it. Dhruba, please file them if you rememeber. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HDFS-200) In HDFS, sync() not yet guarantees data available to the new readers
[ https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur reopened HDFS-200: --- This has to be pulled into the branch-0.20-append branch. In HDFS, sync() not yet guarantees data available to the new readers Key: HDFS-200 URL: https://issues.apache.org/jira/browse/HDFS-200 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: dhruba borthakur Priority: Blocker Attachments: 4379_20081010TC3.java, checkLeases-fix-1.txt, checkLeases-fix-unit-test-1.txt, fsyncConcurrentReaders.txt, fsyncConcurrentReaders11_20.txt, fsyncConcurrentReaders12_20.txt, fsyncConcurrentReaders13_20.txt, fsyncConcurrentReaders14_20.txt, fsyncConcurrentReaders15_20.txt, fsyncConcurrentReaders16_20.txt, fsyncConcurrentReaders3.patch, fsyncConcurrentReaders4.patch, fsyncConcurrentReaders5.txt, fsyncConcurrentReaders6.patch, fsyncConcurrentReaders9.patch, hadoop-stack-namenode-aa0-000-12.u.powerset.com.log.gz, hdfs-200-ryan-existing-file-fail.txt, hypertable-namenode.log.gz, namenode.log, namenode.log, Reader.java, Reader.java, reopen_test.sh, ReopenProblem.java, Writer.java, Writer.java In the append design doc (https://issues.apache.org/jira/secure/attachment/12370562/Appends.doc), it says * A reader is guaranteed to be able to read data that was 'flushed' before the reader opened the file However, this feature is not yet implemented. Note that the operation 'flushed' is now called sync. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-200) In HDFS, sync() not yet guarantees data available to the new readers
[ https://issues.apache.org/jira/browse/HDFS-200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-200: -- Fix Version/s: 0.20-append In HDFS, sync() not yet guarantees data available to the new readers Key: HDFS-200 URL: https://issues.apache.org/jira/browse/HDFS-200 Project: Hadoop HDFS Issue Type: New Feature Reporter: Tsz Wo (Nicholas), SZE Assignee: dhruba borthakur Priority: Blocker Fix For: 0.20-append Attachments: 4379_20081010TC3.java, checkLeases-fix-1.txt, checkLeases-fix-unit-test-1.txt, fsyncConcurrentReaders.txt, fsyncConcurrentReaders11_20.txt, fsyncConcurrentReaders12_20.txt, fsyncConcurrentReaders13_20.txt, fsyncConcurrentReaders14_20.txt, fsyncConcurrentReaders15_20.txt, fsyncConcurrentReaders16_20.txt, fsyncConcurrentReaders3.patch, fsyncConcurrentReaders4.patch, fsyncConcurrentReaders5.txt, fsyncConcurrentReaders6.patch, fsyncConcurrentReaders9.patch, hadoop-stack-namenode-aa0-000-12.u.powerset.com.log.gz, hdfs-200-ryan-existing-file-fail.txt, hypertable-namenode.log.gz, namenode.log, namenode.log, Reader.java, Reader.java, reopen_test.sh, ReopenProblem.java, Writer.java, Writer.java In the append design doc (https://issues.apache.org/jira/secure/attachment/12370562/Appends.doc), it says * A reader is guaranteed to be able to read data that was 'flushed' before the reader opened the file However, this feature is not yet implemented. Note that the operation 'flushed' is now called sync. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1096) allow dfsadmin/mradmin refresh of superuser proxy group mappings
[ https://issues.apache.org/jira/browse/HDFS-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1096: - Release Note: changed protocol name (may be used in hadoop-policy.xml) from security.refresh.usertogroups.mappings.protocol.acl to security.refresh.user.mappings.protocol.acl allow dfsadmin/mradmin refresh of superuser proxy group mappings Key: HDFS-1096 URL: https://issues.apache.org/jira/browse/HDFS-1096 Project: Hadoop HDFS Issue Type: New Feature Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1096-BP20-4.patch, HDFS-1096-BP20-6-fix.patch, HDFS-1096-BP20-6.patch, HDFS-1096-BP20-7.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1019) Incorrect default values for delegation tokens in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1019: --- Attachment: HDFS-1019.4.patch Patch rebased against latest trunk. Incorrect default values for delegation tokens in hdfs-default.xml -- Key: HDFS-1019 URL: https://issues.apache.org/jira/browse/HDFS-1019 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1019-y20.1.patch, HDFS-1019.1.patch, HDFS-1019.2.patch, HDFS-1019.3.patch, HDFS-1019.4.patch The default values for delegation token parameters in hdfs-default.xml are incorrect. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1019) Incorrect default values for delegation tokens in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874700#action_12874700 ] Jitendra Nath Pandey commented on HDFS-1019: Tests were run manually. All tests passed except TestHDFSTrash, which is unrelated. Incorrect default values for delegation tokens in hdfs-default.xml -- Key: HDFS-1019 URL: https://issues.apache.org/jira/browse/HDFS-1019 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1019-y20.1.patch, HDFS-1019.1.patch, HDFS-1019.2.patch, HDFS-1019.3.patch, HDFS-1019.4.patch The default values for delegation token parameters in hdfs-default.xml are incorrect. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1019) Incorrect default values for delegation tokens in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874711#action_12874711 ] Jitendra Nath Pandey commented on HDFS-1019: Test patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. It is a configuration change, therefore no tests have been added or modified. However, the patch has been tested manually for 20 branch. Incorrect default values for delegation tokens in hdfs-default.xml -- Key: HDFS-1019 URL: https://issues.apache.org/jira/browse/HDFS-1019 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1019-y20.1.patch, HDFS-1019.1.patch, HDFS-1019.2.patch, HDFS-1019.3.patch, HDFS-1019.4.patch The default values for delegation token parameters in hdfs-default.xml are incorrect. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1019) Incorrect default values for delegation tokens in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1019: --- Status: Open (was: Patch Available) Incorrect default values for delegation tokens in hdfs-default.xml -- Key: HDFS-1019 URL: https://issues.apache.org/jira/browse/HDFS-1019 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1019-y20.1.patch, HDFS-1019.1.patch, HDFS-1019.2.patch, HDFS-1019.3.patch, HDFS-1019.4.patch The default values for delegation token parameters in hdfs-default.xml are incorrect. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1019) Incorrect default values for delegation tokens in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1019: - Hadoop Flags: [Reviewed] +1 Committed to trunk. Thanks Jitendra. Incorrect default values for delegation tokens in hdfs-default.xml -- Key: HDFS-1019 URL: https://issues.apache.org/jira/browse/HDFS-1019 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1019-y20.1.patch, HDFS-1019.1.patch, HDFS-1019.2.patch, HDFS-1019.3.patch, HDFS-1019.4.patch The default values for delegation token parameters in hdfs-default.xml are incorrect. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-445) pread() fails when cached block locations are no longer valid
[ https://issues.apache.org/jira/browse/HDFS-445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-445: - Affects Version/s: 0.20-append Issue appeared during HBase testing. This should be pulled into the branch-0.20-append branch. pread() fails when cached block locations are no longer valid - Key: HDFS-445 URL: https://issues.apache.org/jira/browse/HDFS-445 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20-append Reporter: Kan Zhang Assignee: Kan Zhang Fix For: 0.21.0 Attachments: 445-06.patch, 445-08.patch, HDFS-445-0_20.2.patch when cached block locations are no longer valid (e.g., datanodes restart on different ports), pread() will fail, whereas normal read() still succeeds through re-fetching of block locations from namenode (up to a max number of times). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-927) DFSInputStream retries too many times for new block locations
[ https://issues.apache.org/jira/browse/HDFS-927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-927: - Affects Version/s: 0.20-append This should be pulled into the branch-0.20-append branch. DFSInputStream retries too many times for new block locations - Key: HDFS-927 URL: https://issues.apache.org/jira/browse/HDFS-927 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.21.0, 0.22.0, 0.20-append Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.20.2, 0.21.0 Attachments: hdfs-927-branch-0.21.txt, hdfs-927-branch0.20.txt, hdfs-927.txt I think this is a regression caused by HDFS-127 -- DFSInputStream is supposed to only go back to the NN max.block.acquires times, but in trunk it goes back twice as many - the default is 3, but I am counting 7 calls to getBlockLocations before an exception is thrown. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-127) DFSClient block read failures cause open DFSInputStream to become unusable
[ https://issues.apache.org/jira/browse/HDFS-127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-127: - Affects Version/s: 0.20-append This should be pulled into the branch-0.20-append branch. DFSClient block read failures cause open DFSInputStream to become unusable -- Key: HDFS-127 URL: https://issues.apache.org/jira/browse/HDFS-127 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.20-append Reporter: Igor Bolotin Assignee: Igor Bolotin Fix For: 0.21.0 Attachments: 4681.patch, h127_20091016.patch, h127_20091019.patch, h127_20091019b.patch, hdfs-127-branch20-redone-v2.txt, hdfs-127-branch20-redone.txt, hdfs-127-regression-test.txt We are using some Lucene indexes directly from HDFS and for quite long time we were using Hadoop version 0.15.3. When tried to upgrade to Hadoop 0.19 - index searches started to fail with exceptions like: 2008-11-13 16:50:20,314 WARN [Listener-4] [] DFSClient : DFS Read: java.io.IOException: Could not obtain block: blk_5604690829708125511_15489 file=/usr/collarity/data/urls-new/part-0/20081110-163426/_0.tis at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1708) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1536) at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1663) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.nutch.indexer.FsDirectory$DfsIndexInput.readInternal(FsDirectory.java:174) at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:152) at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:38) at org.apache.lucene.store.IndexInput.readVInt(IndexInput.java:76) at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:63) at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:131) at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:162) at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java:223) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:217) at org.apache.lucene.index.SegmentTermDocs.seek(SegmentTermDocs.java:54) ... The investigation showed that the root of this issue is that we exceeded # of xcievers in the data nodes and that was fixed by changing configuration settings to 2k. However - one thing that bothered me was that even after datanodes recovered from overload and most of client servers had been shut down - we still observed errors in the logs of running servers. Further investigation showed that fix for HADOOP-1911 introduced another problem - the DFSInputStream instance might become unusable once number of failures over lifetime of this instance exceeds configured threshold. The fix for this specific issue seems to be trivial - just reset failure counter before reading next block (patch will be attached shortly). This seems to be also related to HADOOP-3185, but I'm not sure I really understand necessity of keeping track of failed block accesses in the DFS client. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-988) saveNamespace can corrupt edits log
[ https://issues.apache.org/jira/browse/HDFS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-988: - Affects Version/s: 0.20-append This should be pulled into the branch-0.20-append branch. Besides stability fix, causes append tests to fail. saveNamespace can corrupt edits log --- Key: HDFS-988 URL: https://issues.apache.org/jira/browse/HDFS-988 Project: Hadoop HDFS Issue Type: Bug Components: name-node Affects Versions: 0.21.0, 0.22.0, 0.20-append Reporter: dhruba borthakur Assignee: Todd Lipcon Priority: Blocker Attachments: hdfs-988.txt, saveNamespace.txt The adminstrator puts the namenode is safemode and then issues the savenamespace command. This can corrupt the edits log. The problem is that when the NN enters safemode, there could still be pending logSycs occuring from other threads. Now, the saveNamespace command, when executed, would save a edits log with partial writes. I have seen this happen on 0.20. https://issues.apache.org/jira/browse/HDFS-909?focusedCommentId=12828853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12828853 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-606) ConcurrentModificationException in invalidateCorruptReplicas()
[
https://issues.apache.org/jira/browse/HDFS-606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Spiegelberg updated HDFS-606:
-
Affects Version/s: 0.20-append
This should be pulled into the branch-0.20-append branch.
ConcurrentModificationException in invalidateCorruptReplicas()
--
Key: HDFS-606
URL: https://issues.apache.org/jira/browse/HDFS-606
Project: Hadoop HDFS
Issue Type: Bug
Components: name-node
Affects Versions: 0.21.0, 0.20-append
Reporter: Konstantin Shvachko
Assignee: Konstantin Shvachko
Fix For: 0.21.0
Attachments: CMEinCorruptReplicas.patch, hdfs-606-branch20.txt
{{BlockManager.invalidateCorruptReplicas()}} iterates over
DatanodeDescriptor-s while removing corrupt replicas from the descriptors.
This causes {{ConcurrentModificationException}} if there is more than one
replicas of the block. I ran into this exception debugging different
scenarios in append, but it should be fixed in the trunk too.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-947) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas
[ https://issues.apache.org/jira/browse/HDFS-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-947: --- Attachment: HDFS-947.2.patch Attaching a patch that is not broken in so many ways The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas Key: HDFS-947 URL: https://issues.apache.org/jira/browse/HDFS-947 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Dmytro Molkov Attachments: HDFS-947.2.patch, HDFS-947.patch, hftpRedirection.patch A client that uses the Hftp protocol to read a file is redirected by the namenode to a random datanode. It would be nice if the client gets redirected to a datanode that has the maximum number of local replicas of the blocks of the file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-947) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas
[ https://issues.apache.org/jira/browse/HDFS-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-947: --- Status: Open (was: Patch Available) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas Key: HDFS-947 URL: https://issues.apache.org/jira/browse/HDFS-947 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Dmytro Molkov Attachments: HDFS-947.2.patch, HDFS-947.patch, hftpRedirection.patch A client that uses the Hftp protocol to read a file is redirected by the namenode to a random datanode. It would be nice if the client gets redirected to a datanode that has the maximum number of local replicas of the blocks of the file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-947) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas
[ https://issues.apache.org/jira/browse/HDFS-947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmytro Molkov updated HDFS-947: --- Status: Patch Available (was: Open) The namenode should redirect a hftp request to read a file to the datanode that has the maximum number of local replicas Key: HDFS-947 URL: https://issues.apache.org/jira/browse/HDFS-947 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: Dmytro Molkov Attachments: HDFS-947.2.patch, HDFS-947.patch, hftpRedirection.patch A client that uses the Hftp protocol to read a file is redirected by the namenode to a random datanode. It would be nice if the client gets redirected to a datanode that has the maximum number of local replicas of the blocks of the file. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HDFS-1019) Incorrect default values for delegation tokens in hdfs-default.xml
[ https://issues.apache.org/jira/browse/HDFS-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey resolved HDFS-1019. Fix Version/s: 0.22.0 Resolution: Fixed Committed by Boris. Incorrect default values for delegation tokens in hdfs-default.xml -- Key: HDFS-1019 URL: https://issues.apache.org/jira/browse/HDFS-1019 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Fix For: 0.22.0 Attachments: HDFS-1019-y20.1.patch, HDFS-1019.1.patch, HDFS-1019.2.patch, HDFS-1019.3.patch, HDFS-1019.4.patch The default values for delegation token parameters in hdfs-default.xml are incorrect. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-793) DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet
[ https://issues.apache.org/jira/browse/HDFS-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-793: - Affects Version/s: 0.20-append This should be pulled into the branch-0.20-append branch. DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet Key: HDFS-793 URL: https://issues.apache.org/jira/browse/HDFS-793 Project: Hadoop HDFS Issue Type: Bug Components: data-node Affects Versions: 0.20-append Reporter: Hairong Kuang Assignee: Hairong Kuang Priority: Blocker Fix For: 0.20.2, 0.21.0 Attachments: separateSendRcvAck-0.20-yahoo.patch, separateSendRcvAck-0.20.patch, separateSendRcvAck.patch, separateSendRcvAck1.patch, separateSendRcvAck2.patch Currently BlockReceiver#PacketResponder interleaves receiving ack message and sending ack message for the same packet. It reads a portion of the message, sends a portion of its ack, and continues like this until it hits the end of the message. The problem is that if it gets an error receiving the ack, it is not able to send an ack that reflects the source of the error. The PacketResponder should receives the whole packet ack message first and then constuct and sends out its ack. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-101) DFS write pipeline : DFSClient sometimes does not detect second datanode failure
[ https://issues.apache.org/jira/browse/HDFS-101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-101: - Affects Version/s: 0.20-append This should be pulled into the branch-0.20-append branch. DFS write pipeline : DFSClient sometimes does not detect second datanode failure - Key: HDFS-101 URL: https://issues.apache.org/jira/browse/HDFS-101 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.20.1, 0.20-append Reporter: Raghu Angadi Assignee: Hairong Kuang Priority: Blocker Fix For: 0.20.2, 0.21.0 Attachments: detectDownDN-0.20.patch, detectDownDN1-0.20.patch, detectDownDN2.patch, detectDownDN3-0.20-yahoo.patch, detectDownDN3-0.20.patch, detectDownDN3.patch, hdfs-101.tar.gz, pipelineHeartbeat.patch, pipelineHeartbeat_yahoo.patch When the first datanode's write to second datanode fails or times out DFSClient ends up marking first datanode as the bad one and removes it from the pipeline. Similar problem exists on DataNode as well and it is fixed in HADOOP-3339. From HADOOP-3339 : The main issue is that BlockReceiver thread (and DataStreamer in the case of DFSClient) interrupt() the 'responder' thread. But interrupting is a pretty coarse control. We don't know what state the responder is in and interrupting has different effects depending on responder state. To fix this properly we need to redesign how we handle these interactions. When the first datanode closes its socket from DFSClient, DFSClient should properly read all the data left in the socket.. Also, DataNode's closing of the socket should not result in a TCP reset, otherwise I think DFSClient will not be able to read from the socket. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-826) Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline
[ https://issues.apache.org/jira/browse/HDFS-826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-826: - Affects Version/s: 0.20-append This should be pulled into the branch-0.20-append branch. Allow a mechanism for an application to detect that datanode(s) have died in the write pipeline Key: HDFS-826 URL: https://issues.apache.org/jira/browse/HDFS-826 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs client Affects Versions: 0.20-append Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: HDFS-826-0.20-v2.patch, HDFS-826-0.20.patch, Replicable4.txt, ReplicableHdfs.txt, ReplicableHdfs2.txt, ReplicableHdfs3.txt HDFS does not replicate the last block of the file that is being currently written to by an application. Every datanode death in the write pipeline decreases the reliability of the last block of the currently-being-written block. This situation can be improved if the application can be notified of a datanode death in the write pipeline. Then, the application can decide what is the right course of action to be taken on this event. In our use-case, the application can close the file on the first datanode death, and start writing to a newly created file. This ensures that the reliability guarantee of a block is close to 3 at all time. One idea is to make DFSOutoutStream. write() throw an exception if the number of datanodes in the write pipeline fall below minimum.replication.factor that is set on the client (this is backward compatible). -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1096) allow dfsadmin/mradmin refresh of superuser proxy group mappings
[ https://issues.apache.org/jira/browse/HDFS-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Boris Shkolnik updated HDFS-1096: - Attachment: HDFS-1096-3.patch allow dfsadmin/mradmin refresh of superuser proxy group mappings Key: HDFS-1096 URL: https://issues.apache.org/jira/browse/HDFS-1096 Project: Hadoop HDFS Issue Type: New Feature Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1096-3.patch, HDFS-1096-BP20-4.patch, HDFS-1096-BP20-6-fix.patch, HDFS-1096-BP20-6.patch, HDFS-1096-BP20-7.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1146) Javadoc for getDelegationTokenSecretManager in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874798#action_12874798 ] Jitendra Nath Pandey commented on HDFS-1146: test patch results. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. Javadoc for getDelegationTokenSecretManager in FSNamesystem --- Key: HDFS-1146 URL: https://issues.apache.org/jira/browse/HDFS-1146 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1146-y20.1.patch Javadoc is missing for public method getDelegationTokenSecretManager in FSNamesystem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1146) Javadoc for getDelegationTokenSecretManager in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874800#action_12874800 ] Jitendra Nath Pandey commented on HDFS-1146: ant test is not required because only javadoc is added in this patch. Javadoc for getDelegationTokenSecretManager in FSNamesystem --- Key: HDFS-1146 URL: https://issues.apache.org/jira/browse/HDFS-1146 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1146-y20.1.patch, HDFS-1146.1.patch Javadoc is missing for public method getDelegationTokenSecretManager in FSNamesystem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1146) Javadoc for getDelegationTokenSecretManager in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jitendra Nath Pandey updated HDFS-1146: --- Attachment: HDFS-1146.1.patch Javadoc for getDelegationTokenSecretManager in FSNamesystem --- Key: HDFS-1146 URL: https://issues.apache.org/jira/browse/HDFS-1146 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1146-y20.1.patch, HDFS-1146.1.patch Javadoc is missing for public method getDelegationTokenSecretManager in FSNamesystem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1146) Javadoc for getDelegationTokenSecretManager in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-1146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874799#action_12874799 ] Jitendra Nath Pandey commented on HDFS-1146: This patch just adds javadoc, therefore no tests have been added or modified. Javadoc for getDelegationTokenSecretManager in FSNamesystem --- Key: HDFS-1146 URL: https://issues.apache.org/jira/browse/HDFS-1146 Project: Hadoop HDFS Issue Type: Bug Reporter: Jitendra Nath Pandey Assignee: Jitendra Nath Pandey Attachments: HDFS-1146-y20.1.patch, HDFS-1146.1.patch Javadoc is missing for public method getDelegationTokenSecretManager in FSNamesystem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1057) Concurrent readers hit ChecksumExceptions if following a writer to very end of file
[ https://issues.apache.org/jira/browse/HDFS-1057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-1057: -- Affects Version/s: 0.20-append This should be pulled into the branch-0.20-append branch. Concurrent readers hit ChecksumExceptions if following a writer to very end of file --- Key: HDFS-1057 URL: https://issues.apache.org/jira/browse/HDFS-1057 Project: Hadoop HDFS Issue Type: Sub-task Components: data-node Affects Versions: 0.21.0, 0.22.0, 0.20-append Reporter: Todd Lipcon Assignee: sam rash Priority: Blocker Attachments: conurrent-reader-patch-1.txt, conurrent-reader-patch-2.txt, conurrent-reader-patch-3.txt, hdfs-1057-trunk-1.txt In BlockReceiver.receivePacket, it calls replicaInfo.setBytesOnDisk before calling flush(). Therefore, if there is a concurrent reader, it's possible to race here - the reader will see the new length while those bytes are still in the buffers of BlockReceiver. Thus the client will potentially see checksum errors or EOFs. Additionally, the last checksum chunk of the file is made accessible to readers even though it is not stable. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1060) Append/flush should support concurrent tailer use case
[ https://issues.apache.org/jira/browse/HDFS-1060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-1060: -- Affects Version/s: 0.20-append This should be pulled into the branch-0.20-append branch. Append/flush should support concurrent tailer use case Key: HDFS-1060 URL: https://issues.apache.org/jira/browse/HDFS-1060 Project: Hadoop HDFS Issue Type: Bug Components: data-node, hdfs client Affects Versions: 0.21.0, 0.22.0, 0.20-append Reporter: Todd Lipcon Attachments: hdfs-1060-tests.txt Several people have a usecase for a writer logging edits and using hflush() while one or more readers tails the file by periodically reopening and seeking to get the latest bytes. Currently there are several bugs. Using this ticket as a supertask for those bugs (and to add test cases for this use case) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1122) client block verification may result in blocks in DataBlockScanner prematurely
[ https://issues.apache.org/jira/browse/HDFS-1122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Spiegelberg updated HDFS-1122: -- Affects Version/s: 0.20-append This should be pulled into the 0.20-append branch. client block verification may result in blocks in DataBlockScanner prematurely -- Key: HDFS-1122 URL: https://issues.apache.org/jira/browse/HDFS-1122 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 0.20-append Reporter: sam rash Assignee: sam rash Attachments: hdfs-1122-for-0.20.txt found that when the DN uses client verification of a block that is open for writing, it will add it to the DataBlockScanner prematurely. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1114) Reducing NameNode memory usage by an alternate hash table
[ https://issues.apache.org/jira/browse/HDFS-1114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874895#action_12874895 ] Tsz Wo (Nicholas), SZE commented on HDFS-1114: -- Thanks Suresh that he found out an interesting post explaining [why java.util.HashMap uses power of two table length|http://www.roseindia.net/javatutorials/javahashmap.shtml]. Reducing NameNode memory usage by an alternate hash table - Key: HDFS-1114 URL: https://issues.apache.org/jira/browse/HDFS-1114 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: GSet20100525.pdf NameNode uses a java.util.HashMap to store BlockInfo objects. When there are many blocks in HDFS, this map uses a lot of memory in the NameNode. We may optimize the memory usage by a light weight hash table implementation. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-1096) allow dfsadmin/mradmin refresh of superuser proxy group mappings
[ https://issues.apache.org/jira/browse/HDFS-1096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874896#action_12874896 ] Jitendra Nath Pandey commented on HDFS-1096: +1 for the patch. allow dfsadmin/mradmin refresh of superuser proxy group mappings Key: HDFS-1096 URL: https://issues.apache.org/jira/browse/HDFS-1096 Project: Hadoop HDFS Issue Type: New Feature Reporter: Boris Shkolnik Assignee: Boris Shkolnik Attachments: HDFS-1096-3.patch, HDFS-1096-BP20-4.patch, HDFS-1096-BP20-6-fix.patch, HDFS-1096-BP20-6.patch, HDFS-1096-BP20-7.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1150) Verify datanodes' identities to clients in secure clusters
[ https://issues.apache.org/jira/browse/HDFS-1150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-1150: -- Attachment: HDFS-1150-Y20-BetterJsvcHandling.patch Updating patch, not for commit. We found that manually building the jsvc binary did not work well; there were problems if the build machine is of different architecture (we run 32 bit JVMs for DNs/TTs). New process is to download the file, either directly from Apache, or overridden via the -Djsvc.location param. This greatly simplifies the jsvc process. Also for those interested, in our testing here at Y!, this solution has worked great for securing the datanodes. We've run into no problems with running the datanodes under jsvc. Verify datanodes' identities to clients in secure clusters -- Key: HDFS-1150 URL: https://issues.apache.org/jira/browse/HDFS-1150 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.22.0 Reporter: Jakob Homan Assignee: Jakob Homan Attachments: commons-daemon-1.0.2-src.tar.gz, HDFS-1150-BF-Y20-LOG-DIRS-2.patch, HDFS-1150-BF-Y20-LOG-DIRS.patch, HDFS-1150-BF1-Y20.patch, hdfs-1150-bugfix-1.1.patch, hdfs-1150-bugfix-1.2.patch, hdfs-1150-bugfix-1.patch, HDFS-1150-Y20-BetterJsvcHandling.patch, HDFS-1150-y20.build-script.patch, HDFS-1150-Y20S-ready-5.patch, HDFS-1150-Y20S-ready-6.patch, HDFS-1150-Y20S-ready-7.patch, HDFS-1150-Y20S-ready-8.patch, HDFS-1150-Y20S-Rough-2.patch, HDFS-1150-Y20S-Rough-3.patch, HDFS-1150-Y20S-Rough-4.patch, HDFS-1150-Y20S-Rough.txt Currently we use block access tokens to allow datanodes to verify clients' identities, however we don't have a way for clients to verify the authenticity of the datanodes themselves. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12874902#action_12874902 ] Rodrigo Schmidt commented on HDFS-503: -- The tags are XML. There is no documentation for the tags, either. In short, Raid is still being optimized and changes are constant. Any strong documentation effort at this point would be meaningful for a very short period of time. The source code is the best and most precise documentation you can rely upon. That's the good thing about open source projects. You can easily get around stale documentation. Implement erasure coding as a layer on HDFS --- Key: HDFS-503 URL: https://issues.apache.org/jira/browse/HDFS-503 Project: Hadoop HDFS Issue Type: New Feature Components: contrib/raid Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: raid1.txt, raid2.txt The goal of this JIRA is to discuss how the cost of raw storage for a HDFS file system can be reduced. Keeping three copies of the same data is very costly, especially when the size of storage is huge. One idea is to reduce the replication factor and do erasure coding of a set of blocks so that the over probability of failure of a block remains the same as before. Many forms of error-correcting codes are available, see http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has described DiskReduce https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. My opinion is to discuss implementation strategies that are not part of base HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HDFS-1183) Remove some duplicate code in NamenodeJspHelper.java
Remove some duplicate code in NamenodeJspHelper.java Key: HDFS-1183 URL: https://issues.apache.org/jira/browse/HDFS-1183 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jeff Priority: Minor Attachments: HDFS-1183.patch This patch refactors out some duplicate code in generateNodeData and generateDecommissioningNodeData in NamenodeJspHelper.java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1183) Remove some duplicate code in NamenodeJspHelper.java
[ https://issues.apache.org/jira/browse/HDFS-1183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff updated HDFS-1183: --- Attachment: HDFS-1183.patch Remove some duplicate code in NamenodeJspHelper.java Key: HDFS-1183 URL: https://issues.apache.org/jira/browse/HDFS-1183 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Jeff Priority: Minor Attachments: HDFS-1183.patch This patch refactors out some duplicate code in generateNodeData and generateDecommissioningNodeData in NamenodeJspHelper.java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-1184) Replace tabs in code with spaces
[ https://issues.apache.org/jira/browse/HDFS-1184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff updated HDFS-1184: --- Attachment: HDFS-1184.patch Replace tabs in code with spaces Key: HDFS-1184 URL: https://issues.apache.org/jira/browse/HDFS-1184 Project: Hadoop HDFS Issue Type: Improvement Reporter: Jeff Priority: Minor Attachments: HDFS-1184.patch Replace some code indentation that uses tabs with spaces, to match the coding convention -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
