[jira] Created: (HDFS-503) Implement erasure coding as a layer on HDFS
Implement erasure coding as a layer on HDFS --- Key: HDFS-503 URL: https://issues.apache.org/jira/browse/HDFS-503 Project: Hadoop HDFS Issue Type: New Feature Reporter: dhruba borthakur Assignee: dhruba borthakur The goal of this JIRA is to discuss how the cost of raw storage for a HDFS file system can be reduced. Keeping three copies of the same data is very costly, especially when the size of storage is huge. One idea is to reduce the replication factor and do erasure coding of a set of blocks so that the over probability of failure of a block remains the same as before. Many forms of error-correcting codes are available, see http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has described DiskReduce https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. My opinion is to discuss implementation strategies that are not part of base HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735011#action_12735011 ] Hong Tang commented on HDFS-503: As a reference, FAST 09 has a paper that benchmarks the performance of various open source erasure coding implementations: http://www.cs.utk.edu/~plank/plank/papers/FAST-2009.html. Implement erasure coding as a layer on HDFS --- Key: HDFS-503 URL: https://issues.apache.org/jira/browse/HDFS-503 Project: Hadoop HDFS Issue Type: New Feature Reporter: dhruba borthakur Assignee: dhruba borthakur The goal of this JIRA is to discuss how the cost of raw storage for a HDFS file system can be reduced. Keeping three copies of the same data is very costly, especially when the size of storage is huge. One idea is to reduce the replication factor and do erasure coding of a set of blocks so that the over probability of failure of a block remains the same as before. Many forms of error-correcting codes are available, see http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has described DiskReduce https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. My opinion is to discuss implementation strategies that are not part of base HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-385) Design a pluggable interface to place replicas of blocks in HDFS
[ https://issues.apache.org/jira/browse/HDFS-385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735106#action_12735106 ] Hairong Kuang commented on HDFS-385: Sorry for giving comments at this late stage. I was out of the country in the past few weeks. 1. FsInodeName#getLocalName should return the full path name of the file not the name of the last component of the path. Better rename the method name to be getFullPathName. The implementation of this interface in INode should return the full path name as well. Also to allow FsInodeName to be able to return more information about a file in the future, better change the interface name to be FsInodeInfo. 2. FsInodeName should also be a parameter to verifyBlockPlacement and isValidMove. This makes the interface consistent and makes it easier to get a file name for a placement policy like colocation. 3. Would it provide better readability if the name of the class BlockPlacementPolicyDefault is changed to FirstLocalTwoRemoteBlockPlacementPolicy? 4. For the default policy, isValidDelete should not be empty. It should implement the deletion policy in trunk. So other placement policy can override the default deletion policy. 5. Even with the inValidMove API, the default balancer does not work with the colocation placement policy because it moves one block a time. I'd like to propose to remove isValidMove API until we could figure out a general balancer implementation. For this round, the default balancer could check with NameNode in the very beginning if NameNode runs the default block placement policy. If no, it stops to run. Design a pluggable interface to place replicas of blocks in HDFS Key: HDFS-385 URL: https://issues.apache.org/jira/browse/HDFS-385 Project: Hadoop HDFS Issue Type: Improvement Reporter: dhruba borthakur Assignee: dhruba borthakur Fix For: 0.21.0 Attachments: BlockPlacementPluggable.txt, BlockPlacementPluggable2.txt, BlockPlacementPluggable3.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable4.txt, BlockPlacementPluggable5.txt The current HDFS code typically places one replica on local rack, the second replica on remote random rack and the third replica on a random node of that remote rack. This algorithm is baked in the NameNode's code. It would be nice to make the block placement algorithm a pluggable interface. This will allow experimentation of different placement algorithms based on workloads, availability guarantees and failure models. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-500) Fix lingering and new javac warnings
[ https://issues.apache.org/jira/browse/HDFS-500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Homan updated HDFS-500: - Status: Patch Available (was: Open) submitting patch. Fix lingering and new javac warnings Key: HDFS-500 URL: https://issues.apache.org/jira/browse/HDFS-500 Project: Hadoop HDFS Issue Type: Bug Reporter: Jakob Homan Assignee: Jakob Homan Priority: Minor Attachments: HDFS-500.patch The broken javac warnings issues documented in HADOOP-6124 have allowed some new javac warnings into the codebase, and there are a few lingering from the previous code cleanup. These should be fixed. Patch shortly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-435) Add orthogonal fault injection mechanism/framework
[ https://issues.apache.org/jira/browse/HDFS-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735118#action_12735118 ] Nigel Daley commented on HDFS-435: -- Yes, the doc should be converted to forrest and put in src/docs/src/documentation/content/xdocs Add orthogonal fault injection mechanism/framework -- Key: HDFS-435 URL: https://issues.apache.org/jira/browse/HDFS-435 Project: Hadoop HDFS Issue Type: Test Components: test Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Attachments: Fault injection development guide and Framework HowTo.pdf, Fault injection development guide and Framework HowTo.pdf It'd be great to have a fault injection mechanism for Hadoop. Having such solution in place will allow to increase test coverage of error handling and recovery mechanisms, reduce reproduction time and increase the reproduction rate of the problems. Ideally, the system has to be orthogonal to the current code and test base. E.g. faults have to be injected at build time and would have to be configurable, e.g. all faults could be turned off, or only some of them would be allowed to happen. Also, fault injection has to be separated from production build. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-435) Add orthogonal fault injection mechanism/framework
[ https://issues.apache.org/jira/browse/HDFS-435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735127#action_12735127 ] Konstantin Boudnik commented on HDFS-435: - It has been already done - please see HDFS-498 subtask of this JIRA Add orthogonal fault injection mechanism/framework -- Key: HDFS-435 URL: https://issues.apache.org/jira/browse/HDFS-435 Project: Hadoop HDFS Issue Type: Test Components: test Reporter: Konstantin Boudnik Assignee: Konstantin Boudnik Attachments: Fault injection development guide and Framework HowTo.pdf, Fault injection development guide and Framework HowTo.pdf It'd be great to have a fault injection mechanism for Hadoop. Having such solution in place will allow to increase test coverage of error handling and recovery mechanisms, reduce reproduction time and increase the reproduction rate of the problems. Ideally, the system has to be orthogonal to the current code and test base. E.g. faults have to be injected at build time and would have to be configurable, e.g. all faults could be turned off, or only some of them would be allowed to happen. Also, fault injection has to be separated from production build. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-504) HDFS updates the modification time of a file when the file is closed.
[ https://issues.apache.org/jira/browse/HDFS-504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur reassigned HDFS-504: - Assignee: Chun Zhang HDFS updates the modification time of a file when the file is closed. - Key: HDFS-504 URL: https://issues.apache.org/jira/browse/HDFS-504 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.20.1 Reporter: Chun Zhang Assignee: Chun Zhang Priority: Minor Fix For: 0.20.1 Original Estimate: 2h Remaining Estimate: 2h Current HDFS updates the modification time of a file when the file is created. We would like to update the modification time of the file when the file is closed after being written to. This helps HDFS Raid to detect file changes more aggressively. Solution includes: 1. Made changes to closeFile@/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java; 2. Added unit test to testTimesAtClose@/org/apache/hadoop/hdfs/TestSetTimes.java. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-505) Convert the file including 10 minutes run's tests into the test harness test suite
[ https://issues.apache.org/jira/browse/HDFS-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735149#action_12735149 ] Konstantin Boudnik commented on HDFS-505: - In order to clear possible confusion, I'd like to add a couple of comments here: - it's big shame that JUnit doesn't support tests tagging contrary to a popular believe (see my comments for HADOOP-4901) - because there's some opposition to bring in a 3rd party libraries into the project (again see HADOOP-4901) the tagging would have to be implemented through JUnit test suites instead. - apparently, as Jacob has pointed out - we'd need to work around a potential problem of running the same tests twice (10 minutes ones and all of them in a single run). And there are some possible solutions for this, e.g. setting up test.exclude property to remove 10 minutes suite from a normal run; or to have a certain test class naming conventions, i.e. add a token 'Suite' into a class name and filter such classes out in a normal runs. Other approaches are, perhaps, possible to. Convert the file including 10 minutes run's tests into the test harness test suite -- Key: HDFS-505 URL: https://issues.apache.org/jira/browse/HDFS-505 Project: Hadoop HDFS Issue Type: Improvement Components: build, test Reporter: Konstantin Boudnik This issue is created to track the conversion of external testlist for 10 minutes tests into JUnit test suite. It would be consistent through the bigger Hadoop project to have 10 minutes tests to be collected in a JUnit's test suite instead of an external file (as of now). The reason is simple: with the file we'd have two points of maintenance: tests source code and one an auxiliary text files. Besides, a general approach is to use JUnit's suites for tagging purposes, thus having a suite here would be more uniform. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-505) Convert the file including 10 minutes run's tests into the test harness test suite
[ https://issues.apache.org/jira/browse/HDFS-505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735153#action_12735153 ] Jakob Homan commented on HDFS-505: -- In my opinion, having the 10-minute-tests defined in an external file is a feature, not a bug. It makes it easy to quickly see what's in the suite quickly and easily update it without touching code. The set membership seems to be a matter of configuration, which should go into a separate file, rather than code, which should go into a test suite. Tagging had seemed like a good solution because a test's membership into the set can be viewed as metadata, but if we don't really have annotation-type tagging, it may not be worth it. I had hoped to use tagging because I wanted to tag individual tests from a test suite rather than running all the tests in the suite, as we have to do now. But it sounds like that's not possible, even creating a new test suite. Or am I wrong about that? If it's not possible to include individual tests into the 10min suite via annotation-type tagging, and if we're going to have to do something tricky/clever to get the test suite approach to work (as Cos mentions above), I'd rather keep the flat files. Convert the file including 10 minutes run's tests into the test harness test suite -- Key: HDFS-505 URL: https://issues.apache.org/jira/browse/HDFS-505 Project: Hadoop HDFS Issue Type: Improvement Components: build, test Reporter: Konstantin Boudnik This issue is created to track the conversion of external testlist for 10 minutes tests into JUnit test suite. It would be consistent through the bigger Hadoop project to have 10 minutes tests to be collected in a JUnit's test suite instead of an external file (as of now). The reason is simple: with the file we'd have two points of maintenance: tests source code and one an auxiliary text files. Besides, a general approach is to use JUnit's suites for tagging purposes, thus having a suite here would be more uniform. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-501) Use enum to define the constants in DataTransferProtocol
[ https://issues.apache.org/jira/browse/HDFS-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-501: Attachment: h501_20090724.patch h501_20090724.patch: define constants with enum. Use enum to define the constants in DataTransferProtocol Key: HDFS-501 URL: https://issues.apache.org/jira/browse/HDFS-501 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, hdfs client Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h501_20090724.patch There are quite a few integer (int/byte) constants defined in DataTransferProtocol. It is better to replace them by enums. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-501) Use enum to define the constants in DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735183#action_12735183
]
Tsz Wo (Nicholas), SZE commented on HDFS-501:
-
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 9 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
{noformat}
Running unit tests ...
Use enum to define the constants in DataTransferProtocol
Key: HDFS-501
URL: https://issues.apache.org/jira/browse/HDFS-501
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node, hdfs client
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Attachments: h501_20090724.patch
There are quite a few integer (int/byte) constants defined in
DataTransferProtocol. It is better to replace them by enums.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-501) Use enum to define the constants in DataTransferProtocol
[ https://issues.apache.org/jira/browse/HDFS-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735204#action_12735204 ] Tsz Wo (Nicholas), SZE commented on HDFS-501: - Only TestBackupNode failed for ant test but it is not related. See HDFS-192. Use enum to define the constants in DataTransferProtocol Key: HDFS-501 URL: https://issues.apache.org/jira/browse/HDFS-501 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, hdfs client Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h501_20090724.patch There are quite a few integer (int/byte) constants defined in DataTransferProtocol. It is better to replace them by enums. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-501) Use enum to define the constants in DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12735219#action_12735219
]
Suresh Srinivas commented on HDFS-501:
--
+1
Nits:
# BlockReceiver.java - Indenting of {{op}} declaration around line 854, is not
indented with the rest of the code (though the existing code does not follow 2
space indentation)
# Use {{read(in)}} method instead {{valueOf(in.readByte()}}. Currently both
these methods are used. Perhaps {{valueOf()}} could be private. This will also
be better in case these fields change from byte to some other type.
Use enum to define the constants in DataTransferProtocol
Key: HDFS-501
URL: https://issues.apache.org/jira/browse/HDFS-501
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node, hdfs client
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Attachments: h501_20090724.patch
There are quite a few integer (int/byte) constants defined in
DataTransferProtocol. It is better to replace them by enums.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-501) Use enum to define the constants in DataTransferProtocol
[
https://issues.apache.org/jira/browse/HDFS-501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-501:
Attachment: h501_20090724b.patch
h501_20090724b.patch: incorporated Suresh's comments and cleaned up some codes.
{noformat}
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 12 new or
modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
{noformat}
Use enum to define the constants in DataTransferProtocol
Key: HDFS-501
URL: https://issues.apache.org/jira/browse/HDFS-501
Project: Hadoop HDFS
Issue Type: Improvement
Components: data-node, hdfs client
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Attachments: h501_20090724.patch, h501_20090724b.patch
There are quite a few integer (int/byte) constants defined in
DataTransferProtocol. It is better to replace them by enums.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
