[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12917286#action_12917286 ] shravankumar commented on HDFS-503: --- @Ramkumar vadali.Thank you sir Where can i access the raid code(which have been fixed). > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12914757#action_12914757 ] shravankumar commented on HDFS-503: --- Can any one help me. In which stable version of hadoop this raid become a part and how can i access the API documents related to raid > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876187#action_12876187 ] shravankumar commented on HDFS-503: --- Hello sir, 1. what is the meaning for this 2. In ADMINISTRATION they mentioned RaidNode Software what it means. 3. In HADOOP_HOME, run ant package to build Hadoop and its contrib packages. This will come when we installed hadoop 0.20.1 or we need download ant package. > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12876178#action_12876178 ] shravankumar commented on HDFS-503: --- Thank you sir. > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874518#action_12874518 ] shravankumar commented on HDFS-503: --- The tags(property,description) used in programming are normal HTML Tags or they have different meaning. Can you send me the document which consist of meanings of these tags. > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12874514#action_12874514 ] shravankumar commented on HDFS-503: --- Hello sir, 1. what is the meaning for this 2. In ADMINISTRATION they mentioned RaidNode Software what it means. 3. In HADOOP_HOME, run ant package to build Hadoop and its contrib packages. This will come when we installed hadoop 0.20.1 or we need download ant package. > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873913#action_12873913 ] shravankumar commented on HDFS-503: --- Thank you. > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873572#action_12873572 ] shravankumar commented on HDFS-503: --- For raid1.txt and raid2.txt any DESIGN DIAGRAMS like Class Diagram are there. > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12873571#action_12873571 ] shravankumar commented on HDFS-503: --- Thank you sir. I have one more query both raid1.txt and raid2.txt looks similar what is the difference between them. In the implementation for parity whether they are used NORMAL CRC OR SOME OTHER MECHANISMS like REED SOLOMON CODES. Shravan Kumar. > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.21.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-503) Implement erasure coding as a layer on HDFS
[ https://issues.apache.org/jira/browse/HDFS-503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12861740#action_12861740 ] shravankumar commented on HDFS-503: --- Dear sir, I have downloaded the code for "Implement erasure coding as a layer on HDFS". But i was unable to execute it. Please guide me regarding this. > Implement erasure coding as a layer on HDFS > --- > > Key: HDFS-503 > URL: https://issues.apache.org/jira/browse/HDFS-503 > Project: Hadoop HDFS > Issue Type: New Feature > Components: contrib/raid >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Fix For: 0.22.0 > > Attachments: raid1.txt, raid2.txt > > > The goal of this JIRA is to discuss how the cost of raw storage for a HDFS > file system can be reduced. Keeping three copies of the same data is very > costly, especially when the size of storage is huge. One idea is to reduce > the replication factor and do erasure coding of a set of blocks so that the > over probability of failure of a block remains the same as before. > Many forms of error-correcting codes are available, see > http://en.wikipedia.org/wiki/Erasure_code. Also, recent research from CMU has > described DiskReduce > https://opencirrus.org/system/files/Gibson-OpenCirrus-June9-09.ppt. > My opinion is to discuss implementation strategies that are not part of base > HDFS, but is a layer on top of HDFS. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
