[
https://issues.apache.org/jira/browse/MAPREDUCE-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921024#action_12921024
]
Scott Chen commented on MAPREDUCE-2036:
---------------------------------------
Hey Wittawat,
bq. 2) Before reducing the replication, I would like to make sure that no block
in the same group located on the same datanode. I have been working on a tool
(similar Balancer) to migrate blocks so that no block in the same group located
on the same datanode.
I like this idea of migrating blocks. Is it possible that you can implement
this on the current RAID project. That will be really helpful.
bq. Plus, I am also working on porting RS codes from Jerasure
(http://www.cs.utk.edu/~plank/plank/papers/CS-08-627.html), so it can support
more than 2 parities.
We have also implemented a java version of RS code in MAPREDUCE-1970. It has
been deployed on our test cluster which holds 300TB of data.
In this patch, we have an interface for general erasure codes.
Maybe you can make your patch implements the same interface. So we can
configure different codecs to use.
I think the encode/decode is more IO-bounded because the parity length we are
using is really small comparing to the regular use cases of RS codes.
> Enable Erasure Code in Tool similar to Hadoop Archive
> -----------------------------------------------------
>
> Key: MAPREDUCE-2036
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2036
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/raid, harchive
> Reporter: Wittawat Tantisiriroj
> Assignee: Wittawat Tantisiriroj
> Priority: Minor
> Attachments: hdfs-raid.tar.gz, MAPREDUCE-2036.patch, RaidTool.pdf
>
>
> Features:
> 1) HAR-like Tool
> 2) RAID5/RAID6 & pluggable interface to implement additional coding
> 3) Enable to group blocks across files
> 4) Portable across cluster since all necessary metadata is embedded
> While it was developed separately from HAR or RAID due to time constraints,
> it would make sense to integrate with either of them.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
