[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Hadoop Flags: Reviewed > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Fix For: HDFS-7285 > > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-HDFS-7285-v6.patch, HADOOP-11847-HDFS-7285-v7.patch, > HADOOP-11847-HDFS-7285-v8.patch, HADOOP-11847-HDFS-7285-v9.patch, > HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Fix Version/s: HDFS-7285 > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Fix For: HDFS-7285 > > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-HDFS-7285-v6.patch, HADOOP-11847-HDFS-7285-v7.patch, > HADOOP-11847-HDFS-7285-v8.patch, HADOOP-11847-HDFS-7285-v9.patch, > HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Resolution: Fixed Status: Resolved (was: Patch Available) It was just committed to the branch. Thanks [~hitliuyi] and [~zhz] for the great review and comments! > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-HDFS-7285-v6.patch, HADOOP-11847-HDFS-7285-v7.patch, > HADOOP-11847-HDFS-7285-v8.patch, HADOOP-11847-HDFS-7285-v9.patch, > HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-HDFS-7285-v9.patch Thanks Yi for the more review. Updated the patch addressing the comment. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-HDFS-7285-v6.patch, HADOOP-11847-HDFS-7285-v7.patch, > HADOOP-11847-HDFS-7285-v8.patch, HADOOP-11847-HDFS-7285-v9.patch, > HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-HDFS-7285-v8.patch Updated the patch addressing the Jenkins reported issues. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-HDFS-7285-v6.patch, HADOOP-11847-HDFS-7285-v7.patch, > HADOOP-11847-HDFS-7285-v8.patch, HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-HDFS-7285-v7.patch Updated the patch according to review comments and above discussions. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-HDFS-7285-v6.patch, HADOOP-11847-HDFS-7285-v7.patch, > HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-HDFS-7285-v6.patch Re-uploaded wiht the correct patch. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-HDFS-7285-v6.patch, HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: (was: HADOOP-11847-v6.patch) > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-v6.patch Updated patch according to review comments. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-v1.patch, HADOOP-11847-v2.patch, HADOOP-11847-v6.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: (was: HADOOP-11847-v5.patch) > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-HDFS-7285-v5.patch Rebased after HADOOP-11938. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-HDFS-7285-v5.patch, > HADOOP-11847-v1.patch, HADOOP-11847-v2.patch, HADOOP-11847-v5.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-v5.patch Updated the patch rebased, only focusing on the task defined in the issue description. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-v1.patch, > HADOOP-11847-v2.patch, HADOOP-11847-v5.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HADOOP-11847: -- Labels: BB2015-05-TBR (was: ) > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Labels: BB2015-05-TBR > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-HDFS-7285-v4.patch Renamed the patch to trigger the building successfully. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HADOOP-11847-HDFS-7285-v3.patch, > HADOOP-11847-HDFS-7285-v4.patch, HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: (was: HADOOP-11847-v4.patch) > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HADOOP-11847-HDFS-7285-v3.patch, HADOOP-11847-v1.patch, > HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-v4.patch Updated the patch: * Incorportated some refactoring changes from work for other issues; * Checked and fixed some issues found by Hadoop QA building; * Added more tests and comments. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HADOOP-11847-HDFS-7285-v3.patch, HADOOP-11847-v1.patch, > HADOOP-11847-v2.patch, HADOOP-11847-v4.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-HDFS-7285-v3.patch Updated the patch. Changes summary: 1. Addressed some review comments as discussed above; 2. Enhanced the raw coder tests to ensure: * A coder can be repeatedly used with either bytes array buffers or direct buffers; * A coder can be repeatedly used sometimes using array buffers, and sometimes using direct buffers; * Added more comments. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HADOOP-11847-HDFS-7285-v3.patch, HADOOP-11847-v1.patch, > HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Target Version/s: HDFS-7285 Status: Patch Available (was: Open) > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HADOOP-11847-HDFS-7285-v3.patch, HADOOP-11847-v1.patch, > HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-v2.patch Refactored related codes to make it more clear: * As not all decoders having to adjust the decoding parameters, I moved the related codes from base class; * Removed the trick of RS->XOR, as it's overhead is obvious; * Renamed some variables, more comments and etc. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HADOOP-11847-v1.patch, HADOOP-11847-v2.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Description: This is to enhance raw erasure coder to allow only reading least required inputs while decoding. It will also refine and document the relevant APIs for better understanding and usage. When using least required inputs, it may add computating overhead but will possiblly outperform overall since less network traffic and disk IO are involved. This is something planned to do but just got reminded by [~zhz]' s question raised in HDFS-7678, also copied here: bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should I construct the inputs to RawErasureDecoder#decode? With this work, hopefully the answer to above question would be obvious. was: This is to enhance raw erasure coder to allow only reading least required inputs while decoding. It will also refine and document the relevant APIs for better understanding and usage. This is something planned to do but just got reminded by [~zhz]' s question raised in HDFS-7678, also copied here: bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should I construct the inputs to RawErasureDecoder#decode? With this work, hopefully the answer to above question would be obvious. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HADOOP-11847-v1.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. When using least required inputs, it may add > computating overhead but will possiblly outperform overall since less network > traffic and disk IO are involved. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11847) Enhance raw coder allowing to read least required inputs in decoding
[ https://issues.apache.org/jira/browse/HADOOP-11847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kai Zheng updated HADOOP-11847: --- Attachment: HADOOP-11847-v1.patch Uploaded the patch. Changes summary: * Allowing least required inputs when decoding, using null to indicate not to read; * Refining tests allowing to erase parity units; * Refactored some codes incorporating change from other issues. Will open similar issue separately for erasure coders as this is already rather large. Pending for review. > Enhance raw coder allowing to read least required inputs in decoding > > > Key: HADOOP-11847 > URL: https://issues.apache.org/jira/browse/HADOOP-11847 > Project: Hadoop Common > Issue Type: Sub-task > Components: io >Reporter: Kai Zheng >Assignee: Kai Zheng > Attachments: HADOOP-11847-v1.patch > > > This is to enhance raw erasure coder to allow only reading least required > inputs while decoding. It will also refine and document the relevant APIs for > better understanding and usage. > This is something planned to do but just got reminded by [~zhz]' s question > raised in HDFS-7678, also copied here: > bq.Kai Zheng I have a question about decoding: in a (6+3) schema, if block #2 > is missing, and I want to repair it with blocks 0, 1, 3, 4, 5, 8, how should > I construct the inputs to RawErasureDecoder#decode? > With this work, hopefully the answer to above question would be obvious. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
