[
https://issues.apache.org/jira/browse/HDFS-7678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7678:
----------------------------
Attachment: HDFS-7678-HDFS-7285.013.patch
013 patch passes all existing tests.
One caveat is that as [~hitliuyi] [found |
https://issues.apache.org/jira/browse/HDFS-8347?focusedCommentId=14533932&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14533932]
yesterday, there's a codec bug. So I'm filling fixed content into decoded
buffer and also manually verified that the correct range from parity blocks is
fetched. I will file a separate JIRA to create a simulated codec algorithm for
more isolated unit testing.
[~jingzhao] I added the test you suggested and it passes now (I tested failing
each individual DN). I'm working on adding extensive unit tests for the new
{{StripedBlockUtil}} logic. But as the patch is getting big now, please let me
know if you think I should split the {{StripedBlockUtil}} changes to HDFS-8320.
If we assume the new {{StripedBlockUtil}} arithmetic calculations are correct,
the new pread logic in {{DFSStripedInputStream}} is quite simple. So maybe it's
easier to review them separately.
> Erasure coding: DFSInputStream with decode functionality (pread)
> ----------------------------------------------------------------
>
> Key: HDFS-7678
> URL: https://issues.apache.org/jira/browse/HDFS-7678
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Affects Versions: HDFS-7285
> Reporter: Li Bo
> Assignee: Zhe Zhang
> Attachments: BlockGroupReader.patch, HDFS-7678-HDFS-7285.002.patch,
> HDFS-7678-HDFS-7285.003.patch, HDFS-7678-HDFS-7285.004.patch,
> HDFS-7678-HDFS-7285.005.patch, HDFS-7678-HDFS-7285.006.patch,
> HDFS-7678-HDFS-7285.007.patch, HDFS-7678-HDFS-7285.008.patch,
> HDFS-7678-HDFS-7285.009.patch, HDFS-7678-HDFS-7285.010.patch,
> HDFS-7678-HDFS-7285.012.patch, HDFS-7678-HDFS-7285.013.patch,
> HDFS-7678.000.patch, HDFS-7678.001.patch
>
>
> A block group reader will read data from BlockGroup no matter in striping
> layout or contiguous layout. The corrupt blocks can be known before
> reading(told by namenode), or just be found during reading. The block group
> reader needs to do decoding work when some blocks are found corrupt.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
