[
https://issues.apache.org/jira/browse/HDFS-12222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16131022#comment-16131022
]
Andrew Wang commented on HDFS-12222:
------------------------------------
Thanks for picking this up Huafeng!
I liked Alex's proposal above to have the current public APIs for BlockLocation
return just data blocks. Then rework the HDFS internals that need both data and
parity blocks to call a new API. We might already have this separation in
place, since the DFSClient uses the private-only LocatedBlock and
HdfsFileStatus classes.
This sketch looks a bit different, but I think can be tweaked to fit. A couple
review comments:
* If we agree that FileSystem users likely don't care about the details of the
EC schema or even the parity blocks, then we don't need the
ErasureCodedBlockLocation class. Just change the makeQualifiedLocated and
related to just return data blocks.
* We need to be careful about strictly adding new methods when adding a new
parameter, for compatibility.
> Add EC information to BlockLocation
> -----------------------------------
>
> Key: HDFS-12222
> URL: https://issues.apache.org/jira/browse/HDFS-12222
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 3.0.0-alpha1
> Reporter: Andrew Wang
> Assignee: Huafeng Wang
> Labels: hdfs-ec-3.0-nice-to-have
> Attachments: HDFS-12222.001.patch
>
>
> HDFS applications query block location information to compute splits. One
> example of this is FileInputFormat:
> https://github.com/apache/hadoop/blob/d4015f8628dd973c7433639451a9acc3e741d2a2/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L346
> You see bits of code like this that calculate offsets as follows:
> {noformat}
> long bytesInThisBlock = blkLocations[startIndex].getOffset() +
> blkLocations[startIndex].getLength() - offset;
> {noformat}
> EC confuses this since the block locations include parity block locations as
> well, which are not part of the logical file length. This messes up the
> offset calculation and thus topology/caching information too.
> Applications can figure out what's a parity block by reading the EC policy
> and then parsing the schema, but it'd be a lot better if we exposed this more
> generically in BlockLocation instead.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
