[
https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744281#comment-14744281
]
Jing Zhao commented on HDFS-9040:
---------------------------------
Thanks for the comments, Zhe, Walter and Bo!
bq. With BlockGroupDataStreamer I can make 9 internal streamers to wait for
error-handling to be finished, until then I put empty_last_packet to all 9
internal streamers to let them close blockStreams.
I actually did similar thing: closeImpl() first let all the streamers to flush
out all the data packets, then call checkStreamerFailures to handle any failure
during the data transfer, and in the end to send out the last empty packet to
close the packet. But the challenge here is, we could not use the same way to
handle the failure for the last empty packet, since successful streamers may
have closed the block already.
bq. preallocate GS when NN creates a new striped block group
(FSN#createNewBlock).
This is a very good idea. We should spend more time exploring this
optimization, but maybe as a follow-on task?
bq. The reason I prefer not to do locateFollowingBlock in DFSOutputStream is,
DFSOutputStream is async with DataStreamer
Yeah, I can understand your concern. In the replication mechanism, the async
implementation matches the single write pipeline model, and the datastreamer
can handle its failure perfectly. But with 9 streamers in parallel, we need to
1) sync all the streamers when writing a new block, and 2) stop all the
streamers and assign them with new GS when failure happens. Thus I think we'd
better add some sync code in DFSStripedOutputStream. Also in this way it
becomes easier to calculate block length and set/reset external error state.
> Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests
> to Coordinator)
> -------------------------------------------------------------------------------------------
>
> Key: HDFS-9040
> URL: https://issues.apache.org/jira/browse/HDFS-9040
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Walter Su
> Attachments: HDFS-9040.00.patch, HDFS-9040.001.wip.patch,
> HDFS-9040.02.bgstreamer.patch
>
>
> The general idea is to simplify error handling logic.
> Proposal 1:
> A BlockGroupDataStreamer to communicate with NN to allocate/update block, and
> StripedDataStreamer s only have to stream blocks to DNs.
> Proposal 2:
> See below the
> [comment|https://issues.apache.org/jira/browse/HDFS-9040?focusedCommentId=14741388&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14741388]
> from [~jingzhao].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
