[
https://issues.apache.org/jira/browse/HDFS-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744316#comment-14744316
]
Zhe Zhang commented on HDFS-9040:
---------------------------------
bq. With BlockGroupDataStreamer I can make 9 internal streamers to wait for
error-handling to be finished, until then I put empty_last_packet to all 9
internal streamers to let them close blockStreams.
bq. I actually did similar thing: closeImpl() first let all the streamers to
flush out all the data packets, then call checkStreamerFailures to handle any
failure during the data transfer, and in the end to send out the last empty
packet to close the packet. But the challenge here is, we could not use the
same way to handle the failure for the last empty packet, since successful
streamers may have closed the block already.
If we can preallocate a fixed number of GS's (e.g. {{NUM_PARITY_BLOCKS}}), we
can bump a streamer's GS by {{NUM_PARITY_BLOCKS}} when it successfully closes.
When all healthy streamers successfully close, we should bump the NN version of
GS. We might need to add hooks in NN accordingly.
> Erasure coding: Refactor DFSStripedOutputStream (Move Namenode RPC Requests
> to Coordinator)
> -------------------------------------------------------------------------------------------
>
> Key: HDFS-9040
> URL: https://issues.apache.org/jira/browse/HDFS-9040
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Reporter: Walter Su
> Attachments: HDFS-9040.00.patch, HDFS-9040.001.wip.patch,
> HDFS-9040.02.bgstreamer.patch
>
>
> The general idea is to simplify error handling logic.
> Proposal 1:
> A BlockGroupDataStreamer to communicate with NN to allocate/update block, and
> StripedDataStreamer s only have to stream blocks to DNs.
> Proposal 2:
> See below the
> [comment|https://issues.apache.org/jira/browse/HDFS-9040?focusedCommentId=14741388&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14741388]
> from [~jingzhao].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
