[
https://issues.apache.org/jira/browse/HDFS-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12880991#action_12880991
]
sam rash commented on HDFS-895:
-------------------------------
re: the patch
I realize in the hadoop code we already swallow InterruptedException
frequently, but I think you can change the trend here:
{code}
// wait for all acks to be received back from datanodes
synchronized (ackQueue) {
if (!closed && ackQueue.size() != 0) {
try {
ackQueue.wait();
} catch (InterruptedException e) {
Thread.currentThread.interrupt(); //add this
}
continue;
}
}
{code}
otherwise, it's very easy to have a thread that I own and manage that has a
DFSOutputStream in it that swallows an interrupt. when i check
Thread.currentThread.isInterrupted() to see if one of my other threads has
interrupted me, i will not see it
(the crux here is that swallowing interrupts in threads that hadoop controls
are less harmful--this is directly in client code when you call sync()/close())
> Allow hflush/sync to occur in parallel with new writes to the file
> ------------------------------------------------------------------
>
> Key: HDFS-895
> URL: https://issues.apache.org/jira/browse/HDFS-895
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs client
> Affects Versions: 0.22.0
> Reporter: dhruba borthakur
> Assignee: Todd Lipcon
> Fix For: 0.22.0
>
> Attachments: hdfs-895-0.20-append.txt, hdfs-895-20.txt,
> hdfs-895-trunk.txt, hdfs-895.txt
>
>
> In the current trunk, the HDFS client methods writeChunk() and hflush./sync
> are syncronized. This means that if a hflush/sync is in progress, an
> applicationn cannot write data to the HDFS client buffer. This reduces the
> write throughput of the transaction log in HBase.
> The hflush/sync should allow new writes to happen to the HDFS client even
> when a hflush/sync is in progress. It can record the seqno of the message for
> which it should receice the ack, indicate to the DataStream thread to star
> flushing those messages, exit the synchronized section and just wai for that
> ack to arrive.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
