Hi Mingxi, In your thread dump, did you check DataStreamer thread? is it running?
If DataStreamer thread is not running, then this issue would be mostly same as HDFS-2850. Did you find any OOME in your clients? Regards, Uma ________________________________________ From: Mingxi Wu [mingxi...@turn.com] Sent: Monday, April 16, 2012 7:25 AM To: common-user@hadoop.apache.org Subject: upload hang at DFSClient$DFSOutputStream.close(3488) Hi, I use hadoop cloudera 0.20.2-cdh3u0. I have a program which uploads local files to HDFS every hour. Basically, I open a gzip input stream by in= new GZIPInputStream(fin); And write to HDFS file. After less than two days, it will hang. It hangs at FSDataOutputStream.close(86). Here is the stack: State: WAITING Running 16660 ms (user 13770 ms) blocked 11276 times for <> ms waiting 11209 times for <> ms LockName: java.util.LinkedList@f1ca0de LockOwnerId: -1 java.lang.Object.wait(-2) java.lang.Object.wait(485) org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.waitForAckedSeqno(3468) org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.flushInternal(3457) org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(3549) org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(3488) org.apache.hadoop.fs.FSDataOutputStream$PositionCache.close(61) org.apache.hadoop.fs.FSDataOutputStream.close(86) org.apache.hadoop.io.IOUtils.copyBytes(59) org.apache.hadoop.io.IOUtils.copyBytes(74) Any suggestion to avoid this issue? It seems this is a bug in hadoop. I found this issue is less severe when my upload server do one upload at a time, instead of using multiple concurrent uploads. Thanks, Mingxi