Hello Krishna, Judging from that stack trace, it looks like you have written a client application that uses Apache HTTP Components to make HTTP calls to WebHDFS. The exception is a client-side timeout. I recommend troubleshooting this from the perspective of that client application. The HDFS configuration properties that you mentioned would not control socket timeout for a custom application like this.
I'm not sure what Apache HTTP Components uses by default for socket timeouts. You can explicitly control the timeout programmatically: https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/client/config/RequestConfig.Builder.html#setSocketTimeout(int) You also have the option of writing a custom socket factory class if you need to do very customized socket configuration: http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d5e431 There is one known issue that can impact WebHDFS throughput: https://issues.apache.org/jira/browse/HDFS-8696 This issue would impact Apache Hadoop 2.7 versions, and the fix is currently targeted to Apache Hadoop 2.8.0. I hope this helps. --Chris Nauroth From: Krishna Kishore Bonagiri <write2kish...@gmail.com<mailto:write2kish...@gmail.com>> Date: Wednesday, December 9, 2015 at 2:17 AM To: "user@hadoop.apache.org<mailto:user@hadoop.apache.org>" <user@hadoop.apache.org<mailto:user@hadoop.apache.org>> Subject: Socket Timeout Exception while multiple concurrent applications are reading HDFS data through WebHDFS interface Hi, We are seeing this SocketTImeout exception while a number of concurrent applications (probably, 50 of them) are trying to read HDFS data through WebHDFS interface. Are there any parameters we can tune so it doesn't happen? An exception occurred: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.read(SocketInputStream.java:163) at java.net.SocketInputStream.read(SocketInputStream.java:133) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) at org.apache.http.impl.conn.LoggingSessionInputBuffer.readLine(LoggingSessionInputBuffer.java:115) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:715) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:520) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784) at com.ibm.iis.cc.filesystem.impl.webhdfs.WebHDFS.appendFromBuffer(WebHDFS.java:306) at com.ibm.iis.cc.filesystem.impl.webhdfs.WebHDFS.writeFromStream(WebHDFS.java:198) at com.ibm.iis.cc.filesystem.AbstractFileSystem.writeFromStream(AbstractFileSystem.java:45) at com.ibm.iis.cc.filesystem.FileSystem$Uploader.call(FileSystem.java:3393) at com.ibm.iis.cc.filesystem.FileSystem$Uploader.call(FileSystem.java:3358) at java.util.concurrent.FutureTask.run(FutureTask.java:273) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1176) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641) at java.lang.Thread.run(Thread.java:853) We have tried increasing the values of these parameters, but there is no change. 1) dfs.datanode.handler.count 2) dfs.client.socket-timeout (the new parameter to define the socket timeout) 3) dfs.socket.timeout (the deprecated parameter) 4) dfs.datanode.socket.write.timeout Thanks, Kishore