Re: scala: apache httpclient in multi-threaded environment. confusion on correct way of use
On Sat, 2016-11-12 at 20:16 +0530, Nishant Kumar wrote: > Hi, > > I am writing a singleton class (Object in scala) which uses apache > httpclient(4.5.2) to send some file or some json content to a server and > return status to caller. Here is my class - > > object HttpUtils{ > protected val retryHandler = new HttpRequestRetryHandler() { > def retryRequest(exception: IOException, executionCount: Int, > context: HttpContext): Boolean = { > //retry logic > true > } > } > private val connectionManager = new > PoolingHttpClientConnectionManager() > > // Reusing same client for each request that might be coming from > different threads . > // Is it correct Yes. > val httpClient = HttpClients.custom() > .setConnectionManager(connectionManager) > .setRetryHandler(retryHandler) > .build() > > def restApiCall (url : String, rDD: RDD[SomeMessage]) : Boolean = { > // Creating new context for each request > val httpContext: HttpClientContext = HttpClientContext.create > val post = new HttpPost(url) > > // convert RDD to text file using rDD.collect > > // add this file as MultipartEntity to post > > var response = None: Option[CloseableHttpResponse] // Is it correct > way of using it ??? Not really. HttpClient never returns null response messages. The 'response' instance is non-optional. > try { > response = Some(httpClient.execute(post, httpContext)) > val responseCode = response.get.getStatusLine.getStatusCode > EntityUtils.consume(response.get.getEntity) // Is it require ??? Not required but advised if you want to ensure re-use of persistent connections. > if (responseCode == 200) true > else false > } > finally { > if (response.isDefined) response.get.close > post.releaseConnection() // Is it require ??? No, it is not. > } > } > def onShutDown = { > connectionManager.close() > httpClient.close() CloseableHttpClient#close also shuts down the connection pool associated with it. Oleg > } > } > > > Multiple threads (More specifically from spark streaming context) are > calling `restApiCall` method. I am relatively new to `scala` and `apache > httpClient`. I have to make frequent connections to only few fixed server > (i.e. 5-6 fixed URL's with different request parameters). I have also > written comments in above code as question. Please address them as well. > > I went through multiple online resource but still not confident about it. > > - Is it the best way to use http client in multi-threaded environment? > - Is it possible to keep live connections and use it for various requests > ? Will it be beneficial in this case ? > - Am i using/releasing all resources efficiently ? If not please suggest. > - Is it good to use it in Scala or there exist some better library ? > > Thanks for your help in advance. > > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: SSL Handshake : timing and timeout
On Fri, 2016-11-11 at 11:01 +0100, Stefan Magnus Landrø wrote: > True. The way SSL sockets are created is complex stuff. They are layered > and I believe the handshake is async - maybe Oleg could clarify? > There is nothing special or unusual about SSL/TLS handshakes. They involve multiple IP packet exchanges over a TCP connection. Standard socket settings fully apply. Oleg > Anyways - We ran into an issue a few years ago with an SSL handshake taking > for ever (a switch was dropping packets of a certain size ... ), and > tracked our hanging threads down to blocking because the SoTimeout was set > to 0 by default. Changing this prevented threads from hanging for ever. > > Ref javadoc: > > Determines the default socket timeout value for non-blocking I/O operations. > > 2016-11-11 10:01 GMT+01:00 Philippe Mouawad : > > > Thanks but I see not property related to Ssl Handshake timeout. > > > > Regards > > > > On Friday, November 11, 2016, Stefan Magnus Landrø < > > stefan.lan...@gmail.com> > > wrote: > > > > > https://hc.apache.org/httpcomponents-client-ga/ > > > httpclient/apidocs/org/apache/http/impl/client/HttpClientBuilder.html# > > > setDefaultSocketConfig(org.apache.http.config.SocketConfig) > > > > > > 2016-11-11 9:34 GMT+01:00 Philippe Mouawad > > >: > > > > > > > Thank you Stefan > > > > Could you point me to some doc ? > > > > > > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: RetryHandler Per GetMethod
On Thu, 2016-11-10 at 12:54 -0500, Murat Balkan wrote: > Hi, > So I need to put to the Context an attribute and read it in the handler? > Thank you > Yes, you do. Oleg > On 10 November 2016 at 05:38, Bernd Eckenfels > wrote: > > > In fact the retry handlers (some implementations) already look at the > > method and only retry idempotent methods > > > > Gruss > > Bernd > > -- > > http://bernd.eckenfels.net > > > > > > > > > > On Thu, Nov 10, 2016 at 11:15 AM +0100, "Oleg Kalnichevski" < > > ol...@apache.org> wrote: > > > > > > > > > > > > > > > > > > > > > > On Wed, 2016-11-09 at 09:05 -0500, Murat Balkan wrote: > > > Hello, > > > > > > Is it possible to set a retry handler per HttpGet? The following code > > taken > > > from the documentation only applies to 3x and seems to be deprecated. > > > > > > > > > httpget.getParams(). > > > setParameter(HttpMethodParams.RETRY_HANDLER, myretryhandler); > > > > > > > > > What I want to achieve is to set disableAutomaticRetries on the > > httpclient > > > level and set them at the httpget level. > > > > > > Is this supported? > > > > > > > No it is not, but one can use HttpContext attributes to customize the > > handler's behavior on the per request basis. > > > > Oleg > > > > > > - > > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > > > > > > > > > > > > > > > > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
Re: NoHttpResponseException in the async client
On Thu, 2016-11-10 at 11:31 +0100, Joan Balagueró - ventusproxy wrote: > Hello, > > We have replaced the httpclient by the async client in our application. > Everything works fine, but the ‘NoHttpResponseException’ has disappeared > from our error statistics reports. So, or with the new async pool this error > does not occur for some reason (that I don’t know) or we are not catching > this error correctly (more likely). Async HC does not throw this exception. It is specific to the classic (blocking) HC. Oleg > > We are using an ‘HttpAsyncResponseConsumer’ and overwriting the > ‘responseReceived’, ‘consumeContent’ and ‘failed’ methods. We > understand that when a ‘NoHttpResponseException’ occurs, > ‘responseReceived’ and ‘consumeContent’ are not called, and the > ‘failed’ method is the only one that is directly called. > > Our ‘failed’ method looks like this: > > @Override > public void failed(final Exception e) > { > ProxyServletException pse = null; > > if (e.getClass() == java.net.SocketTimeoutException.class) pse = new > ProxyServletException(ErrorConstants.HTTP_RESPONSE_TIMEOUT, e); > else if (e.getClass() == java.net.ConnectException.class) pse = new > ProxyServletException(ErrorConstants.HTTP_CONNECT_TIMEOUT, e); > else if (e.getClass() == org.apache.http.NoHttpResponseException.class) > pse = new ProxyServletException(ErrorConstants.HTTP_NO_RESPONSE, e); ← > the error is caugth here > else if (e.getClass() == java.io.IOException.class) pse = new > ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e); > else if (e.getClass() == > com.ventusproxy.proxy.servlet.ProxyServletException.class) pse = > (ProxyServletException)e; > else if (e.getClass() == > org.apache.http.conn.ConnectionPoolTimeoutException.class) pse = new > ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e); > else if (e.getClass() == java.util.concurrent.TimeoutException.class) pse > = new ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e); > > pse = (pse != null ? pse : new > ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e)); > > ( . . . ) > } > > > Is this ok? Or I'm missing something? > > Thanks, > > Joan. > > > - > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org > For additional commands, e-mail: httpclient-users-h...@hc.apache.org > - To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org For additional commands, e-mail: httpclient-users-h...@hc.apache.org
scala: apache httpclient in multi-threaded environment. confusion on correct way of use
Hi, I am writing a singleton class (Object in scala) which uses apache httpclient(4.5.2) to send some file or some json content to a server and return status to caller. Here is my class - object HttpUtils{ protected val retryHandler = new HttpRequestRetryHandler() { def retryRequest(exception: IOException, executionCount: Int, context: HttpContext): Boolean = { //retry logic true } } private val connectionManager = new PoolingHttpClientConnectionManager() // Reusing same client for each request that might be coming from different threads . // Is it correct val httpClient = HttpClients.custom() .setConnectionManager(connectionManager) .setRetryHandler(retryHandler) .build() def restApiCall (url : String, rDD: RDD[SomeMessage]) : Boolean = { // Creating new context for each request val httpContext: HttpClientContext = HttpClientContext.create val post = new HttpPost(url) // convert RDD to text file using rDD.collect // add this file as MultipartEntity to post var response = None: Option[CloseableHttpResponse] // Is it correct way of using it ??? try { response = Some(httpClient.execute(post, httpContext)) val responseCode = response.get.getStatusLine.getStatusCode EntityUtils.consume(response.get.getEntity) // Is it require ??? if (responseCode == 200) true else false } finally { if (response.isDefined) response.get.close post.releaseConnection() // Is it require ??? } } def onShutDown = { connectionManager.close() httpClient.close() } } Multiple threads (More specifically from spark streaming context) are calling `restApiCall` method. I am relatively new to `scala` and `apache httpClient`. I have to make frequent connections to only few fixed server (i.e. 5-6 fixed URL's with different request parameters). I have also written comments in above code as question. Please address them as well. I went through multiple online resource but still not confident about it. - Is it the best way to use http client in multi-threaded environment? - Is it possible to keep live connections and use it for various requests ? Will it be beneficial in this case ? - Am i using/releasing all resources efficiently ? If not please suggest. - Is it good to use it in Scala or there exist some better library ? Thanks for your help in advance. -- Nishant Kumar Bangalore, India Mob: +91 80088 42030 Email: nishantkuma...@gmail.com