Re: scala: apache httpclient in multi-threaded environment. confusion on correct way of use

[Oleg Kalnichevski]

On Sat, 2016-11-12 at 20:16 +0530, Nishant Kumar wrote:
> Hi,
> 
> I am writing a singleton class (Object in scala) which uses apache
> httpclient(4.5.2) to send some file or some json content to a server and
> return status to caller. Here is my class -
> 
> object HttpUtils{
>   protected val retryHandler = new HttpRequestRetryHandler() {
> def retryRequest(exception: IOException, executionCount: Int,
> context: HttpContext): Boolean = {
>   //retry logic
>   true
> }
>   }
>   private val connectionManager = new
> PoolingHttpClientConnectionManager()
> 
>   // Reusing same client for each request that might be coming from
> different threads .
>   // Is it correct 

Yes.

>   val httpClient = HttpClients.custom()
> .setConnectionManager(connectionManager)
> .setRetryHandler(retryHandler)
> .build()
> 
>   def restApiCall (url : String, rDD: RDD[SomeMessage]) : Boolean = {
> // Creating new context for each request
> val httpContext: HttpClientContext = HttpClientContext.create
> val post = new HttpPost(url)
> 
> // convert RDD to text file using rDD.collect
> 
> // add this file as MultipartEntity to post
> 
> var response = None: Option[CloseableHttpResponse] // Is it correct
> way of using it ???

Not really. HttpClient never returns null response messages. The
'response' instance is non-optional.

> try {
>   response = Some(httpClient.execute(post, httpContext))
>   val responseCode = response.get.getStatusLine.getStatusCode
>   EntityUtils.consume(response.get.getEntity) // Is it require ???

Not required but advised if you want to ensure re-use of persistent
connections.

>   if (responseCode == 200) true
>   else false
> }
> finally {
>   if (response.isDefined) response.get.close
>   post.releaseConnection() // Is it require ???

No, it is not.

> }
>   }
>   def onShutDown = {
> connectionManager.close()
> httpClient.close()

CloseableHttpClient#close also shuts down the connection pool associated
with it.

Oleg

>   }
> }
> 
> 
> Multiple threads (More specifically from spark streaming context) are
> calling `restApiCall` method. I am relatively new to `scala` and `apache
> httpClient`. I have to make frequent connections to only few fixed server
> (i.e. 5-6 fixed URL's with different request parameters). I have also
> written comments in above code as question. Please address them as well.
> 
> I went through multiple online resource but still not confident about it.
> 
>  - Is it the best way to use http client in multi-threaded environment?
>  - Is it possible to keep live connections and use it for various requests
> ? Will it be beneficial in this case ?
>  - Am i using/releasing all resources efficiently ? If not please suggest.
>  - Is it good to use it in Scala or there exist some better library ?
> 
>  Thanks for your help in advance.
> 
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: SSL Handshake : timing and timeout

[Oleg Kalnichevski]

On Fri, 2016-11-11 at 11:01 +0100, Stefan Magnus Landrø wrote:
> True. The way SSL sockets are created is complex stuff. They are layered
> and I believe the handshake is async - maybe Oleg could clarify?
> 

There is nothing special or unusual about SSL/TLS handshakes. They
involve multiple IP packet exchanges over a TCP connection. Standard
socket settings fully apply.

Oleg

> Anyways - We ran into an issue a few years ago with an SSL handshake taking
> for ever (a switch was dropping packets of a certain size ... ), and
> tracked our hanging threads down to blocking because the SoTimeout was set
> to 0 by default. Changing this prevented threads from hanging for ever.
> 
> Ref javadoc:
> 
> Determines the default socket timeout value for non-blocking I/O operations.
> 
> 2016-11-11 10:01 GMT+01:00 Philippe Mouawad :
> 
> > Thanks but I see not property related to Ssl Handshake timeout.
> >
> > Regards
> >
> > On Friday, November 11, 2016, Stefan Magnus Landrø <
> > stefan.lan...@gmail.com>
> > wrote:
> >
> > > https://hc.apache.org/httpcomponents-client-ga/
> > > httpclient/apidocs/org/apache/http/impl/client/HttpClientBuilder.html#
> > > setDefaultSocketConfig(org.apache.http.config.SocketConfig)
> > >
> > > 2016-11-11 9:34 GMT+01:00 Philippe Mouawad  > > >:
> > >
> > > > Thank you Stefan
> > > > Could you point me to some doc ?
> > > >
> >



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: RetryHandler Per GetMethod

[Oleg Kalnichevski]

On Thu, 2016-11-10 at 12:54 -0500, Murat Balkan wrote:
> Hi,
> So I need to put to the Context an attribute and read it in the handler?
> Thank you
> 

Yes, you do. 

Oleg

> On 10 November 2016 at 05:38, Bernd Eckenfels 
> wrote:
> 
> > In fact the retry handlers (some implementations)  already look at the
> > method and only retry idempotent methods
> >
> > Gruss
> > Bernd
> > --
> > http://bernd.eckenfels.net
> >
> >
> >
> >
> > On Thu, Nov 10, 2016 at 11:15 AM +0100, "Oleg Kalnichevski" <
> > ol...@apache.org> wrote:
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, 2016-11-09 at 09:05 -0500, Murat Balkan wrote:
> > > Hello,
> > >
> > > Is it possible to set a retry handler per HttpGet? The following code
> > taken
> > > from the documentation only applies to 3x and seems to be deprecated.
> > >
> > >
> > > httpget.getParams().
> > > setParameter(HttpMethodParams.RETRY_HANDLER, myretryhandler);
> > >
> > >
> > > What I want to achieve is to set disableAutomaticRetries on the
> > httpclient
> > > level and set them at the httpget level.
> > >
> > > Is this supported?
> > >
> >
> > No it is not, but one can use HttpContext attributes to customize the
> > handler's behavior on the per request basis.
> >
> > Oleg
> >
> >
> > -
> > To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> > For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> >
> >
> >
> >
> >
> >
> >
> 
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

Re: NoHttpResponseException in the async client

[Oleg Kalnichevski]

On Thu, 2016-11-10 at 11:31 +0100, Joan Balagueró - ventusproxy wrote:
> Hello,
> 
> We have replaced the httpclient by the async client in our application.
> Everything works fine, but the ‘NoHttpResponseException’ has disappeared
> from our error statistics reports. So, or with the new async pool this error
> does not occur for some reason (that I don’t know) or we are not catching
> this error correctly (more likely).

Async HC does not throw this exception. It is specific to the classic
(blocking) HC.

Oleg 

> 
> We are using an ‘HttpAsyncResponseConsumer’ and overwriting the
> ‘responseReceived’, ‘consumeContent’ and ‘failed’ methods. We
> understand that when a ‘NoHttpResponseException’ occurs,
> ‘responseReceived’ and ‘consumeContent’ are not called, and  the
> ‘failed’ method is the only one that is directly called.
> 
> Our ‘failed’ method looks like this:
> 
> @Override
> public void failed(final Exception e)
> {
>   ProxyServletException pse = null;
> 
>   if (e.getClass() == java.net.SocketTimeoutException.class) pse = new
> ProxyServletException(ErrorConstants.HTTP_RESPONSE_TIMEOUT, e);
>   else if (e.getClass() == java.net.ConnectException.class) pse = new
> ProxyServletException(ErrorConstants.HTTP_CONNECT_TIMEOUT, e);
>   else if (e.getClass() == org.apache.http.NoHttpResponseException.class)
> pse = new ProxyServletException(ErrorConstants.HTTP_NO_RESPONSE, e);  ←
> the error is caugth here 
>   else if (e.getClass() == java.io.IOException.class) pse = new
> ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e);
>   else if (e.getClass() ==
> com.ventusproxy.proxy.servlet.ProxyServletException.class) pse =
> (ProxyServletException)e;
>   else if (e.getClass() ==
> org.apache.http.conn.ConnectionPoolTimeoutException.class) pse = new
> ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e);
>   else if (e.getClass() == java.util.concurrent.TimeoutException.class) pse
> = new ProxyServletException(ErrorConstants.HTTP_MAX_CONNECTIONS, e);
> 
>   pse = (pse != null ? pse : new
> ProxyServletException(ErrorConstants.HTTP_GENERIC_HTTP, e));
> 
>   ( . . . )
> }
> 
> 
> Is this ok? Or I'm missing something?
> 
> Thanks,
> 
> Joan.
> 
> 
> -
> To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
> For additional commands, e-mail: httpclient-users-h...@hc.apache.org
> 



-
To unsubscribe, e-mail: httpclient-users-unsubscr...@hc.apache.org
For additional commands, e-mail: httpclient-users-h...@hc.apache.org

scala: apache httpclient in multi-threaded environment. confusion on correct way of use

[Nishant Kumar]

Hi,

I am writing a singleton class (Object in scala) which uses apache
httpclient(4.5.2) to send some file or some json content to a server and
return status to caller. Here is my class -

object HttpUtils{
  protected val retryHandler = new HttpRequestRetryHandler() {
def retryRequest(exception: IOException, executionCount: Int,
context: HttpContext): Boolean = {
  //retry logic
  true
}
  }
  private val connectionManager = new
PoolingHttpClientConnectionManager()

  // Reusing same client for each request that might be coming from
different threads .
  // Is it correct 
  val httpClient = HttpClients.custom()
.setConnectionManager(connectionManager)
.setRetryHandler(retryHandler)
.build()

  def restApiCall (url : String, rDD: RDD[SomeMessage]) : Boolean = {
// Creating new context for each request
val httpContext: HttpClientContext = HttpClientContext.create
val post = new HttpPost(url)

// convert RDD to text file using rDD.collect

// add this file as MultipartEntity to post

var response = None: Option[CloseableHttpResponse] // Is it correct
way of using it ???
try {
  response = Some(httpClient.execute(post, httpContext))
  val responseCode = response.get.getStatusLine.getStatusCode
  EntityUtils.consume(response.get.getEntity) // Is it require ???
  if (responseCode == 200) true
  else false
}
finally {
  if (response.isDefined) response.get.close
  post.releaseConnection() // Is it require ???
}
  }
  def onShutDown = {
connectionManager.close()
httpClient.close()
  }
}


Multiple threads (More specifically from spark streaming context) are
calling `restApiCall` method. I am relatively new to `scala` and `apache
httpClient`. I have to make frequent connections to only few fixed server
(i.e. 5-6 fixed URL's with different request parameters). I have also
written comments in above code as question. Please address them as well.

I went through multiple online resource but still not confident about it.

 - Is it the best way to use http client in multi-threaded environment?
 - Is it possible to keep live connections and use it for various requests
? Will it be beneficial in this case ?
 - Am i using/releasing all resources efficiently ? If not please suggest.
 - Is it good to use it in Scala or there exist some better library ?

 Thanks for your help in advance.


-- 
Nishant Kumar
Bangalore, India
Mob: +91 80088 42030
Email: nishantkuma...@gmail.com