Idan,
Consider you have 1000 concurrent tuples ... and the spout does not throttle traffic. It means that the last bolt would be handling 1000 concurrent requests. Now consider you have 100,000 concurrent tuples.... Eventually the operating system or the NIO buffer would exhaust its resources. You would have been better off with throtteling. The output collector is the object that you perform "ack" or "fail" the tuple. You probably call them from a future callback. Make sure that all of these callbacks are called from the same thread, or are synchronized. Itai ________________________________ From: Idan Fridman <[email protected]> Sent: Tuesday, December 16, 2014 3:58 PM To: [email protected] Subject: Re: Using AsyncHttpReuqest inside a Bolt Hi, Any non-blocking bolt does not push back on the previous bolt if it is out of resources. So you should consider using max-spout-pending for spout level throttling. @Itai, My async bolt is the last bolt in the chain. so i guess I dont have this problem?? Keep in mind you'll need to synchronize the OutputCollector when your NIO response workers handle the returned requests as OutputCollector is not thread safe. @Michael, I am not sure how the OutputCollector is concerned to my issue? This is my execution code.. that code could caouse me any side-effects in my topology? @Override public void execute(Tuple tuple, BasicOutputCollector basicOutputCollector) { PushMessage pushMessage = (PushMessage) tuple.getValueByField("pushMessage"); final String messageId = pushMessage.getMessageId(); asyncHttpClient.preparePost("some_url").execute(new AsyncCompletionHandler<Response>() { @Override public Response onCompleted(Response response) throws Exception { String innerMessageId = messageId; System.out.printf("\n messageId=" + innerMessageId + "responseBody=" + response.getResponseBody()); return response; } @Override public void onThrowable(Throwable t) { t.printStackTrace(); } }); } thanks. 2014-12-15 19:30 GMT+02:00 Michael Rose <[email protected]<mailto:[email protected]>>: Keep in mind you'll need to synchronize the OutputCollector when your NIO response workers handle the returned requests as OutputCollector is not thread safe. Michael Rose (@Xorlev<https://twitter.com/xorlev>) Senior Platform Engineer, FullContact<http://www.fullcontact.com/> [email protected]<mailto:[email protected]> On Mon, Dec 15, 2014 at 9:20 AM, Itai Frenkel <[email protected]<mailto:[email protected]>> wrote: Hi, Any non-blocking bolt does not push back on the previous bolt if it is out of resources. So you should consider using max-spout-pending for spout level throttling. Regards, Itai ________________________________ From: Idan Fridman <[email protected]<mailto:[email protected]>> Sent: Monday, December 15, 2014 10:19 AM To: [email protected]<mailto:[email protected]> Subject: Using AsyncHttpReuqest inside a Bolt Hi All, My bolt need to dispatch async request to remote service. I am using AsyncHttpReuest library(https://github.com/AsyncHttpClient/async-http-client) which based on NIO channels to get the response asynchronously while not allocating Thread for each request. I was wondering if any side-effects could cause this implementation within Storm Bolt ? thank you.
