Will the HTTP event sink respond with some acknowledgement that it
received whatever was sent?
If so, could this be as simple as telling your bolt not to ack the tuple
until this response is received from the HTTP service?
--
Derek
On 9/26/14 10:10, Peter Neumark wrote:
Thanks for the quick response!
Unfortunately, we're forced to use HTTP.
Any ideas?
On Fri, Sep 26, 2014 at 5:07 PM, Supun Kamburugamuva <supu...@gmail.com>
wrote:
On Fri, Sep 26, 2014 at 10:49 AM, Peter Neumark <peter.neum...@prezi.com>
wrote:
Hi all,
We want to replace a legacy custom app with storm, but -being storm
newbies- we're not sure what's the best way to solve the following problem:
An HTTP endpoint returns the list of events which occurred between two
timestamps. The task is to continuously poll this event source for new
events, optionally perform some transformation and aggregation operations
on them, and finally make an HTTP request to an endpoint with some events.
We thought of a simple topology:
1. A clock-spout determines which time interval to process.
2. A bolt takes the time interval as input, and fetches the event list
for that interval fro the event source, emitting them as individual tuples.
3. After some processing of the tuples, we aggregate them into fixed size
groups, which we send in HTTP requests to an event sink.
The big question is how to make sure that all events are successfully
delivered to the event sink. I know storm guarantees the delivery of tuples
within the topology, but how could I guarantee that the HTTP requests to
the event sink are also successful (and retried if necessary).
I think this is not a question about Storm and rather a question about how
to deliver a message reliably to some sink. From my experience it is bit
hard to achieve something like this with HTTP. This functionality is built
in to message brokers like RabbitMQ, ActiveMQ, Kafka etc and if you use a
broker to send your events to the sink you can get a delivery guarantee.
Thanks,
Supun..
All help, suggestions and pointers welcome!
Peter
--
*Peter Neumark*
DevOps guy @Prezi <http://prezi.com>
--
Supun Kamburugamuva
Member, Apache Software Foundation; http://www.apache.org
E-mail: supu...@gmail.com; Mobile: +1 812 369 6762
Blog: http://supunk.blogspot.com
