Why not just ack the tuple once it's been written to a file. If your topology fails then the data will be re-read from Kafka. Kafka spout already does this for you. Then uploading files to S3 is the responsibility of another job. For example, a storm topology that monitors the output folder.
Monitoring the data from Kafka all the way out to S3 seems unnecessary. On Wed, May 11, 2016 at 1:50 PM, Milind Vaidya <kava...@gmail.com> wrote: > It does not matter, in the sense I am ready to upgrade if this thing is in > the roadmap. > > None the less > > kafka_2.9.2-0.8.1.1 apache-storm-0.9.4 > > > > > On Wed, May 11, 2016 at 5:53 AM, Abhishek Agarwal <abhishc...@gmail.com> > wrote: > >> which version of storm-kafka, are you using? >> >> On Wed, May 11, 2016 at 12:29 AM, Milind Vaidya <kava...@gmail.com> >> wrote: >> >>> Anybody ? Anything about this ? >>> >>> On Wed, May 4, 2016 at 11:31 AM, Milind Vaidya <kava...@gmail.com> >>> wrote: >>> >>>> Is there any way I can know what Kafka offset corresponds to current >>>> tuple I am processing in a bolt ? >>>> >>>> Use case : Need to batch events from Kafka, persists them to a local >>>> file and eventually upload it to the S3. To manager failure cases, need to >>>> know the Kafka offset for a message, so that it can be persisted to >>>> Zookeeper and will be used to write / upload file. >>>> >>>> >>> >> >> >> -- >> Regards, >> Abhishek Agarwal >> >> >
