I'm doing both tridentKafkaConfig.forceFromStart = false; as well as tridentKafkaConfig.startOffsetTime = -1;
Neither are working for me. Looking at my nimbus UI, I still get a large spike in processed data, before it levels off and seems to not process anything. On Wed, May 28, 2014 at 3:17 PM, Shaikh Riyaz <shaikh....@gmail.com> wrote: > I think you can use kafkaConfig.forceFromStart = *false*; > > We have implemented this and its working fine. > > Regards, > Riyaz > > > > On Thu, May 29, 2014 at 1:02 AM, Raphael Hsieh <raffihs...@gmail.com>wrote: > >> This is still not working for me. I've set the offset to -1 and it is >> still backfilling data. >> Is there any documentation on the start offsets that I could take a look >> at ? >> Or even documentation on kafka.api.OffsetRequest.LatestTime() ? >> >> >> On Wed, May 28, 2014 at 1:01 PM, Raphael Hsieh <raffihs...@gmail.com>wrote: >> >>> would the Trident version of this be >>> tridentKafkaConfig.startOffsetTime ? >>> >>> >>> On Wed, May 28, 2014 at 12:23 PM, Danijel Schiavuzzi < >>> dani...@schiavuzzi.com> wrote: >>> >>>> By default, the Kafka spout resumes consuming where it last left off. >>>> That offset is stored in ZooKeeper. >>>> >>>> You can set forceStartOffset to -2 to start consuming from the earliest >>>> available offset, or -1 to start consuming from the latest available >>>> offset. >>>> >>>> >>>> On Wednesday, May 28, 2014, Raphael Hsieh <raffihs...@gmail.com> wrote: >>>> >>>>> If I don't tell trident to start consuming data from the beginning of >>>>> the Kafka stream, where does it start from? >>>>> If I were to do: >>>>> tridentKafkaConfig.forceFromStart = true; >>>>> Then it will tell the spout to start consuming from the start of the >>>>> stream. If that is not set, then where does it start consuming from? and >>>>> How might I go about telling it to start consuming from the very end of >>>>> the >>>>> stream? >>>>> >>>>> If a disaster were to happen and all my hosts died, when I start my >>>>> cluster back up, it might start consuming from where it left off. I would >>>>> rather manually process that old data, and have my storm system start >>>>> processing the live data. >>>>> >>>>> Thanks >>>>> -- >>>>> Raphael Hsieh >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> Danijel Schiavuzzi >>>> >>>> E: dani...@schiavuzzi.com >>>> W: www.schiavuzzi.com >>>> T: +385989035562 >>>> Skype: danijels7 >>>> >>> >>> >>> >>> -- >>> Raphael Hsieh >>> Amazon.com >>> Software Development Engineer I >>> (978) 764-9014 >>> >>> >>> >>> >> >> >> >> -- >> Raphael Hsieh >> Amazon.com >> Software Development Engineer I >> (978) 764-9014 >> >> >> >> > > > > -- > Regards, > > Riyaz > > -- Raphael Hsieh
