Hello, The GetSplunk processor has 3 time range strategies...
- Managed from beginning - Managed from current - Provided The "managed" strategies store the last successful time range in processor state, so when splunk is down it should keep trying to query Splunk and failing and never updating the state, so each next execution would still have the start time as the end of the last success, and therefore would capture the whole range that Splunk was down. It looks like you may be using the "provided" strategy since you are specifying values for Earliest and Latest. The provided strategy was meant to just query exactly what is provided, so it does not store state or create incremental time ranges automatically, which means you won't get the backfill in this case. Thanks, Bryan On Wed, Jan 12, 2022 at 4:53 PM Chirthani, Deepak Reddy <c-deepakreddy.chirth...@charter.com> wrote: > > Hey Mark, > > Good to see you. I'm specifically talking about GetSplunk Processor which we > use to pull data previous 15 minutes data(earliest time: -15m@m and latest > time: now()) from the Splunk Enterprise every 15 mins. The processor > absolutely works fine. But it is Splunk with which we have issues of getting > down so that we miss data when it is down. So trying to find a way of > backfilling the data. Does this makes sense? Let me know if you have further > questions. > > > > On 2022/01/11 22:36:39 "Chirthani, Deepak Reddy" wrote: > > > Hello, > > > > > > Is Nifi good at Backfilling the data? Let's say that I have a dataflow > > which is cron scheduled to run every 15 mins. Lets say that the source was > > down and the dataflow did not pulled any data for a couple of runs and > > after some time the source is up and running. If this happens frequently, > > how do you handle backfilling in Nifi? Please let me know if there is any > > automated way of doing this? > > > > > > Thanks > > > E-MAIL CONFIDENTIALITY NOTICE: > > > The contents of this e-mail message and any attachments are intended solely > > for the addressee(s) and may contain confidential and/or legally privileged > > information. If you are not the intended recipient of this message or if > > this message has been addressed to you in error, please immediately alert > > the sender by reply e-mail and then delete this message and any > > attachments. If you are not the intended recipient, you are notified that > > any use, dissemination, distribution, copying, or storage of this message > > or any attachment is strictly prohibited. > > > > E-MAIL CONFIDENTIALITY NOTICE: > The contents of this e-mail message and any attachments are intended solely > for the addressee(s) and may contain confidential and/or legally privileged > information. If you are not the intended recipient of this message or if this > message has been addressed to you in error, please immediately alert the > sender by reply e-mail and then delete this message and any attachments. If > you are not the intended recipient, you are notified that any use, > dissemination, distribution, copying, or storage of this message or any > attachment is strictly prohibited.
