The S3Fetch processor is a stateful processor using NiFi state storage to track the time of the most recent object name list extracted from the bucket. On subsequent runs, it will only pull objects that have an update time newer than the internally stored time stored in state (the time of the last S3Fetch run).
Is this possibly contributing to your missing files? mark. On Fri, Sep 21, 2018 at 6:21 AM Vets, Laurens <laur...@daemon.be> wrote: > Hello, > > I'm using NiFi to read an S3 bucket containing all our AWS CloudTrail > logs. While debugging an issue, I noticed that not all objects are > fetched or listed. Basically, some events which I can find manually by > grepping the S3 files, I can't find in our Kibana dashboard. Is it > therefor possible that there might be an issue with the S3 processors > whereby it doesn't pick up all S3 objects? > > I'm using NiFi 1.3.0. While reading the release notes for the newer NiFi > versions, I found https://issues.apache.org/jira/browse/NIFI-4876 and > wondering whether this might be related? > > Can anyone shed some light on this? > >
