I don't know, hence my initial question. All I can say is that certain CloudTrail events in S3 don't seem to be picked up somehow...
On 20-Sep-18 19:55, Mark Rachelski wrote: > The S3Fetch processor is a stateful processor using NiFi state storage > to track the time of the most recent object name list extracted from > the bucket. On subsequent runs, it will only pull objects that have an > update time newer than the internally stored time stored in state (the > time of the last S3Fetch run). > > Is this possibly contributing to your missing files? > > mark. > > On Fri, Sep 21, 2018 at 6:21 AM Vets, Laurens <laur...@daemon.be > <mailto:laur...@daemon.be>> wrote: > > Hello, > > I'm using NiFi to read an S3 bucket containing all our AWS CloudTrail > logs. While debugging an issue, I noticed that not all objects are > fetched or listed. Basically, some events which I can find manually by > grepping the S3 files, I can't find in our Kibana dashboard. Is it > therefor possible that there might be an issue with the S3 processors > whereby it doesn't pick up all S3 objects? > > I'm using NiFi 1.3.0. While reading the release notes for the > newer NiFi > versions, I found https://issues.apache.org/jira/browse/NIFI-4876 and > wondering whether this might be related? > > Can anyone shed some light on this? >
