If you wish to consume the latest CloudTrail events as a stream, have you considered subscribing to CloudTrail notifications via SNS and SQS? NiFi can read the SQS messages with the GetSQS processor, and extract the S3 key with EvaluateJsonPath. In contrast, I expect ListS3 would be more useful for reprocessing historical events.
Configuring CloudTrail to Send Notifications https://docs.aws.amazon.com/awscloudtrail/latest/userguide/configure-cloudtrail-to-send-notifications.html On Fri, Sep 21, 2018 at 11:55 PM Vets, Laurens <laur...@daemon.be> wrote: > Is there a way to troubleshoot this further? I did an upgrade to 1.7.1. > Changed the 'Minimum Object Age' to 1 minute, but I _think_ NiFi is > somehow still not picking up all objects from the S3 bucket. > > On 20-Sep-18 19:55, Mark Rachelski wrote: > > The S3Fetch processor is a stateful processor using NiFi state storage > > to track the time of the most recent object name list extracted from > > the bucket. On subsequent runs, it will only pull objects that have an > > update time newer than the internally stored time stored in state (the > > time of the last S3Fetch run). > > > > Is this possibly contributing to your missing files? > > > > mark. > > > > On Fri, Sep 21, 2018 at 6:21 AM Vets, Laurens <laur...@daemon.be > > <mailto:laur...@daemon.be>> wrote: > > > > Hello, > > > > I'm using NiFi to read an S3 bucket containing all our AWS CloudTrail > > logs. While debugging an issue, I noticed that not all objects are > > fetched or listed. Basically, some events which I can find manually > by > > grepping the S3 files, I can't find in our Kibana dashboard. Is it > > therefor possible that there might be an issue with the S3 processors > > whereby it doesn't pick up all S3 objects? > > > > I'm using NiFi 1.3.0. While reading the release notes for the > > newer NiFi > > versions, I found https://issues.apache.org/jira/browse/NIFI-4876 > and > > wondering whether this might be related? > > > > Can anyone shed some light on this? > > >
