I think it is possible to upload a file to S3 with original last modified timestamp. If this timestamp less than a timestamp of previously found file (so the state will be greater), the the file won't be picked up.
On Fri, Sep 21, 2018, 7:34 PM Joe Witt <joe.w...@gmail.com> wrote: > hello > > i think we would need to see directory listing from s3, state info and > config from nifi, and logs. > > you can step through in a debug session to see what the proc sees as well. > > thanks > > On Fri, Sep 21, 2018, 6:55 PM Vets, Laurens <laur...@daemon.be> wrote: > >> Is there a way to troubleshoot this further? I did an upgrade to 1.7.1. >> Changed the 'Minimum Object Age' to 1 minute, but I _think_ NiFi is >> somehow still not picking up all objects from the S3 bucket. >> >> On 20-Sep-18 19:55, Mark Rachelski wrote: >> > The S3Fetch processor is a stateful processor using NiFi state storage >> > to track the time of the most recent object name list extracted from >> > the bucket. On subsequent runs, it will only pull objects that have an >> > update time newer than the internally stored time stored in state (the >> > time of the last S3Fetch run). >> > >> > Is this possibly contributing to your missing files? >> > >> > mark. >> > >> > On Fri, Sep 21, 2018 at 6:21 AM Vets, Laurens <laur...@daemon.be >> > <mailto:laur...@daemon.be>> wrote: >> > >> > Hello, >> > >> > I'm using NiFi to read an S3 bucket containing all our AWS >> CloudTrail >> > logs. While debugging an issue, I noticed that not all objects are >> > fetched or listed. Basically, some events which I can find manually >> by >> > grepping the S3 files, I can't find in our Kibana dashboard. Is it >> > therefor possible that there might be an issue with the S3 >> processors >> > whereby it doesn't pick up all S3 objects? >> > >> > I'm using NiFi 1.3.0. While reading the release notes for the >> > newer NiFi >> > versions, I found https://issues.apache.org/jira/browse/NIFI-4876 >> and >> > wondering whether this might be related? >> > >> > Can anyone shed some light on this? >> > >> >
