[ https://issues.apache.org/jira/browse/NIFI-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bryan Bende updated NIFI-4069: ------------------------------ Fix Version/s: 1.4.0 > ListXXX processors can miss files those created while the processor is > listing and filesystem does not provide timestamp milliseconds precision > ----------------------------------------------------------------------------------------------------------------------------------------------- > > Key: NIFI-4069 > URL: https://issues.apache.org/jira/browse/NIFI-4069 > Project: Apache NiFi > Issue Type: Bug > Components: Extensions > Affects Versions: 1.0.0 > Reporter: Koji Kawamura > Assignee: Koji Kawamura > Fix For: 1.4.0 > > Attachments: ListFilesWithoutMilliseconds.png > > > For some filesystems such as Mac OS X HFS (Hierarchical File System) or EXT3 > are known that only support timestamp in seconds precision. Also some FTP > server is reported that it can only provides timestamp precision in minutes. > This can cause files to NOT be listed as ListXXX processors logic expects > timestamps in milliseconds. > Specifically, if generate several files in one second, not all files will be > listened. > Steps to reproduce: > 1. start processor ListFile > 2. generate 10000 zero size files with following command: > {code} > for i in {1..10000}; do touch ./test_$i; done > {code} > 3. see processor stats: out 3952 (0 bytes) > Current AbstractListProcessor logic adopts LISTING_LAG_NANOS (100ms) and > postponing the files those have the latest timestamp within a listing > iteration to next iteration, however with those filesystem without > milliseconds precision, these logics do not work as expected. > This issue is originally reported at nifi-dev ML. > http://apache-nifi-developer-list.39713.n7.nabble.com/processors-ListFile-ListSFTP-do-not-store-milliseconds-in-timestamp-td16037.html -- This message was sent by Atlassian JIRA (v6.4.14#64029)