Munagala V. Ramanath created APEXCORE-343:
---------------------------------------------
Summary: Add property to AbstractFileInputOperator to trim
processedFiles and ignoredFiles
Key: APEXCORE-343
URL: https://issues.apache.org/jira/browse/APEXCORE-343
Project: Apache Apex Core
Issue Type: Improvement
Affects Versions: 3.3.0
Reporter: Munagala V. Ramanath
Assignee: Munagala V. Ramanath
In AbstractFileInputOperator, the processedFiles and
DirectoryScanner.ignoredFiles sets can continue to grow without bound as new
files are added to the monitored directory. There are scenarios where an input
file is deleted by the application once it is processed; here, it is useful to
provide a property that, if true, will cause the scanner to remove deleted
files from these sets to prevent unbounded growth.
The only way to do this currently is to extend DirectoryScanner and provide
implementations of the scan() and createPartition() methods which forces the
operator writer to unnecessarily grapple with the internals of the base class.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
