[ https://issues.apache.org/jira/browse/NIFI-10404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tamas Palfy resolved NIFI-10404. -------------------------------- Fix Version/s: 1.18.0 Resolution: Fixed > TailFile processor's persistent state not cleaned up > ---------------------------------------------------- > > Key: NIFI-10404 > URL: https://issues.apache.org/jira/browse/NIFI-10404 > Project: Apache NiFi > Issue Type: Bug > Reporter: Zsihovszki Krisztina > Assignee: Zsihovszki Krisztina > Priority: Major > Fix For: 1.18.0 > > Time Spent: 50m > Remaining Estimate: 0h > > TailFile processor's persistent state map is not maintained properly, entries > are added to this map but the references to non existent files are not moved > from the map. > At NiFi restart the states map (storing the TailFileObjects) is empty and it > is > [filled|https://github.com/hortonworks/nifi/blob/CFM-2.0.4.0/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java#L360] > with entries based on the persistent state map. > Each TailFileObject contains a byte buffer with the size of 65536 bytes. > Since the persistent state map is not cleaned up and it may contain thousands > of file references, a lot of TailFileObject may be created at startup, each > consuming 65536 bytes which can end up in out of memory error. > The persistent state map needs to be actualized based on the actual files as > it is done for the [states > map|https://github.com/hortonworks/nifi/blob/CFM-2.0.4.0/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/TailFile.java#L374]. > The allocated byte buffer size (65536 bytes) should be configurable to be > able to use smaller buffer size (default value would be 65536 bytes for > backward compatibility). > -- This message was sent by Atlassian Jira (v8.20.10#820010)