Michael W Moser created NIFI-16416:
--------------------------------------
Summary: Optimize WriteAheadProvenanceRepository storage
maintenance and bug fixes
Key: NIFI-16416
URL: https://issues.apache.org/jira/browse/NIFI-16416
Project: Apache NiFi
Issue Type: Improvement
Components: Core Framework
Reporter: Michael W Moser
Assignee: Michael W Moser
h3. Background
WriteAheadProvenanceRepository has a background maintenance task that checks if
the max.storage.time and max.storage.size is exceeded and cleanup needs to
happen. The maintenance task performs many provenance repo directory listings,
including a new listing after each file is deleted during max.storage.size
checks. This process repeats until enough files are deleted to bring the
provenance repository within storage limits. (Task 1) This could be optimized.
h3. Bug Reports
Users have also reported various other issues with provenance repo maintenance,
including:
* (Task 2) an orphaned events file with a large event ID "3437277653.prov"
could never get cleaned up because files with lower event IDs "572531.prov" are
cleaned up first
* (Task 3) junk files in the prov_repo directory count toward the max storage
size limit, which can cause the maintenance task to get stuck in an infinite
loop if 0 files can be deleted
* (Task 4) .toc files can be removed if the corresponding .prov/.prov.gz file
could not be removed
h3. Goals
* (Task 1) We can improve the provenance repo maintenance by performing only
one directory listing, then reusing that list throughout the maintenance task.
* (Task 2) Sort event files by lastModified instead of event ID in filename
* (Task 3) Avoid infinite loop when prov_repo size exceeds max.storage.size,
but event files cannot be deleted
* (Task 4) Properly handle removing .toc files only if its .prov file is
removed
h3. Stretch Goal
* Remove events files when disk partition usage exceeds 98% (NiFi does this in
FileSystemRepository when content_repo archiving is on, removing archived
data). This could help avoid filling the disk partition where prov_repo stores
data.
h3. Other Observations
* When using multiple provenance repo partitions, the cleanup task deletes one
file from each prov_repo partition per pass. This can delete files that are
not the oldest events overall. This is a minor issue, though, because deleted
events are still very close in time to older events in other partitions.
This was discussed in Slack in May 2026 and August 2026. Feedback on proposed
goals appreciated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
