Hi Martin, This was addressed in 1.13 by NIFI-7856 [1].
Thanks -Mark [1] https://issues.apache.org/jira/browse/NIFI-7856 On May 14, 2021, at 6:02 AM, Hengesbach, Martin <[email protected]<mailto:[email protected]>> wrote: Hi All, I have a problem with the data provenance repository. In the log file there are entries that the file is not present during rollover. If I look in the filesystem, the file is already gzipped and has a timestamp 10 minutes earlier (rollover time is 10 mins). Error message in nifi-app.log: <image004.png> Excerpt from the directory listing, file exists since 10 minutes earlier: <image005.png> This happens round about 20 times a day. And sometimes I’ve the problem that entries are missing when I search the Data provenance. My configuration: • NiFi version: 1.12.1 • OS: Linux • The files for the data provenance are on a mounted NetApp volume (NFS) – could this be a problem? • The relevant config from nifi.properties is as following: # Provenance Repository Properties nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository nifi.provenance.repository.encryption.key.provider.implementation= nifi.provenance.repository.encryption.key.provider.location= nifi.provenance.repository.encryption.key.id= nifi.provenance.repository.encryption.key= # Persistent Provenance Repository Properties nifi.provenance.repository.directory.default=/data/nifi-data/provenance_repository nifi.provenance.repository.max.storage.time=30 days nifi.provenance.repository.max.storage.size=30 GB nifi.provenance.repository.rollover.time=10 mins nifi.provenance.repository.rollover.size=100 MB nifi.provenance.repository.query.threads=2 nifi.provenance.repository.index.threads=2 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.always.sync=false # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are: # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship # FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type nifi.provenance.repository.indexed.attributes=Project,Database,priority,Update.Identifier,batchId,UniventioRequestID,retry # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository # but should provide better performance nifi.provenance.repository.index.shard.size=500 MB # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved. nifi.provenance.repository.max.attribute.length=65536 nifi.provenance.repository.concurrent.merge.threads=2 # Volatile Provenance Respository Properties nifi.provenance.repository.buffer.size=100000 Thanks in advance and best regards Martin ________________________________ FIZ Karlsruhe - Leibniz-Institut für Informationsinfrastruktur GmbH. Sitz der Gesellschaft: Eggenstein-Leopoldshafen, Amtsgericht Mannheim HRB 101892. Geschäftsführerin: Sabine Brünger-Weilandt. Vorsitzender des Aufsichtsrats: MinR Dr. Hans-Josef Linkens. FIZ Karlsruhe ist zertifiziert mit dem Siegel "audit berufundfamilie".
