[
https://issues.apache.org/jira/browse/NIFI-7611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Handermann resolved NIFI-7611.
------------------------------------
Resolution: Cannot Reproduce
Thanks the comment [~Kappelmann], it makes sense that an anti-virus scanner
could introduce this behavior if it locks access to the Provenance repository
files. Closing based on those comments, but this could be re-opened if there
other environmental factors could be described and ruled out as the source of
the problem.
> NiFi fails to index provenance events
> -------------------------------------
>
> Key: NIFI-7611
> URL: https://issues.apache.org/jira/browse/NIFI-7611
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.11.4
> Environment: Microsoft Windows Server 2016 Standard - Intel Xeon Gold
> 6140 CPU @ 2,30 GHz 8 processors, 32 GB RAM, total disk space 877 GB
> Reporter: Michal W
> Priority: Critical
> Labels: index, java.nio.file.FileSystemException, provenance,
> repository
>
> Getting error "failed to index provenance events". Nifi.app log displays
> following information:
> 2020-07-08 09:00:00,406 ERROR [Index Provenance Events-4]
> o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events
> org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:681)
> at
> org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:695)
> at
> org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1281)
> at
> org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1257)
> at
> org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202)
> at
> org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.nio.file.FileSystemException:
> E:\nifi-storage\provenance_repository\lucene-8-index-1593163985970\_11r.cfe:
> The process cannot access the file because it is being used by another
> process.
>
> at
> java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92)
> at
> java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
> at
> java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
> at
> java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:120)
> at
> java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
> at
> java.base/java.nio.channels.FileChannel.open(FileChannel.java:345)
> at
> org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238)
> at
> org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157)
> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.readEntries(Lucene50CompoundReader.java:105)
> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundReader.<init>(Lucene50CompoundReader.java:69)
> at
> org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.getCompoundReader(Lucene50CompoundFormat.java:70)
> at
> org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:100)
> at
> org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:172)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:709)
> at
> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4396)
> at
> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4054)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625)
> at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662)
>
> Logs eventually grow over time and fill up the partition.
>
> Configuration related to provenance repository:
>
> # Provenance Repository Properties
> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
> nifi.provenance.repository.debug.frequency=1_000_000
> nifi.provenance.repository.encryption.key.provider.implementation=
> nifi.provenance.repository.encryption.key.provider.location=
> nifi.provenance.repository.encryption.key.id=
> nifi.provenance.repository.encryption.key=
>
> # Persistent Provenance Repository Properties
> nifi.provenance.repository.directory.default=E:\\nifi-storage\\provenance_repository
> nifi.provenance.repository.directory.content1=F:\\nifi-storage\\provenance_repository
> nifi.provenance.repository.max.storage.time=24 hours
> # nifi.provenance.repository.max.storage.size=1 GB
> nifi.provenance.repository.max.storage.size=8 GB
> nifi.provenance.repository.rollover.time=30 secs
> # nifi.provenance.repository.rollover.size=100 MB
> nifi.provenance.repository.rollover.size=1 GB
> nifi.provenance.repository.query.threads=2
> nifi.provenance.repository.index.threads=4
> #default: nifi.provenance.repository.compress.on.rollover=true
> nifi.provenance.repository.compress.on.rollover=false
> nifi.provenance.repository.always.sync=false
> # Comma-separated list of fields. Fields that are not indexed will not be
> searchable. Valid fields are:
> # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID,
> AlternateIdentifierURI, Relationship, Details
> nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
> ProcessorID, Relationship
> # FlowFile Attributes that should be indexed and made searchable. Some
> examples to consider are filename, uuid, mime.type
> nifi.provenance.repository.indexed.attributes=
> # Large values for the shard size will result in more Java heap usage when
> searching the Provenance Repository
> # but should provide better performance
> # nifi.provenance.repository.index.shard.size=500 MB
> nifi.provenance.repository.index.shard.size=4 GB
>
> # Indicates the maximum length that a FlowFile attribute can be when
> retrieving a Provenance Event from
> # the repository. If the length of any attribute exceeds this value, it will
> be truncated when the event is retrieved.
> nifi.provenance.repository.max.attribute.length=65536
> nifi.provenance.repository.concurrent.merge.threads=2
>
> # Volatile Provenance Respository Properties
> nifi.provenance.repository.buffer.size=100000
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
