Hello Nifi , Greetings of the day!! Please help us to rectify the below provenance error we are facing in our nifi Production UI..
We have recently upgraded from Nifi 1.9.2 version to Nifi version 1.16.2 and we are getting the "Failed to index Provenance Events" Error, not all the time but it occurs occasionally. We have not faced such kind of error in nifi version 1.9.2. Can anyone please help us to figure out the possible cause of the occurrence of that error. Though it was not impacting the flow but still its occurrence is random. Please find the details below: Steps i followed to resolve this issue but no luck: 1: I have re-started nifi services with an empty directory for both flowfile and provenance repository. and placed both repository in seperate directory. Please note I have not deleted the content of older provinance repository directory that we had for 1.9.2. Please find the directory path below For ex: /repository/nifi/nifi-1.16.2/provenanceRepository /repository/nifi/nifi-1.16.2/Flowfilerepository 2: Java version used by nifi: openjdk version "1.8.0_332" OpenJDK Runtime Environment (build 1.8.0_332-b09) OpenJDK 64-Bit Server VM (build 25.332-b09, mixed mode) 3: No other error other then the one which i am sharing in the app log below Logs: 2023-01-04 06:21:23,713 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2022-12-14T11:18:50Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository/lucene-8-index-1671016730507/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2022-12-14T11:18:50.521826Z)) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.<init>(Lucene80DocValuesConsumer.java:79) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.fieldsConsumer(Lucene80DocValuesFormat.java:161) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:227) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:163) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109) at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:108) at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:345) at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:225) at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350) at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3365) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3771) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3729) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.commit(LuceneEventIndexWriter.java:101) at org.apache.nifi.provenance.index.lucene.EventIndexTask.commit(EventIndexTask.java:253) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:232) ... 6 common frames omitted *Please consider it as PRIORITY and do reply as this has been causing issue in production from very long time. We are unable to find any solution from nifi community as well. please feel free to forward it to anyone who can help in this regard..* Thank you, Gaurav Pandey
