Hey Guys, we have started to use the reporting task SiteToSiteProvenanceReportingTask 1.15.3 for auditing purposes on provenance events for a few of our dataflows. We started to see error quite frequently now in the Bulletin Board. ERROR: Failed to index Provenance Events. See logs for more information.
We also found this on the community https://issues.apache.org/jira/browse/NIFI-9572 that many people are having this issue and the root cause is still not resolved. Can you guys help us with any interim solutions which you might have found to solve this error? Also, below are snaps of nifi.provenance.properties file and from logs regarding this error. Please let me know if you need anything more. [cid:image001.png@01D8985C.E4FC41E0] 2022-07-15 02:17:47,227 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2022-07-15T07:17:47.202562Z, (lock=NativeFSLock(path=/apps/nifi/repos/provenance_repository/lucene-8-index-1657869467201/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2022-07-15T07:17:47.201562Z)) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter.<init>(CompressingStoredFieldsWriter.java:118) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsWriter(CompressingStoredFieldsFormat.java:130) at org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsWriter(Lucene87StoredFieldsFormat.java:141) at org.apache.lucene.index.StoredFieldsConsumer.initStoredFieldsWriter(StoredFieldsConsumer.java:48) at org.apache.lucene.index.StoredFieldsConsumer.startDocument(StoredFieldsConsumer.java:55) at org.apache.lucene.index.DefaultIndexingChain.startStoredFields(DefaultIndexingChain.java:449) at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:485) at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:208) at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:415) at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1471) ... 9 common frames omitted Thanks Deepak [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 [image005] Deepak Reddy | Data Engineer IT Centers of Excellence 13736 Riverport Dr., Maryland Heights, MO 63043 E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
