Hi all,
I have one NiFi instance running inside a Docker container and managed by
Kubernetes.
To maintain the application state, the NiFi repositories are under the path
/usr/nifi/data which is a GlusterFS mount.
After sometimes, the lucene index of the provenance repository got corrupted:
2018-01-03 12:05:07,879 INFO [Provenance Repository Rollover Thread-2]
SimpleIndexManager.java:309 Index Writer for
./data/provenance_repository/index-1512127508000 has been returned to Index
Manager and is no longer in use. Closing Index Writer
2018-01-03 12:05:15,669 INFO [Provenance Repository Rollover Thread-2]
PersistentProvenanceRepository.java:1903 Successfully merged 16 journal files
(5 records) into single Provenance Log File
./data/provenance_repository/492.prov in 18387 milliseconds
2018-01-03 12:05:15,754 INFO [Provenance Repository Rollover Thread-2]
PersistentProvenanceRepository.java:1324 Successfully Rolled over Provenance
Event file containing 29 records. In the past 5 minutes, 33 events have been
written to the Provenance Repository, totaling 178.78 KB
2018-01-03 12:05:30,124 INFO [Provenance Maintenance Thread-3]
PersistentProvenanceRepository.java:349 Created new Provenance Event Writers
for events starting with ID 499
2018-01-03 12:05:30,128 INFO [Provenance Maintenance Thread-3]
KyloPersistentProvenanceEventRepository.java:78 Created 16
KyloRecordWriterDelegate objects starting with: 499
2018-01-03 12:05:33,494 ERROR [Lucene Merge Thread #0] NiFi.java:83 An Unknown
Error Occurred in Thread Thread[Lucene Merge Thread #0,6,main]:
org.apache.lucene.index.MergePolicy$MergeException: java.io.EOFException: read
past EOF:
MMapIndexInput(path="/usr/nifi/data/provenance_repository/index-1512127508000/_ii.cfs")
[slice=_ii.fdx]
2018-01-03 12:05:33,523 ERROR [Lucene Merge Thread #0] NiFi.java:84
org.apache.lucene.index.MergePolicy$MergeException: java.io.EOFException: read
past EOF:
MMapIndexInput(path="/usr/nifi/data/provenance_repository/index-1512127508000/_ii.cfs")
[slice=_ii.fdx]
at
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:549)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:522)
Caused by: java.io.EOFException: read past EOF:
MMapIndexInput(path="/usr/nifi/data/provenance_repository/index-1512127508000/_ii.cfs")
[slice=_ii.fdx]
at
org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81)
at
org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:98)
at org.apache.lucene.store.DataInput.readLong(DataInput.java:154)
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:209)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:124)
at
org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113)
at
org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:133)
at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:108)
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145)
at
org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:664)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4152)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3811)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486)
Has anyboy any idea how could I debug this problem?
I found these issues that could be related:
https://bugzilla.redhat.com/show_bug.cgi?id=1379568
https://bugzilla.redhat.com/show_bug.cgi?id=1390050
Thanks,
Fabian
