Hi, I'm a bit stuck here and needs a clue or two in order to continue our investigations. Hope that someone can help. :)
Periodically, around once a month, we get the below ArrayIndexOutOfBoundsException on our system. We use multiple indexes and the error can originate from any of them, but the error always occurs in line 1217 in BKDWriter (during a System.arraycopy). We found a couple of issues on the net regarding JIT optimization problem related to J9, but they all looks like they have been resolved and cannot be reproduced anymore. But nevertheless, we have just added the -Xint flag (disable JIT compiler) in order to see whether this has any impact. Unfortunately we do not have the result of this yet, but I'll of course post it when it is known. Are there any of you clever guys out there, that has some good ideas further investigations? Or have seen such issue before? We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform. Java version java version "1.8.0_191" Java(TM) SE Runtime Environment (build 8.0.5.25 - pap6480sr5fp25-20181030_01(SR5 FP25)) IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References 20181029_400846 (JIT enabled, AOT enabled) OpenJ9 - c5c78da OMR - 3d5ac33 IBM - 8c1bdc2) JCL - 20181022_01 based on Oracle jdk8u191-b26 NOTICE: If no version information is found above, this could indicate a corrupted Java installation! Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java -Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m Stacktrace Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] java.lang.ArrayIndexOutOfBoundsException at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217) at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197) at org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078) at org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245) at org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82) at org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648) at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560) at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212) at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:08.075 CEST [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: java.lang.ArrayIndexOutOfBoundsException at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683) Caused by: 2019-05-01T06:10:08.076 CEST [Lucene Merge Thread #0] java.lang.ArrayIndexOutOfBoundsException at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217) at org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197) at org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078) at org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245) at org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82) at org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648) at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560) at org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212) at org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler BKDWriter.java [cid:image003.png@01D5002C.7B1A9B80] Kind regards Torben Riis ------------------------- Torben Riis Lead Architect [cid:image001.png@01D50032.347008E0] Multi Support * Making good business run better +45 96 600 600 * www.multi-support.com