[jira] [Issue Comment Deleted] (LUCENE-8795) ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter

2019-05-13 Thread Torben Riis (JIRA) 


 [ 
https://issues.apache.org/jira/browse/LUCENE-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Torben Riis updated LUCENE-8795:

Comment: was deleted

(was: I will be out of office until May 14th 2019.

I may or may not get a chance to read mails during my absence.
If you cannot wait for a reply until my return, please send your request to 
helpd...@multi-support.com for a more prompt 
response.


Kind regards

Torben Riis
--
—
Torben Riis
Lead Architect
Multi-Support · Making good business run better
+45 96 600 600 · www.multi-support.com
)

> ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter
> ---
>
> Key: LUCENE-8795
> URL: https://issues.apache.org/jira/browse/LUCENE-8795
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/index
>Affects Versions: 6.6
> Environment: h2. Operating system/Platform
> {code:java}
> IBM i - V7R3M0
> {code}
> h2. Java version
> {code:java}
> java version "1.8.0_191"
> Java(TM) SE Runtime Environment (build 8.0.5.25 - 
> pap6480sr5fp25-20181030_01(SR5 FP25))
> IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References 
> 20181029_400846 (JIT enabled, AOT enabled)
> OpenJ9   - c5c78da
> OMR  - 3d5ac33
> IBM  - 8c1bdc2)
> JCL - 20181022_01 based on Oracle jdk8u191-b26
> NOTICE: If no version information is found above, this could indicate a 
> corrupted Java installation!
> Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java
> -Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m
> {code}
>Reporter: Torben Riis
>Priority: Major
>
> Hi,
> I’m a bit stuck here and needs a clue or two in order to continue our 
> investigations. Hope that someone can help. :)
> Periodically, around once a month, we get the below 
> {{ArrayIndexOutOfBoundsException}} on our system. We use multiple indexes and 
> the error can originate from any of them, but the error always occurs in line 
> 1217 in {{BKDWriter}} (during a {{System.arraycopy}}).
> We found a couple of issues on the net regarding JIT optimization problem 
> related to J9, but they all looks like they have been resolved and cannot be 
> reproduced anymore. But nevertheless, we have added the following {{-Xjit}} 
> flags to excludes JIT optimization for every class / inner classes in the 
> {{bkd}} package. Moreover, we have also made a complete copy of the whole 
> installation in production, and added the opposite arguments (enforce JIT 
> optimizations for the specific classes). First we will try with 
> {{optLevel=hot}}, but if this doesn’t show anything we will afterwards try 
> with {{veryHot}} and {{scorching}}. Unfortunately we do not have the result 
> of this yet, but I’ll of course post it when it is known.
> Unfortunately it’s not possible to run OpenJDK on the IBM i platform, so such 
> a test will not be possible, but it is worth mentioning that our product is a 
> standard product, which typically run on the Windows or Linux platform using 
> AdoptOpenJDK. Currently we have a couple of hundred installations running out 
> there on these platforms, and without any problems. But on the IBM I platform 
> with J9 we sometimes see this exception.
> Any good ideas for further investigations? Or have seen such issue before?
> We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform.
> h2.  Stacktrace
> {code:java}
> Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST 
> [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: 
> java.lang.ArrayIndexOutOfBoundsException
> at 
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
> at 
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
> Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] 
> java.lang.ArrayIndexOutOfBoundsException
> at 
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
> at 
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
> at 
> org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
> at 
> org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
> at 
> org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
> at 
> org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
> at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
> at 
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
> at 
>

[jira] [Commented] (LUCENE-8795) ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter

2019-05-12 Thread Torben Riis (JIRA) 


[ 
https://issues.apache.org/jira/browse/LUCENE-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16838133#comment-16838133
 ] 

Torben Riis commented on LUCENE-8795:
-

I will be out of office until May 14th 2019.

I may or may not get a chance to read mails during my absence.
If you cannot wait for a reply until my return, please send your request to 
helpd...@multi-support.com for a more prompt 
response.


Kind regards

Torben Riis
--
—
Torben Riis
Lead Architect
Multi-Support · Making good business run better
+45 96 600 600 · www.multi-support.com


> ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter
> ---
>
> Key: LUCENE-8795
> URL: https://issues.apache.org/jira/browse/LUCENE-8795
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/index
>Affects Versions: 6.6
> Environment: h2. Operating system/Platform
> {code:java}
> IBM i - V7R3M0
> {code}
> h2. Java version
> {code:java}
> java version "1.8.0_191"
> Java(TM) SE Runtime Environment (build 8.0.5.25 - 
> pap6480sr5fp25-20181030_01(SR5 FP25))
> IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References 
> 20181029_400846 (JIT enabled, AOT enabled)
> OpenJ9   - c5c78da
> OMR  - 3d5ac33
> IBM  - 8c1bdc2)
> JCL - 20181022_01 based on Oracle jdk8u191-b26
> NOTICE: If no version information is found above, this could indicate a 
> corrupted Java installation!
> Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java
> -Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m
> {code}
>Reporter: Torben Riis
>Priority: Major
>
> Hi,
> I’m a bit stuck here and needs a clue or two in order to continue our 
> investigations. Hope that someone can help. :)
> Periodically, around once a month, we get the below 
> {{ArrayIndexOutOfBoundsException}} on our system. We use multiple indexes and 
> the error can originate from any of them, but the error always occurs in line 
> 1217 in {{BKDWriter}} (during a {{System.arraycopy}}).
> We found a couple of issues on the net regarding JIT optimization problem 
> related to J9, but they all looks like they have been resolved and cannot be 
> reproduced anymore. But nevertheless, we have added the following {{-Xjit}} 
> flags to excludes JIT optimization for every class / inner classes in the 
> {{bkd}} package. Moreover, we have also made a complete copy of the whole 
> installation in production, and added the opposite arguments (enforce JIT 
> optimizations for the specific classes). First we will try with 
> {{optLevel=hot}}, but if this doesn’t show anything we will afterwards try 
> with {{veryHot}} and {{scorching}}. Unfortunately we do not have the result 
> of this yet, but I’ll of course post it when it is known.
> Unfortunately it’s not possible to run OpenJDK on the IBM i platform, so such 
> a test will not be possible, but it is worth mentioning that our product is a 
> standard product, which typically run on the Windows or Linux platform using 
> AdoptOpenJDK. Currently we have a couple of hundred installations running out 
> there on these platforms, and without any problems. But on the IBM I platform 
> with J9 we sometimes see this exception.
> Any good ideas for further investigations? Or have seen such issue before?
> We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform.
> h2.  Stacktrace
> {code:java}
> Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST 
> [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: 
> java.lang.ArrayIndexOutOfBoundsException
> at 
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
> at 
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
> Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] 
> java.lang.ArrayIndexOutOfBoundsException
> at 
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
> at 
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
> at 
> org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
> at 
> org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
> at 
> org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
> at 
> org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
> at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
> at 
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
> at 
>

[jira] [Commented] (LUCENE-8795) ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter

2019-05-10 Thread Torben Riis (JIRA) 


[ 
https://issues.apache.org/jira/browse/LUCENE-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837122#comment-16837122
 ] 

Torben Riis commented on LUCENE-8795:
-

Hi you are probably right regarding whether this is a JVM bug or not. We will 
report our findings to IBM (we have a close relationship to them) and hopefully 
this will be fixed soon.

Are there any test classes that you could recommend that we use for provoking 
this issue. I would be nice if we somehow could reproduce this error a bit 
easier. Of cause combined with the -Xjit arguments.

> ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter
> ---
>
> Key: LUCENE-8795
> URL: https://issues.apache.org/jira/browse/LUCENE-8795
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/index
>Affects Versions: 6.6
> Environment: h2. Operating system/Platform
> {code:java}
> IBM i - V7R3M0
> {code}
> h2. Java version
> {code:java}
> java version "1.8.0_191"
> Java(TM) SE Runtime Environment (build 8.0.5.25 - 
> pap6480sr5fp25-20181030_01(SR5 FP25))
> IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References 
> 20181029_400846 (JIT enabled, AOT enabled)
> OpenJ9   - c5c78da
> OMR  - 3d5ac33
> IBM  - 8c1bdc2)
> JCL - 20181022_01 based on Oracle jdk8u191-b26
> NOTICE: If no version information is found above, this could indicate a 
> corrupted Java installation!
> Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java
> -Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m
> {code}
>Reporter: Torben Riis
>Priority: Major
>
> Hi,
> I’m a bit stuck here and needs a clue or two in order to continue our 
> investigations. Hope that someone can help. :)
> Periodically, around once a month, we get the below 
> {{ArrayIndexOutOfBoundsException}} on our system. We use multiple indexes and 
> the error can originate from any of them, but the error always occurs in line 
> 1217 in {{BKDWriter}} (during a {{System.arraycopy}}).
> We found a couple of issues on the net regarding JIT optimization problem 
> related to J9, but they all looks like they have been resolved and cannot be 
> reproduced anymore. But nevertheless, we have added the following {{-Xjit}} 
> flags to excludes JIT optimization for every class / inner classes in the 
> {{bkd}} package. Moreover, we have also made a complete copy of the whole 
> installation in production, and added the opposite arguments (enforce JIT 
> optimizations for the specific classes). First we will try with 
> {{optLevel=hot}}, but if this doesn’t show anything we will afterwards try 
> with {{veryHot}} and {{scorching}}. Unfortunately we do not have the result 
> of this yet, but I’ll of course post it when it is known.
> Unfortunately it’s not possible to run OpenJDK on the IBM i platform, so such 
> a test will not be possible, but it is worth mentioning that our product is a 
> standard product, which typically run on the Windows or Linux platform using 
> AdoptOpenJDK. Currently we have a couple of hundred installations running out 
> there on these platforms, and without any problems. But on the IBM I platform 
> with J9 we sometimes see this exception.
> Any good ideas for further investigations? Or have seen such issue before?
> We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform.
> h2.  Stacktrace
> {code:java}
> Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST 
> [Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: 
> java.lang.ArrayIndexOutOfBoundsException
> at 
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
> at 
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
> Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] 
> java.lang.ArrayIndexOutOfBoundsException
> at 
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
> at 
> org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
> at 
> org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
> at 
> org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
> at 
> org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
> at 
> org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
> at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
> at 
> org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
> at 
>

[jira] [Created] (LUCENE-8795) ArrayIndexOutOfBoundsException during System.arraycopy in BKDWriter

2019-05-07 Thread Torben Riis (JIRA) 
Torben Riis created LUCENE-8795:
---

 Summary: ArrayIndexOutOfBoundsException during System.arraycopy in 
BKDWriter
 Key: LUCENE-8795
 URL: https://issues.apache.org/jira/browse/LUCENE-8795
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/index
Affects Versions: 6.6
 Environment: h2. Operating system/Platform
{code:java}
IBM i - V7R3M0
{code}

h2. Java version

{code:java}
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 8.0.5.25 - 
pap6480sr5fp25-20181030_01(SR5 FP25))
IBM J9 VM (build 2.9, JRE 1.8.0 OS/400 ppc64-64-Bit Compressed References 
20181029_400846 (JIT enabled, AOT enabled)
OpenJ9   - c5c78da
OMR  - 3d5ac33
IBM  - 8c1bdc2)
JCL - 20181022_01 based on Oracle jdk8u191-b26
NOTICE: If no version information is found above, this could indicate a 
corrupted Java installation!
Java detected was: /QOpenSys/QIBM/ProdData/JavaVM/jdk80/64bit/bin/java
-Dmultiarchive.basepath=/home/NEXTOWN/Multi-Support/Next -Xms128m -Xmx2048m
{code}
Reporter: Torben Riis


Hi,

I’m a bit stuck here and needs a clue or two in order to continue our 
investigations. Hope that someone can help. :)

Periodically, around once a month, we get the below 
{{ArrayIndexOutOfBoundsException}} on our system. We use multiple indexes and 
the error can originate from any of them, but the error always occurs in line 
1217 in {{BKDWriter}} (during a {{System.arraycopy}}).

We found a couple of issues on the net regarding JIT optimization problem 
related to J9, but they all looks like they have been resolved and cannot be 
reproduced anymore. But nevertheless, we have added the following {{-Xjit}} 
flags to excludes JIT optimization for every class / inner classes in the 
{{bkd}} package. Moreover, we have also made a complete copy of the whole 
installation in production, and added the opposite arguments (enforce JIT 
optimizations for the specific classes). First we will try with 
{{optLevel=hot}}, but if this doesn’t show anything we will afterwards try with 
{{veryHot}} and {{scorching}}. Unfortunately we do not have the result of this 
yet, but I’ll of course post it when it is known.

Unfortunately it’s not possible to run OpenJDK on the IBM i platform, so such a 
test will not be possible, but it is worth mentioning that our product is a 
standard product, which typically run on the Windows or Linux platform using 
AdoptOpenJDK. Currently we have a couple of hundred installations running out 
there on these platforms, and without any problems. But on the IBM I platform 
with J9 we sometimes see this exception.

Any good ideas for further investigations? Or have seen such issue before?

We are using Lucene 6.6.0 and runs on IBM J9 on the IBM I platform.
h2.  Stacktrace
{code:java}
Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:07.970 CEST 
[Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.ArrayIndexOutOfBoundsException
at 
org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:703)
at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:683)
Caused by: 2019-05-01T06:10:07.971 CEST [Lucene Merge Thread #0] 
java.lang.ArrayIndexOutOfBoundsException
at 
org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1217)
at 
org.apache.lucene.util.bkd.BKDWriter.recursePackIndex(BKDWriter.java:1197)
at 
org.apache.lucene.util.bkd.BKDWriter.packIndex(BKDWriter.java:1078)
at 
org.apache.lucene.util.bkd.BKDWriter.writeIndex(BKDWriter.java:1245)
at 
org.apache.lucene.util.bkd.BKDWriter.access$600(BKDWriter.java:82)
at 
org.apache.lucene.util.bkd.BKDWriter$OneDimensionBKDWriter.finish(BKDWriter.java:648)
at org.apache.lucene.util.bkd.BKDWriter.merge(BKDWriter.java:560)
at 
org.apache.lucene.codecs.lucene60.Lucene60PointsWriter.merge(Lucene60PointsWriter.java:212)
at 
org.apache.lucene.index.SegmentMerger.mergePoints(SegmentMerger.java:173)
at 
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:122)
at 
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4356)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3931)
at 
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624)
at 
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661)
Exception in thread "Lucene Merge Thread #0" 2019-05-01T06:10:08.075 CEST 
[Lucene Merge Thread #0] org.apache.lucene.index.MergePolicy$MergeException: 
java.lang.ArrayIndexOutOfBoundsException
at