Thanks Uwe! Responses inlined below: On Thu, Aug 17, 2023 at 9:46 AM Uwe Schindler <u...@thetaphi.de> wrote:
> this error indeed cannot happen as all our segments are shared. It could > still be some bug in the Java 19 version, did you try Java 21 or Java 20? > OK, phew, that is what I thought (from reading Lucene's sources). We are testing Java 20 now (though EOL is in ~4 weeks -- hard to keep up!), and soon Java 21. Maybe we'll just go straight to Java 21 if there are no problems. This exception only happened on a single host (and JVM) across a great many Lucene instances we run so whatever bug it is: it looks very rare. But, for that host, the exception happened a great many times, for the few hours that this JVM was alive. > It may also be a Coretto problem, maybe contact their team, maybe they > have applied some changes. ScopedMemoryAccess is using an extension to the > original Java memory model internally (I think the changed something in the > specs), so it changed quite a lot internally. Maybe Coretto has some > patches for hotspot that make the memory model changes hit us? > Good idea -- we will reach out to the Corretto team. Scary to be using extensions to JMM which was already complex enough to begin with! > I don't think the bug is in Lucene's code, because if a thread is shared, > it is shared. Maybe some other problem could be: Have you maybe > accidentally closed the IndexInput too early. Normally this should cause an > IllegalStateException (we have a test for this), but I am not fully sure > what happens if the shared scope was already closed. I remmeber there were > some bugs in 19, but it is already too long ago. So please try with plain > OpenJDK Java 21 (or 20). > I don't think we are closing IndexInputs too early -- we would see many many exceptions if so, I hope. We will expedite getting off 19. > I would like to know more about the speed improvements! In our > benchmarking they were not so visible (only a slight change), so happy to > see more. > Right, we were surprised too! We are still trying to isolate where the gains came from, but they were impactful for us (~5-7% reduction in CPU time somehow). We kept Java at 19 (we try not to upgrade both at the same time!). Mike McCandless http://blog.mikemccandless.com >
