Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!
Will check tomorrow, it's too late now. On Jenkins there were no windows builds with IBM and Java 11 yet: https://jenkins.thetaphi.de/job/Lucene-9.x-Windows/ Am 12.11.2023 um 22:00 schrieb Dawid Weiss: Hi Uwe, Can you reproduce this on Windows with the same JVM versions though? Seems like I have exactly the same setup and yet this works for me just fine. Strange. Dawid On Sun, Nov 12, 2023 at 9:52 PM Uwe Schindler wrote: This one was my first idea, too. It fails only with IBM Semeru in combination with Gradle using Temurin. I will dig tomorrow on Jenkins server and print all debug info. Uwe Am 12. November 2023 21:48:54 MEZ schrieb Dawid Weiss : I can't reproduce this though - used exactly the same JVMs (on Windows): > gradlew :lucene:distribution.tests:compileTestJava --rerun-tasks --console=plain Generating gradle.properties ... > Task :altJvmWarning NOTE: Alternative java toolchain will be used for compilation and tests: Project will use 11 (IBM JDK 11.0.20.1+1, home at: c:\_tmp\jdk-11.0.20.1+1) Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at: C:\_tmp\jdk-11.0.21+9) ... > Task :lucene:distribution.tests:compileJava NO-SOURCE > Task :lucene:distribution.tests:classes UP-TO-DATE > Task :lucene:distribution.tests:compileTestJava BUILD SUCCESSFUL in 23s 5 actionable tasks: 5 executed On main branch it works, no idea why: O thought it's because of this: https://github.com/apache/lucene/commit/2e12a35c876a but I don't think so... seems to work for me on Windows on branch_9x just fine? D. -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de -- Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail:u...@thetaphi.de
Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!
Hi Uwe, Can you reproduce this on Windows with the same JVM versions though? Seems like I have exactly the same setup and yet this works for me just fine. Strange. Dawid On Sun, Nov 12, 2023 at 9:52 PM Uwe Schindler wrote: > This one was my first idea, too. > > It fails only with IBM Semeru in combination with Gradle using Temurin. > > I will dig tomorrow on Jenkins server and print all debug info. > > Uwe > > > Am 12. November 2023 21:48:54 MEZ schrieb Dawid Weiss < > dawid.we...@gmail.com>: > >> >> I can't reproduce this though - used exactly the same JVMs (on Windows): >> >> > gradlew :lucene:distribution.tests:compileTestJava --rerun-tasks >> --console=plain >> Generating gradle.properties >> ... >> > Task :altJvmWarning >> NOTE: Alternative java toolchain will be used for compilation and tests: >> Project will use 11 (IBM JDK 11.0.20.1+1, home at: >> c:\_tmp\jdk-11.0.20.1+1) >> Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at: >> C:\_tmp\jdk-11.0.21+9) >> ... >> > Task :lucene:distribution.tests:compileJava NO-SOURCE >> > Task :lucene:distribution.tests:classes UP-TO-DATE >> > Task :lucene:distribution.tests:compileTestJava >> >> BUILD SUCCESSFUL in 23s >> 5 actionable tasks: 5 executed >> >> On main branch it works, no idea why: >>> >> >> O thought it's because of this: >> >> https://github.com/apache/lucene/commit/2e12a35c876a >> >> but I don't think so... seems to work for me on Windows on branch_9x just >> fine? >> >> D. >> >>> -- > Uwe Schindler > Achterdiek 19, 28357 Bremen > https://www.thetaphi.de >
Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!
This one was my first idea, too. It fails only with IBM Semeru in combination with Gradle using Temurin. I will dig tomorrow on Jenkins server and print all debug info. Uwe Am 12. November 2023 21:48:54 MEZ schrieb Dawid Weiss : >I can't reproduce this though - used exactly the same JVMs (on Windows): > >> gradlew :lucene:distribution.tests:compileTestJava --rerun-tasks >--console=plain >Generating gradle.properties >... >> Task :altJvmWarning >NOTE: Alternative java toolchain will be used for compilation and tests: > Project will use 11 (IBM JDK 11.0.20.1+1, home at: >c:\_tmp\jdk-11.0.20.1+1) > Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at: >C:\_tmp\jdk-11.0.21+9) >... >> Task :lucene:distribution.tests:compileJava NO-SOURCE >> Task :lucene:distribution.tests:classes UP-TO-DATE >> Task :lucene:distribution.tests:compileTestJava > >BUILD SUCCESSFUL in 23s >5 actionable tasks: 5 executed > >On main branch it works, no idea why: >> > >O thought it's because of this: > >https://github.com/apache/lucene/commit/2e12a35c876a > >but I don't think so... seems to work for me on Windows on branch_9x just >fine? > >D. > >> -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!
I can't reproduce this though - used exactly the same JVMs (on Windows): > gradlew :lucene:distribution.tests:compileTestJava --rerun-tasks --console=plain Generating gradle.properties ... > Task :altJvmWarning NOTE: Alternative java toolchain will be used for compilation and tests: Project will use 11 (IBM JDK 11.0.20.1+1, home at: c:\_tmp\jdk-11.0.20.1+1) Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at: C:\_tmp\jdk-11.0.21+9) ... > Task :lucene:distribution.tests:compileJava NO-SOURCE > Task :lucene:distribution.tests:classes UP-TO-DATE > Task :lucene:distribution.tests:compileTestJava BUILD SUCCESSFUL in 23s 5 actionable tasks: 5 executed On main branch it works, no idea why: > O thought it's because of this: https://github.com/apache/lucene/commit/2e12a35c876a but I don't think so... seems to work for me on Windows on branch_9x just fine? D. >
Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!
One addition: Solr is affected on same way. Am 12. November 2023 21:39:07 MEZ schrieb Uwe Schindler : >Thanks. > >I have the feeling it is because of same major version. If Gradle JDK is >Eclipse Temurin and runtime JDK is same version but OpenJ9 it fails. > >Interestingly only in 11 (branch 9x). On main it worked for long time. Jdk17 >as Gradle runtime by Temurin and openj9 as runtime. > >If runtime and Gradle VM is identical home dir it does not fork. Here is a >flight difference and that matters. The alternate VM info is printed, but it >looks like it f*cks up when providing options and something thinks it does not >fork, but it does. > >> Task :altJvmWarning >NOTE: Alternative java toolchain will be used for compilation and tests: > Project will use 11 (IBM JDK 11.0.20.1+1, home at: > /home/jenkins/tools/java/64bit/openj9/jdk-11.0.20) > Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at: > /home/jenkins/tools/java/64bit/hotspot/jdk-11.0.21) > >On main branch it works, no idea why: > >> Task :altJvmWarning >NOTE: Alternative java toolchain will be used for compilation and tests: > Project will use 17 (IBM JDK 17.0.8.1+1, home at: > /home/jenkins/tools/java/64bit/openj9/jdk-17.0.8) > Gradle runs with 17 (Eclipse Temurin JDK 17.0.9+9, home at: > /home/jenkins/tools/java/64bit/hotspot/jdk-17.0.9) > >In case you ask, die to randomized jvms the Jenkins job Always runs with >default temurin minimum jdk for Gradle and just exchanges runtime for testing >(because Gradle itself can't run on all JDKs out there). > >One other thing: if you set OpenJ9 as default jdk and run Gradle with it, it >fails while building buildSrc: it can't compile the profiling stuff as flight >recorder module does not ship with OpenJ9. We should possibly fix this, maybe >by rewriting the code with pure Groovy class and not compile it at all. > >Uwe > >Am 12. November 2023 21:08:41 MEZ schrieb Dawid Weiss : >>Hi Uwe, >> >>Will dig tomorrow. Maybe Dawid has an idea? It looks like the alternate >>> runtime is correctly detected, but why is Gradle passing compiler runzine >>> options without -J in just this case. In Main the same works where Gradle >>> runs with Java17-Temurin and J9 is used as runtime. >>> >> >>I think I know what this is - please let me verify and provide a PR, if >>it's indeed that. >> >>Dawid > >-- >Uwe Schindler >Achterdiek 19, 28357 Bremen >https://www.thetaphi.de -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!
Thanks. I have the feeling it is because of same major version. If Gradle JDK is Eclipse Temurin and runtime JDK is same version but OpenJ9 it fails. Interestingly only in 11 (branch 9x). On main it worked for long time. Jdk17 as Gradle runtime by Temurin and openj9 as runtime. If runtime and Gradle VM is identical home dir it does not fork. Here is a flight difference and that matters. The alternate VM info is printed, but it looks like it f*cks up when providing options and something thinks it does not fork, but it does. > Task :altJvmWarning NOTE: Alternative java toolchain will be used for compilation and tests: Project will use 11 (IBM JDK 11.0.20.1+1, home at: /home/jenkins/tools/java/64bit/openj9/jdk-11.0.20) Gradle runs with 11 (Eclipse Temurin JDK 11.0.21+9, home at: /home/jenkins/tools/java/64bit/hotspot/jdk-11.0.21) On main branch it works, no idea why: > Task :altJvmWarning NOTE: Alternative java toolchain will be used for compilation and tests: Project will use 17 (IBM JDK 17.0.8.1+1, home at: /home/jenkins/tools/java/64bit/openj9/jdk-17.0.8) Gradle runs with 17 (Eclipse Temurin JDK 17.0.9+9, home at: /home/jenkins/tools/java/64bit/hotspot/jdk-17.0.9) In case you ask, die to randomized jvms the Jenkins job Always runs with default temurin minimum jdk for Gradle and just exchanges runtime for testing (because Gradle itself can't run on all JDKs out there). One other thing: if you set OpenJ9 as default jdk and run Gradle with it, it fails while building buildSrc: it can't compile the profiling stuff as flight recorder module does not ship with OpenJ9. We should possibly fix this, maybe by rewriting the code with pure Groovy class and not compile it at all. Uwe Am 12. November 2023 21:08:41 MEZ schrieb Dawid Weiss : >Hi Uwe, > >Will dig tomorrow. Maybe Dawid has an idea? It looks like the alternate >> runtime is correctly detected, but why is Gradle passing compiler runzine >> options without -J in just this case. In Main the same works where Gradle >> runs with Java17-Temurin and J9 is used as runtime. >> > >I think I know what this is - please let me verify and provide a PR, if >it's indeed that. > >Dawid -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!
Hi Uwe, Will dig tomorrow. Maybe Dawid has an idea? It looks like the alternate > runtime is correctly detected, but why is Gradle passing compiler runzine > options without -J in just this case. In Main the same works where Gradle > runs with Java17-Temurin and J9 is used as runtime. > I think I know what this is - please let me verify and provide a PR, if it's indeed that. Dawid
Re: [JENKINS] Lucene-9.x-Linux (64bit/openj9/jdk-11.0.20) - Build # 14015 - Still Failing!
It is unclear why this happens with Java 11 version of OpenJ9 only. Java 17 works. The compiler is forked as Gradle runs with Temurin, J9 is used via RUNTIME_JAVA_HOME. Will dig tomorrow. Maybe Dawid has an idea? It looks like the alternate runtime is correctly detected, but why is Gradle passing compiler runzine options without -J in just this case. In Main the same works where Gradle runs with Java17-Temurin and J9 is used as runtime. Uwe Am 12. November 2023 18:37:04 MEZ schrieb Policeman Jenkins Server : >Build: https://jenkins.thetaphi.de/job/Lucene-9.x-Linux/14015/ >Java: 64bit/openj9/jdk-11.0.20 -XX:+UseCompressedOops -Xgcpolicy:balanced > >No tests ran. -- Uwe Schindler Achterdiek 19, 28357 Bremen https://www.thetaphi.de
[jira] [Commented] (PYLUCENE-69) Linking libjvm seems to prevent PyPi upload
[
https://issues.apache.org/jira/browse/PYLUCENE-69?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17785302#comment-17785302
]
Clément Jonglez commented on PYLUCENE-69:
-
Thank you for the honest answer! I'll try uploading a source distribution to
PyPi instead of binaries then.
> Linking libjvm seems to prevent PyPi upload
> ---
>
> Key: PYLUCENE-69
> URL: https://issues.apache.org/jira/browse/PYLUCENE-69
> Project: PyLucene
> Issue Type: Question
>Reporter: Clément Jonglez
>Priority: Major
>
> As mentioned in
> [https://issues.apache.org/jira/projects/PYLUCENE/issues/PYLUCENE-68] , I am
> trying to package the Orekit Python wrapper from [~petrush] (which uses JCC)
> to PyPi.
> I used the --wheel option to compile a wheel and not an egg, and I tried to
> upload the wheel to PyPi.
> But PyPi refuses my wheel with the answer :
> {noformat}
> Binary wheel 'orekit-11.3.3-cp312-cp312-linux_x86_64.whl' has an unsupported
> platform tag 'linux_x86_64'. {noformat}
> After reading about why this error occurs (
> [https://peps.python.org/pep-0513/#rationale] ) , I tried to convert the
> wheel to a manylinux wheel by using:
>
> {code:java}
> auditwheel repair dist/orekit-11.3.3-cp312-cp312-linux_x86_64.whl{code}
>
> Which returned the following error:
>
> {noformat}
> auditwheel: error: cannot repair
> "dist/orekit-11.3.3-cp312-cp312-linux_x86_64.whl" to "manylinux_2_5_x86_64"
> ABI because of the presence of too-recent versioned symbols. You'll need to
> compile the wheel on an older toolchain.{noformat}
>
> I then ran the following command to get more information about which symbols
> are problematic in the wheel:
>
> {code:java}
> auditwheel-symbols --manylinux 2_34
> dist/orekit-11.3.3-cp312-cp312-linux_x86_64.whl{code}
>
> Which returned:
>
> {noformat}
> orekit/_orekit.cpython-312-x86_64-linux-gnu.so is not manylinux_2_34
> compliant because it links the following forbidden libraries:
> libjvm.so{noformat}
>
>
> I tried removing -ljvm from the LFLAGS list in jcc's config.py, but as
> expected the Python program then fails on starting the JVM:
> {noformat}
> Traceback (most recent call last):
> File "/media/ssd/git/orekit_python_artifacts/test/AbstractDetectorTest.py",
> line 3, in
> import orekit
> File
> "/home/yzokras/Documents/orekit-pip/orekit312/lib/python3.12/site-packages/orekit-12.0-py3.12-linux-x86_64.egg/orekit/__init__.py",
> line 7, in
> from . import _orekit
> ImportError:
> /home/yzokras/Documents/orekit-pip/orekit312/lib/python3.12/site-packages/orekit-12.0-py3.12-linux-x86_64.egg/orekit/_orekit.cpython-312-x86_64-linux-gnu.so:
> undefined symbol: JNI_GetDefaultJavaVMInitArgs{noformat}
>
> I don't have any more clues... Alternatively, I could try to package the
> Orekit Python wrapper as a source distribution (using
> [https://issues.apache.org/jira/projects/PYLUCENE/issues/PYLUCENE-27] and
> [https://issues.apache.org/jira/projects/PYLUCENE/issues/PYLUCENE-68] ), but
> this would mean that a user would need to wait approx. 10 minutes for the
> wheel to compile when running pip install...
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: Welcome Patrick Zhai to the Lucene PMC
Thank you everyone! On Sun, Nov 12, 2023, 09:34 Dawid Weiss wrote: > > > Congratulations and welcome, Patrick! > > Dawid > > On Fri, Nov 10, 2023 at 9:05 PM Michael McCandless < > luc...@mikemccandless.com> wrote: > >> I'm happy to announce that Patrick Zhai has accepted an invitation to >> join the Lucene Project Management Committee (PMC)! >> >> Congratulations Patrick, thank you for all your hard work improving >> Lucene's community and source code, and welcome aboard! >> >> Mike McCandless >> >> http://blog.mikemccandless.com >> >
Re: Welcome Patrick Zhai to the Lucene PMC
Congratulations and welcome, Patrick! Dawid On Fri, Nov 10, 2023 at 9:05 PM Michael McCandless < luc...@mikemccandless.com> wrote: > I'm happy to announce that Patrick Zhai has accepted an invitation to join > the Lucene Project Management Committee (PMC)! > > Congratulations Patrick, thank you for all your hard work improving > Lucene's community and source code, and welcome aboard! > > Mike McCandless > > http://blog.mikemccandless.com >
Re: Ascii folding
Thanks Robert, Uwe - all this is enlightening. I didn't know about those
things you mentioned.
Dawid
On Sat, Nov 11, 2023 at 2:02 PM Uwe Schindler wrote:
> Hi Dawid,
>
> the ASCII folding filter is meant to remove accents. You would like to
> have searching for visually similar characters. These are 2 different
> things.
>
> Actually Robert also has some config options, waht I generally use for
> wester european searches where some documents may contain names of people
> (Author names, titles in cyrillic or other languages) it to convert the
> tokens using ICU transliteration (use one of the ICU folding filters with
> the below config):
>
> Transliterator.getInstance("Any-Latin; NFD; [:Nonspacing Mark:] Remove;
> NFKC; CaseFold", Transliterator.FORWARD);
>
> This does convert everything to latin characters in a language-neutral way
> and then removes all accents by the trick "decompose, remove non-spacing
> mark, compose again and case-fold the result.
>
> Uwe
> Am 10.11.2023 um 19:03 schrieb Dawid Weiss:
>
>
> Hi Steve, Chris,
>
> Ok, makes sense. Thanks for the pointers. I agree the justification for
> the use of character-level normalization filters is highly
> context-dependent (for example, unsuitable when mixed languages are present
> on input).
>
> Dawid
>
> On Fri, Nov 10, 2023 at 6:58 PM Chris Hostetter
> wrote:
>
>>
>> : Here's the unicode letter after "th":
>> : https://www.fileformat.info/info/unicode/char/0435/index.htm
>> :
>> : To my surprise, I couldn't find it in the ascii folding filter:
>> :
>> :
>> https://github.com/apache/lucene/blob/main/lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/ASCIIFoldingFilter.java
>> :
>> : Anybody remembers whether the omission of Cyrillic characters was
>> : intentional (there is quite a few of them that are nearly identical in
>> : appearance to Latin letters).
>>
>> From the javadocs, i'm going to guess it's because the the filter focuses
>> on "Latin_characters_in_Unicode" ... and your "CYRILLIC SMALL LETTER IE"
>> isn't described as being a "(adjective) LATIN noun (WITH noun)" like all
>> of the other characters that are considered to have a direct mapping to
>> the "ASCII" / latin characters.
>>
>> If you look back at when it was added...
>>
>> https://issues.apache.org/jira/browse/LUCENE-1390
>>
>> ...the original focus was on deprecating "ISOLatin1AccentFilter" and
>> replacing it with "a more comprehensive version of this code that
>> included
>> not just ISO-Latin-1 (ISO-8859-1) but the entire Latin 1 and Latin
>> Extended A unicode blocks." (The originally proposed name was
>> 'ISOLatinAccentFilter') ... subsequent discussion focused on adding more
>> Latin blocks.
>>
>> There was a related issue at the time which initially aimed to add a
>> more general "UnicodeNormalizationFilter" that ultimated resulted in
>> adding the "ICU" analysis classes...
>>
>> https://issues.apache.org/jira/browse/LUCENE-1343
>>
>> ..which IIUC may better handle "CYRILLIC SMALL LETTER IE" (but i haven't
>> tested that)
>>
>>
>>
>> -Hoss
>> http://www.lucidworks.com/
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>> --
> Uwe Schindler
> Achterdiek 19, D-28357 Bremenhttps://www.thetaphi.de
> eMail: u...@thetaphi.de
>
>
Re: Welcome Patrick Zhai to the Lucene PMC
Congrats, Patrick! On Sun, Nov 12, 2023 at 4:30 AM Michael Sokolov wrote: > Welcome, Patrick! > > On Sun, Nov 12, 2023, 2:12 AM Ignacio Vera wrote: > >> Welcome Patrick! >> >> On Sat, Nov 11, 2023 at 3:29 PM Uwe Schindler wrote: >> >>> Welcome Patrick! >>> >>> Uwe >>> >>> >>> Am 10. November 2023 21:04:32 MEZ schrieb Michael McCandless < >>> luc...@mikemccandless.com>: >>> I'm happy to announce that Patrick Zhai has accepted an invitation to join the Lucene Project Management Committee (PMC)! Congratulations Patrick, thank you for all your hard work improving Lucene's community and source code, and welcome aboard! Mike McCandless http://blog.mikemccandless.com >>> -- >>> Uwe Schindler >>> Achterdiek 19, 28357 Bremen >>> https://www.thetaphi.de >>> >>
Re: Welcome Patrick Zhai to the Lucene PMC
Welcome, Patrick! On Sun, Nov 12, 2023, 2:12 AM Ignacio Vera wrote: > Welcome Patrick! > > On Sat, Nov 11, 2023 at 3:29 PM Uwe Schindler wrote: > >> Welcome Patrick! >> >> Uwe >> >> >> Am 10. November 2023 21:04:32 MEZ schrieb Michael McCandless < >> luc...@mikemccandless.com>: >> >>> I'm happy to announce that Patrick Zhai has accepted an invitation to >>> join the Lucene Project Management Committee (PMC)! >>> >>> Congratulations Patrick, thank you for all your hard work improving >>> Lucene's community and source code, and welcome aboard! >>> >>> Mike McCandless >>> >>> http://blog.mikemccandless.com >>> >> -- >> Uwe Schindler >> Achterdiek 19, 28357 Bremen >> https://www.thetaphi.de >> >
