Hi
This "finding" is related to [KARAF-7095] "JDK11+: we should use full lib path
when patch-module", but not exactly the same.
We have observed that, for at least Karaf 4.4.1; when using JDK11, and starting
an instance the patch-module option uses a relative path to the endorsed
libraries.
Per my understanding, this would be relative to the instance root
(${KARAF_HOME}/instances/xxx/), which by default does not contain any lib
directory.
This would only be a problem for instances, since the root-instance would find
the libraries where expected (${KARAF_HOME}/lib/endorsed/)
We are already doing some adaptations to the installation, explaining the
"apache-custom-karaf" in some of the paths below, so it would be fairly easy
for us to fix instance-specific endorsed libraries in one way or the other.
But, the question is, have I misunderstood something fundamental here?
--- More details, and examples ---
Starting the instance 'iodtest' (used for our own integration tests, i.e. not a
trivial HelloWorld instance) with JDK11 and Karaf 4.4.1.
Some iodtest-specific flags/options have been excluded.
---
[org.apache.karaf.instance.core.internal.InstanceServiceImpl] DEBUG : Starting
instance iodtest with command:
"/usr/lib/jvm/java-11-openjdk-11.0.17.0.8-2.el7_9.x86_64/bin/java" -server
-XX:+UseCompressedOops -XX:+UseBiasedLocking -XX:+TieredCompilation
-XX:+AlwaysPreTouch -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSWaitDuration=60000 -XX:+CMSScavengeBeforeRemark
-XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways
-XX:+DisableExplicitGC -Xloggc:/var/opt/log/iodtest/iodtestGc.log
-XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256M -XX:ReservedCodeCacheSize=128M
-XX:CompressedClassSpaceSize=32M --add-reads=java.xml=java.logging
--add-exports=java.base/org.apache.karaf.specs.locator=java.xml,ALL-UNNAMED
--patch-module java.base=lib/endorsed/org.apache.karaf.specs.locator-4.4.1.jar
--patch-module java.xml=lib/endorsed/org.apache.karaf.specs.java.xml-4.4.1.jar
--add-opens java.base/java.security=ALL-UNNAMED --add-opens
java.base/java.net=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED
--add-opens java.base/java.util=ALL-UNNAMED --add-opens
java.naming/javax.naming.spi=ALL-UNNAMED --add-opens
java.rmi/sun.rmi.transport.tcp=ALL-UNNAMED
--add-exports=java.base/sun.net.www.protocol.file=ALL-UNNAMED
--add-exports=java.base/sun.net.www.protocol.ftp=ALL-UNNAMED
--add-exports=java.base/sun.net.www.protocol.http=ALL-UNNAMED
--add-exports=java.base/sun.net.www.protocol.https=ALL-UNNAMED
--add-exports=java.base/sun.net.www.protocol.jar=ALL-UNNAMED
--add-exports=java.base/sun.net.www.content.text=ALL-UNNAMED
--add-exports=jdk.xml.dom/org.w3c.dom.html=ALL-UNNAMED
--add-exports=java.rmi/sun.rmi.registry=ALL-UNNAMED
--add-exports=jdk.naming.rmi/com.sun.jndi.url.rmi=ALL-UNNAMED
-Djava.util.logging.config.file="/opt/osgi/apache-custom-karaf/instances/iodtest/etc/java.util.logging.properties"
-Dkaraf.home="/opt/osgi/apache-custom-karaf"
-Dkaraf.base="/opt/osgi/apache-custom-karaf/instances/iodtest"
-Dkaraf.data="/opt/osgi/apache-custom-karaf/instances/iodtest/data"
-Dkaraf.etc="/opt/osgi/apache-custom-karaf/instances/iodtest/etc"
-Dkaraf.log="/opt/osgi/apache-custom-karaf/instances/iodtest/data/log"
-Djava.io.tmpdir="/opt/osgi/apache-custom-karaf/instances/iodtest/data/tmp"
-Dkaraf.restart.jvm.supported=true -Dkaraf.startLocalConsole=false
-Dkaraf.startRemoteShell=true -classpath
"/opt/osgi/apache-custom-karaf/lib/boot/org.apache.karaf.diagnostic.boot-4.4.1.jar:/opt/osgi/apache-custom-karaf/lib/boot/osgi.core-8.0.0.jar:/opt/osgi/apache-custom-karaf/lib/boot/org.apache.karaf.jaas.boot-4.4.1.jar:/opt/osgi/apache-custom-karaf/lib/boot/org.apache.karaf.specs.activator-4.4.1.jar:/opt/osgi/apache-custom-karaf/lib/boot/org.apache.karaf.main-4.4.1.jar:/opt/osgi/apache-custom-karaf/lib/jdk9plus/jakarta.xml.bind-api-2.3.3.jar:/opt/osgi/apache-custom-karaf/lib/jdk9plus/javax.annotation-api-1.3.2.jar:/opt/osgi/apache-custom-karaf/lib/jdk9plus/org.apache.servicemix.specs.activation-api-1.2.1-1.2.1_3.jar:/opt/osgi/apache-custom-karaf/lib/jdk9plus/istack-commons-runtime-3.0.10.jar:/opt/osgi/apache-custom-karaf/lib/jdk9plus/txw2-2.3.3.jar:/opt/osgi/apache-custom-karaf/lib/jdk9plus/jaxb-runtime-2.3.3.jar"
org.apache.karaf.main.Main server
The options we have been focused on are in particular these two
--patch-module java.base=lib/endorsed/org.apache.karaf.specs.locator-4.4.1.jar
--patch-module java.xml=lib/endorsed/org.apache.karaf.specs.java.xml-4.4.1.jar
The following is observed in the karaf.log
(/opt/osgi/apache-custom-karaf/instances/iodtest/data/log/karaf.log)
---
Feb 10, 2023 8:26:31 AM org.apache.karaf.main.Main launch
INFO: Installing and starting initial bundles
Feb 10, 2023 8:26:32 AM org.apache.karaf.main.Main launch
INFO: All initial bundles installed and set to start
Feb 10, 2023 8:26:32 AM org.apache.karaf.main.KarafActivatorManager
startKarafActivators
WARNING: Error starting karaf activator
org.apache.karaf.specs.activator.Activator from url
jar:file:/opt/osgi/apache-custom-karaf/lib/boot/org.apache.karaf.specs.activator-4.4.1.jar!/META-INF/MANIFEST.MF
java.lang.NoClassDefFoundError: org/apache/karaf/specs/locator/OsgiLocator
at org.apache.karaf.specs.activator.Activator.register(Activator.java:125)
at org.apache.karaf.specs.activator.Activator.start(Activator.java:72)
at
org.apache.karaf.main.KarafActivatorManager.startKarafActivators(KarafActivatorManager.java:61)
at org.apache.karaf.main.Main.launch(Main.java:316)
at org.apache.karaf.main.Main.main(Main.java:183)
Caused by: java.lang.ClassNotFoundException:
org.apache.karaf.specs.locator.OsgiLocator
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
... 5 more
Feb 10, 2023 8:26:32 AM org.apache.karaf.main.lock.SimpleFileLock lock
INFO: Trying to lock /opt/osgi/apache-custom-karaf/instances/iodtest/data/lock
Feb 10, 2023 8:26:32 AM org.apache.karaf.main.lock.SimpleFileLock lock
INFO: Lock acquired
Feb 10, 2023 8:26:32 AM org.apache.karaf.main.Main$KarafLockCallback
lockAcquired
INFO: Lock acquired. Setting startlevel to 100
---
The exception is in line with the earlier observations. Copy-pasting
${KARAF_HOME}/lib to ${KARAF_BASE} prior to instance start, will not generate
the exception above.
--- ---
The same operation using Java8
---
[org.apache.karaf.instance.core.internal.InstanceServiceImpl] DEBUG : Starting
instance iodtest with command: "/usr/java/jdk1.8.0_351-amd64/jre/bin/java"
-server -XX:+UseCompressedOops -XX:+UseBiasedLocking -XX:+TieredCompilation
-XX:+AlwaysPreTouch -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=80 -XX:+UseCMSInitiatingOccupancyOnly
-XX:CMSWaitDuration=60000 -XX:+CMSScavengeBeforeRemark
-XX:+CMSParallelInitialMarkEnabled -XX:+CMSEdenChunksRecordAlways
-XX:+DisableExplicitGC -Xloggc:/var/opt/log/iodtest/iodtestGc.log
-XX:MetaspaceSize=256M -XX:MaxMetaspaceSize=256M -XX:ReservedCodeCacheSize=128M
-XX:CompressedClassSpaceSize=32M -XX:+UseParNewGC
-XX:+PrintTenuringDistribution -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
-XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3
-XX:GCLogFileSize=30M
-Djava.endorsed.dirs="/usr/java/jdk1.8.0_351-amd64/jre/jre/lib/endorsed:/usr/java/jdk1.8.0_351-amd64/jre/lib/endorsed:/opt/osgi/apache-custom-karaf/lib/endorsed"
-Djava.ext.dirs="/usr/java/jdk1.8.0_351-amd64/jre/jre/lib/ext:/usr/java/jdk1.8.0_351-amd64/jre/lib/ext:/opt/osgi/apache-custom-karaf/lib/ext"
-Djava.util.logging.config.file="/opt/osgi/apache-custom-karaf/instances/iodtest/etc/java.util.logging.properties"
-Dkaraf.home="/opt/osgi/apache-custom-karaf"
-Dkaraf.base="/opt/osgi/apache-custom-karaf/instances/iodtest"
-Dkaraf.data="/opt/osgi/apache-custom-karaf/instances/iodtest/data"
-Dkaraf.etc="/opt/osgi/apache-custom-karaf/instances/iodtest/etc"
-Dkaraf.log="/opt/osgi/apache-custom-karaf/instances/iodtest/data/log"
-Djava.io.tmpdir="/opt/osgi/apache-custom-karaf/instances/iodtest/data/tmp"
-Dkaraf.restart.jvm.supported=true -Dkaraf.startLocalConsole=false
-Dkaraf.startRemoteShell=true -classpath
"/opt/osgi/apache-custom-karaf/lib/boot/org.apache.karaf.diagnostic.boot-4.4.1.jar:/opt/osgi/apache-custom-karaf/lib/boot/osgi.core-8.0.0.jar:/opt/osgi/apache-custom-karaf/lib/boot/org.apache.karaf.jaas.boot-4.4.1.jar:/opt/osgi/apache-custom-karaf/lib/boot/org.apache.karaf.specs.activator-4.4.1.jar:/opt/osgi/apache-custom-karaf/lib/boot/org.apache.karaf.main-4.4.1.jar"
org.apache.karaf.main.Main server
Here the relevant/corresponding options would be:
-Djava.endorsed.dirs="/usr/java/jdk1.8.0_351-amd64/jre/jre/lib/endorsed:/usr/java/jdk1.8.0_351-amd64/jre/lib/endorsed:/opt/osgi/apache-custom-karaf/lib/endorsed"
(observe that in this case absolute paths are being used)
No exception in karaf.log this time.
--- ---
With my understanding of the code (4.4.3), this behaviour would apply to JDK9
and later.
(doStart-method in
"instance/src/main/java/org/apache/karaf/instance/core/internal/InstanceServiceImpl.java")
regards
/Charlie
