Hi karaf-Users,
we have a sporadic issue with Karaf initialization of JMX
ConnectorServerFactory due to "Unable to lookup configured keystore and/or
truststore" (see stacktrace below [1]). The ConnectorServerFactory uses
JAAS keystore for configuration of encrypted SSL for JMX via blueprint
config and fileInstall for initialization of the keystore.xml at specific
startlevel. In about 1 of 10 restarts (using "clean" for full
initialization), the keys are not loaded into the keystore and the
MBeanServer is not initialized correctly. This seems to be due to the fact
that on "init()" of ConnectorServerFactory bean the setupSSL() is called
and tries to retrieve the keystore from the OsgiKeystoreManager. This call
does not wait for the keystore to be registered so if due to timing issues
the jaas keystore not having been loaded yet the method fails immediately
with the exception [1] leaving the container in an unusable state. Is there
as possibility for the ConnectorServerFactory to wait on the keystore being
available at the keystoreManager and delay initialization?
Current startlevels do not allow for much leeway regarding loading the jaas
keystore:
- level 24 - initialization of the "blueprint" wrapper
- level 25 - recommended startlevel in felix fileinstall for loading
keystore.xml
- level 30 - initialization of karaf-management component
[1]
[2016-02-18 09:48:03,623] [ERROR] [FelixStartLevel]
[o.a.a.b.c.BlueprintContainerImpl] 403 | [] [21 -
org.apache.aries.blueprint.core - 1.4.2] [] [] [] [] [] [] Unable to start
blueprint container for bundl
e org.apache.karaf.management.server
org.osgi.service.blueprint.container.ComponentDefinitionException: Unable
to initialize bean connectorFactory
at
org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:714)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:824)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:787)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)
[org.apache.aries.blueprint.core:1.4.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
[na:1.7.0_51]
at
org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:245)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:183)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:682)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:377)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:269)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:294)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:263)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:253)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)
[org.apache.aries.util:1.1.0]
at
org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)
[org.apache.aries.util:1.1.0]
at
org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)
[org.apache.aries.util:1.1.0]
at
org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)
[org.apache.aries.util:1.1.0]
at
org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)
[org.apache.aries.util:1.1.0]
at
org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1127)
[org.apache.felix.framework-4.4.1.jar:na]
at
org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:696)
[org.apache.felix.framework-4.4.1.jar:na]
at
org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:484)
[org.apache.felix.framework-4.4.1.jar:na]
at
org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4429)
[org.apache.felix.framework-4.4.1.jar:na]
at org.apache.felix.framework.Felix.startBundle(Felix.java:2100)
[org.apache.felix.framework-4.4.1.jar:na]
at
org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1299)
[org.apache.felix.framework-4.4.1.jar:na]
at
org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)
[org.apache.felix.framework-4.4.1.jar:na]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
Caused by: java.security.GeneralSecurityException: Unable to lookup
configured keystore and/or truststore
at
org.apache.karaf.jaas.config.impl.OsgiKeystoreManager.createSSLContext(OsgiKeystoreManager.java:70)
[na:na]
at
org.apache.karaf.jaas.config.impl.OsgiKeystoreManager.createSSLServerFactory(OsgiKeystoreManager.java:100)
[na:na]
at
Proxy28fdb3db_ffe5_42c8_9b3e_26c55cec0cfc.createSSLServerFactory(Unknown
Source) [na:na]
at
org.apache.karaf.management.ConnectorServerFactory.setupSsl(ConnectorServerFactory.java:285)
[na:na]
at
org.apache.karaf.management.ConnectorServerFactory.init(ConnectorServerFactory.java:217)
[na:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[na:1.7.0_51]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
[na:1.7.0_51]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[na:1.7.0_51]
at java.lang.reflect.Method.invoke(Method.java:606) [na:1.7.0_51]
at
org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:297)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:958)
[org.apache.aries.blueprint.core:1.4.2]
at
org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:712)
[org.apache.aries.blueprint.core:1.4.2]
... 26 common frames omitted
Thanks and Best Regards,
Michael
input for analysis:
[2] Fuse Remote JMX SSL guide:
https://access.redhat.com/documentation/en-US/Red_Hat_JBoss_Fuse/6.0/html/Security_Guide/files/ESBSecurityJmxSSL.html
[3] Karaf Security Framework guide:
http://karaf.apache.org/manual/latest-3.0.x/developers-guide/security-framework.html
[4] Previous Karaf-User Question regarding jaas:
http://karaf.922171.n3.nabble.com/JAAS-SSL-Issue-LDAPLoginModule-setupSsl-calls-OsgiKeystoreManager-createSSLFactory-with-timestamp-of0-td4026149.html
[5] ConnectorServerFactory source:
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.karaf.management/org.apache.karaf.management.server/3.0.4/org/apache/karaf/management/ConnectorServerFactory.java#ConnectorServerFactory.init%28%29
