[ https://issues.apache.org/jira/browse/KARAF-7156?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Varga updated KARAF-7156: -------------------------------- Priority: Minor (was: Major) > Intermittent failures to launch Karaf > ------------------------------------- > > Key: KARAF-7156 > URL: https://issues.apache.org/jira/browse/KARAF-7156 > Project: Karaf > Issue Type: Bug > Components: karaf > Affects Versions: 4.3.1 > Environment: This happens regularly in OpenDaylight verification > jobs, which are running Karaf as part of testing feature files via Pax Exam. > Some jobs contain upwards of 50 of such Karaf launches. > The failure is not consistently reproducible, but it occurs now and then. > When this error occurs, the test fails, with Pax Exam reporting: > > {noformat} > [INFO] Running org.opendaylight.odlparent.featuretest.SingleFeatureTest > [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: > 182.139 s <<< FAILURE! - in > org.opendaylight.odlparent.featuretest.SingleFeatureTest > [ERROR] > installFeatureCatchAndLog(org.opendaylight.odlparent.featuretest.SingleFeatureTest)[repoUrl: > > file:/w/workspace/mdsal-maven-verify-master-mvn35-openjdk11/features/odl-mdsal-model-rfc8519/target/feature/feature.xml, > Feature: odl-mdsal-model-rfc8519 8.0.0.SNAPSHOT].installFeatureCatchAndLog > Time elapsed: 181.223 s <<< ERROR! > java.rmi.NotBoundException: aafef1c7-c415-4efc-a6a0-e929277db48d > at java.rmi/sun.rmi.registry.RegistryImpl.lookup(RegistryImpl.java:234) > at > java.rmi/sun.rmi.registry.RegistryImpl_Skel.dispatch(RegistryImpl_Skel.java:133) > at > java.rmi/sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:468) > at > java.rmi/sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:298) > at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:200) > at java.rmi/sun.rmi.transport.Transport$1.run(Transport.java:197) > at java.base/java.security.AccessController.doPrivileged(Native Method) > at java.rmi/sun.rmi.transport.Transport.serviceCall(Transport.java:196) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:562) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:796) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:677) > at java.base/java.security.AccessController.doPrivileged(Native Method) > at > java.rmi/sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:676) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:834) > at > java.rmi/sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:303) > at > java.rmi/sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:279) > at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:380) > at > java.rmi/sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:123) > at > org.ops4j.pax.exam.rbc.client.intern.RemoteBundleContextClientImpl.getRemoteBundleContext(RemoteBundleContextClientImpl.java:248) > at > org.ops4j.pax.exam.rbc.client.intern.RemoteBundleContextClientImpl.waitForState(RemoteBundleContextClientImpl.java:218) > at > org.ops4j.pax.exam.karaf.container.internal.KarafTestContainer.waitForState(KarafTestContainer.java:721) > at > org.ops4j.pax.exam.karaf.container.internal.KarafTestContainer.startKaraf(KarafTestContainer.java:304) > at > org.ops4j.pax.exam.karaf.container.internal.KarafTestContainer.start(KarafTestContainer.java:191) > at > org.ops4j.pax.exam.spi.reactors.AllConfinedStagedReactor.invoke(AllConfinedStagedReactor.java:79) > at > org.ops4j.pax.exam.junit.impl.ProbeRunner$2.evaluate(ProbeRunner.java:267) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100){noformat} > Reporter: Robert Varga > Priority: Minor > Attachments: failed_karaf.log, successful_karaf.log > > > We seem to observe the following failure (similar to KARAF-5640), with > uninteresting threads filtered out: > {noformat} > 2021-05-20T15:46:08,884 | ERROR | Start Level: Equinox Container: > 351dc4de-7b03-4540-9b09-9c603b419792 | BootFeaturesInstaller | 16 > - org.apache.karaf.features.core - 4.3.1 | Error installing boot features > org.osgi.framework.BundleException: Unable to acquire the state change lock > for the module: osgi.identity; > osgi.identity="org.apache.karaf.features.core"; type="osgi.bundle"; > version:Version="4.3.1" [id=16] STOPPED [STARTED] > at org.eclipse.osgi.container.Module.lockStateChange(Module.java:350) > ~[?:?] > at org.eclipse.osgi.container.Module.stop(Module.java:513) ~[?:?] > at > org.eclipse.osgi.internal.framework.EquinoxBundle.stop(EquinoxBundle.java:467) > ~[?:?] > at > org.apache.karaf.features.internal.service.BundleInstallSupportImpl.stopBundle(BundleInstallSupportImpl.java:171) > ~[?:?] > at > org.apache.karaf.features.internal.service.FeaturesServiceImpl.stopBundle(FeaturesServiceImpl.java:1165) > ~[?:?] > at > org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:1000) > ~[?:?] > at > org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1069) > ~[?:?] > at > org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:1004) > ~[?:?] > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > ~[?:?] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > ~[?:?] > at java.lang.Thread.run(Thread.java:834) ~[?:?] > Caused by: java.util.concurrent.TimeoutException: Timeout after waiting 30 > seconds to acquire the lock. > at org.eclipse.osgi.container.Module.lockStateChange(Module.java:347) > ~[?:?] > ... 11 more > Caused by: org.eclipse.osgi.framework.util.ThreadInfoReport: Thread dump > {noformat} > {noformat} > ThreadId: 19 ThreadName: Start Level: Equinox Container: > 351dc4de-7b03-4540-9b09-9c603b419792 ThreadState: WAITING Blocked On: > java.util.concurrent.FutureTask@6f139765 LockOwnerId: -1 LockOwnerName: null > Synchronizers Locked: > java.util.concurrent.locks.ReentrantLock$NonfairSync@151d882b Monitors > Locked: java.lang.Object@3d3be3e4 Stack Trace: > java.base@11.0.10/jdk.internal.misc.Unsafe.park(Native Method) > java.base@11.0.10/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) > > java.base@11.0.10/java.util.concurrent.FutureTask.awaitDone(FutureTask.java:447) > > java.base@11.0.10/java.util.concurrent.FutureTask.get(FutureTask.java:190) > org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvisionInThread(FeaturesServiceImpl.java:1008) > > org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvisionInThread(FeaturesServiceImpl.java:978) > > org.apache.karaf.features.internal.service.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:826) > > org.apache.karaf.features.internal.service.FeaturesServiceImpl.installFeatures(FeaturesServiceImpl.java:746) > > org.apache.karaf.features.internal.service.BootFeaturesInstaller.installBootFeatures(BootFeaturesInstaller.java:97) > > org.apache.karaf.features.internal.service.BootFeaturesInstaller.start(BootFeaturesInstaller.java:87) > > org.apache.karaf.features.internal.osgi.Activator.doStart(Activator.java:214) > org.apache.karaf.util.tracker.BaseActivator.start(BaseActivator.java:92) > > org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:814) > > org.eclipse.osgi.internal.framework.BundleContextImpl$2.run(BundleContextImpl.java:1) > java.base@11.0.10/java.security.AccessController.doPrivileged(Native > Method) > org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:806) > > org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:763) > > org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1012) > > org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:366) > org.eclipse.osgi.container.Module.doStart(Module.java:605) > org.eclipse.osgi.container.Module.start(Module.java:468) > org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel$2.run(ModuleContainer.java:1845) > > org.eclipse.osgi.internal.framework.EquinoxContainerAdaptor$1$1.execute(EquinoxContainerAdaptor.java:136) > > org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1838) > > org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1781) > > org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1743) > > org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1665) > > org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1) > > org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:234) > > org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:345) > {noformat} > > {noformat} > ThreadId: 32 ThreadName: features-2-thread-1 ThreadState: RUNNABLE > Blocked On: > org.eclipse.osgi.internal.container.EquinoxReentrantLock@13da572a[Locked by > thread Start Level: Equinox Container: 351dc4de-7b03-4540-9b09-9c603b419792] > Synchronizers Locked: > java.util.concurrent.ThreadPoolExecutor$Worker@3fcc74ef > Monitors Locked: none > Stack Trace: > java.management@11.0.10/sun.management.ThreadImpl.dumpThreads0(Native > Method) > > java.management@11.0.10/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:521) > > java.management@11.0.10/sun.management.ThreadImpl.dumpAllThreads(ThreadImpl.java:509) > > org.eclipse.osgi.framework.util.ThreadInfoReport.getThreadDump(ThreadInfoReport.java:30) > > org.eclipse.osgi.framework.util.ThreadInfoReport.<init>(ThreadInfoReport.java:23) > org.eclipse.osgi.container.Module.lockStateChange(Module.java:347) > org.eclipse.osgi.container.Module.stop(Module.java:513) > > org.eclipse.osgi.internal.framework.EquinoxBundle.stop(EquinoxBundle.java:467) > > org.apache.karaf.features.internal.service.BundleInstallSupportImpl.stopBundle(BundleInstallSupportImpl.java:171) > > org.apache.karaf.features.internal.service.FeaturesServiceImpl.stopBundle(FeaturesServiceImpl.java:1165) > > org.apache.karaf.features.internal.service.Deployer.deploy(Deployer.java:1000) > > org.apache.karaf.features.internal.service.FeaturesServiceImpl.doProvision(FeaturesServiceImpl.java:1069) > > org.apache.karaf.features.internal.service.FeaturesServiceImpl.lambda$doProvisionInThread$13(FeaturesServiceImpl.java:1004) > > org.apache.karaf.features.internal.service.FeaturesServiceImpl$$Lambda$228/0x00000001004c2c40.call(Unknown > Source) > java.base@11.0.10/java.util.concurrent.FutureTask.run(FutureTask.java:264) > > java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > java.base@11.0.10/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > java.base@11.0.10/java.lang.Thread.run(Thread.java:834) > {noformat} > -- This message was sent by Atlassian Jira (v8.3.4#803005)