The last bit of Java code before entering the JNI code is here [1]. The JNI code [2] attempts to invoke io_queue_init [3] from libaio which results in the "Resource temporarily unavailable" error. So it appears there's something environmental which is preventing libaio from doing its work. My searching indicates you can run "cat /proc/sys/fs/aio-max-nr" to inspect the maximum number of allowable concurrent requests and potentially increase this value to resolve the issue.
Justin [1] https://github.com/apache/activemq-artemis-native/blob/master/src/main/java/org/apache/activemq/artemis/nativo/jlibaio/LibaioContext.java#L161 [2] https://github.com/apache/activemq-artemis-native/blob/master/src/main/c/org_apache_activemq_artemis_nativo_jlibaio_LibaioContext.c#L528 [3] https://manpages.debian.org/testing/libaio-dev/io_queue_init.2.en.html On Fri, Nov 22, 2019 at 8:12 AM Ehsan Zaery Moghaddam < zaerymoghad...@gmail.com> wrote: > Hi > > We're using WildFly 18 which is using ActiveMQ 2.10.1. Since a few days > ago, we're getting the following exception whenever we try to start the > WildFly on some of our servers. It seems to be on some random servers, but > those that have this issue, continuously throw the same exception. It's > from the JNI part of the artemis component. > > We tried running a fresh installation of WildFly and even tried to use the > older version that we had before (WildFly 13) without deploying any > application but nothing changed. So it shouldn't be related to the ActiveMQ > or WildFly but it's also not clear for us what that part of the code is > trying to do with file system. So I thought maybe you guys can help us in > this regard. > > The artemis component is using libaio linux library (as mentioned in their > GitHub page). So we checked if we had any update on that library on given > servers and we find out that it didn't get any update since few months ago. > > Any help would be really appreciated. > > WildFly 18 startup exception: > > 2019-11-21 12:15:47,076 INFO [ServerService Thread > Pool -- 72] [server] AMQ221000: live Message Broker is starting with > configuration Broker Configuration > > (clustered=true,journalDirectory=/opt/wildfly/standalone/data/activemq/journal,bindingsDirectory=/opt/wildfly/standalone/data/activemq/bindings,largeMessagesDirectory=/opt/wildfly/standalone/data/active > > mq/largemessages,pagingDirectory=/opt/wildfly/standalone/data/activemq/paging) > 2019-11-21 12:15:47,145 INFO [ServerService Thread > Pool -- 72] [server] AMQ221012: Using AIO Journal > 2019-11-21 12:15:47,209 INFO [MSC service thread > 1-3] [GlobalComponentRegistry] ISPN000128: Infinispan version: Infinispan > 'Infinity Minus ONE +2' 9.4.16.Final > 2019-11-21 12:15:47,283 INFO [ServerService Thread > Pool -- 72] [server] AMQ221043: Protocol module found: [artemis-server]. > Adding protocol support for: CORE > 2019-11-21 12:15:47,284 INFO [ServerService Thread > Pool -- 72] [server] AMQ221043: Protocol module found: > [artemis-amqp-protocol]. Adding protocol support for: AMQP > 2019-11-21 12:15:47,284 INFO [ServerService Thread > Pool -- 72] [server] AMQ221043: Protocol module found: > [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ > 2019-11-21 12:15:47,285 INFO [ServerService Thread > Pool -- 72] [server] AMQ221043: Protocol module found: > [artemis-stomp-protocol]. Adding protocol support for: STOMP > 2019-11-21 12:15:47,361 ERROR [ServerService Thread > Pool -- 72] [server] AMQ224000: Failure in initialisation: > java.lang.RuntimeException: Cannot initialize queue:Resource temporarily > unavailable > at > org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.newContext(Native > Method) > at > > org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.<init>(LibaioContext.java:161) > at > > org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory.start(AIOSequentialFileFactory.java:260) > at > > org.apache.activemq.artemis.core.journal.impl.JournalImpl.start(JournalImpl.java:2456) > at > > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.start(AbstractJournalStorageManager.java:1586) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:2824) > at > > org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:64) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:600) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:527) > at > > org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:373) > at > > org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:211) > at > > org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) > at > > org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > at > > org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) > at > > org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) > at > > org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) > at java.lang.Thread.run(Thread.java:748) > at org.jboss.threads.JBossThread.run(JBossThread.java:485) > > 2019-11-21 12:15:47,365 ERROR [ServerService Thread > Pool -- 72] [fail] MSC000001: Failed to start service > jboss.messaging-activemq.default.jms.manager: > org.jboss.msc.service.StartException in service > jboss.messaging-activemq.default.jms.manager: java.lang.RuntimeException: > Cannot initialize queue:Resource temporarily unavailable > at > > org.wildfly.extension.messaging.activemq.jms.JMSService.lambda$doStart$0(JMSService.java:147) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callActivationFailureListeners(ActiveMQServerImpl.java:2224) > at > > org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:83) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:600) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:527) > at > > org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:373) > at > > org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:211) > at > > org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) > at > > org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) > at > > org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) > at > > org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) > at > > org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) > at java.lang.Thread.run(Thread.java:748) > at org.jboss.threads.JBossThread.run(JBossThread.java:485) > Caused by: java.lang.RuntimeException: Cannot initialize queue:Resource > temporarily unavailable > at > org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.newContext(Native > Method) > at > > org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.<init>(LibaioContext.java:161) > at > > org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory.start(AIOSequentialFileFactory.java:260) > at > > org.apache.activemq.artemis.core.journal.impl.JournalImpl.start(JournalImpl.java:2456) > at > > org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.start(AbstractJournalStorageManager.java:1586) > at > > org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:2824) > at > > org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:64) > ... 14 more > > Regards > Ehsan >
