One more thing you can try... The default value for <journal-max-io> is 4096 so you could specify something smaller than that, e.g. <journal-max-io>2048</journal-max-io>. This value is used in the call to io_queue_init.
Justin On Fri, Nov 22, 2019 at 11:06 AM Justin Bertram <jbert...@apache.org> wrote: > You could also work-around this issue by specifying > <journal-type>NIO</journal-type> in your broker.xml. The <journal-type> > defaults to ASYNCIO which will use the native libaio for best performance. > > > Justin > > On Fri, Nov 22, 2019 at 11:04 AM Justin Bertram <jbert...@apache.org> > wrote: > >> The last bit of Java code before entering the JNI code is here [1]. The >> JNI code [2] attempts to invoke io_queue_init [3] from libaio which results >> in the "Resource temporarily unavailable" error. So it appears there's >> something environmental which is preventing libaio from doing its work. My >> searching indicates you can run "cat /proc/sys/fs/aio-max-nr" to inspect >> the maximum number of allowable concurrent requests and potentially >> increase this value to resolve the issue. >> >> >> Justin >> >> [1] >> https://github.com/apache/activemq-artemis-native/blob/master/src/main/java/org/apache/activemq/artemis/nativo/jlibaio/LibaioContext.java#L161 >> [2] >> https://github.com/apache/activemq-artemis-native/blob/master/src/main/c/org_apache_activemq_artemis_nativo_jlibaio_LibaioContext.c#L528 >> [3] >> https://manpages.debian.org/testing/libaio-dev/io_queue_init.2.en.html >> >> On Fri, Nov 22, 2019 at 8:12 AM Ehsan Zaery Moghaddam < >> zaerymoghad...@gmail.com> wrote: >> >>> Hi >>> >>> We're using WildFly 18 which is using ActiveMQ 2.10.1. Since a few days >>> ago, we're getting the following exception whenever we try to start the >>> WildFly on some of our servers. It seems to be on some random servers, >>> but >>> those that have this issue, continuously throw the same exception. It's >>> from the JNI part of the artemis component. >>> >>> We tried running a fresh installation of WildFly and even tried to use >>> the >>> older version that we had before (WildFly 13) without deploying any >>> application but nothing changed. So it shouldn't be related to the >>> ActiveMQ >>> or WildFly but it's also not clear for us what that part of the code is >>> trying to do with file system. So I thought maybe you guys can help us in >>> this regard. >>> >>> The artemis component is using libaio linux library (as mentioned in >>> their >>> GitHub page). So we checked if we had any update on that library on given >>> servers and we find out that it didn't get any update since few months >>> ago. >>> >>> Any help would be really appreciated. >>> >>> WildFly 18 startup exception: >>> >>> 2019-11-21 12:15:47,076 INFO [ServerService >>> Thread >>> Pool -- 72] [server] AMQ221000: live Message Broker is starting with >>> configuration Broker Configuration >>> >>> (clustered=true,journalDirectory=/opt/wildfly/standalone/data/activemq/journal,bindingsDirectory=/opt/wildfly/standalone/data/activemq/bindings,largeMessagesDirectory=/opt/wildfly/standalone/data/active >>> >>> mq/largemessages,pagingDirectory=/opt/wildfly/standalone/data/activemq/paging) >>> 2019-11-21 12:15:47,145 INFO [ServerService >>> Thread >>> Pool -- 72] [server] AMQ221012: Using AIO Journal >>> 2019-11-21 12:15:47,209 INFO [MSC service thread >>> 1-3] [GlobalComponentRegistry] ISPN000128: Infinispan version: Infinispan >>> 'Infinity Minus ONE +2' 9.4.16.Final >>> 2019-11-21 12:15:47,283 INFO [ServerService >>> Thread >>> Pool -- 72] [server] AMQ221043: Protocol module found: [artemis-server]. >>> Adding protocol support for: CORE >>> 2019-11-21 12:15:47,284 INFO [ServerService >>> Thread >>> Pool -- 72] [server] AMQ221043: Protocol module found: >>> [artemis-amqp-protocol]. Adding protocol support for: AMQP >>> 2019-11-21 12:15:47,284 INFO [ServerService >>> Thread >>> Pool -- 72] [server] AMQ221043: Protocol module found: >>> [artemis-hornetq-protocol]. Adding protocol support for: HORNETQ >>> 2019-11-21 12:15:47,285 INFO [ServerService >>> Thread >>> Pool -- 72] [server] AMQ221043: Protocol module found: >>> [artemis-stomp-protocol]. Adding protocol support for: STOMP >>> 2019-11-21 12:15:47,361 ERROR [ServerService >>> Thread >>> Pool -- 72] [server] AMQ224000: Failure in initialisation: >>> java.lang.RuntimeException: Cannot initialize queue:Resource temporarily >>> unavailable >>> at >>> >>> org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.newContext(Native >>> Method) >>> at >>> >>> org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.<init>(LibaioContext.java:161) >>> at >>> >>> org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory.start(AIOSequentialFileFactory.java:260) >>> at >>> >>> org.apache.activemq.artemis.core.journal.impl.JournalImpl.start(JournalImpl.java:2456) >>> at >>> >>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.start(AbstractJournalStorageManager.java:1586) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:2824) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:64) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:600) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:527) >>> at >>> >>> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:373) >>> at >>> >>> org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:211) >>> at >>> >>> org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) >>> at >>> >>> org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> >>> org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) >>> at >>> >>> org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) >>> at >>> >>> org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) >>> at >>> >>> org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) >>> at java.lang.Thread.run(Thread.java:748) >>> at org.jboss.threads.JBossThread.run(JBossThread.java:485) >>> >>> 2019-11-21 12:15:47,365 ERROR [ServerService >>> Thread >>> Pool -- 72] [fail] MSC000001: Failed to start service >>> jboss.messaging-activemq.default.jms.manager: >>> org.jboss.msc.service.StartException in service >>> jboss.messaging-activemq.default.jms.manager: java.lang.RuntimeException: >>> Cannot initialize queue:Resource temporarily unavailable >>> at >>> >>> org.wildfly.extension.messaging.activemq.jms.JMSService.lambda$doStart$0(JMSService.java:147) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.callActivationFailureListeners(ActiveMQServerImpl.java:2224) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:83) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.internalStart(ActiveMQServerImpl.java:600) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.start(ActiveMQServerImpl.java:527) >>> at >>> >>> org.apache.activemq.artemis.jms.server.impl.JMSServerManagerImpl.start(JMSServerManagerImpl.java:373) >>> at >>> >>> org.wildfly.extension.messaging.activemq.jms.JMSService.doStart(JMSService.java:211) >>> at >>> >>> org.wildfly.extension.messaging.activemq.jms.JMSService.access$000(JMSService.java:65) >>> at >>> >>> org.wildfly.extension.messaging.activemq.jms.JMSService$1.run(JMSService.java:100) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> >>> org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35) >>> at >>> >>> org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1982) >>> at >>> >>> org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486) >>> at >>> >>> org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377) >>> at java.lang.Thread.run(Thread.java:748) >>> at org.jboss.threads.JBossThread.run(JBossThread.java:485) >>> Caused by: java.lang.RuntimeException: Cannot initialize queue:Resource >>> temporarily unavailable >>> at >>> >>> org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.newContext(Native >>> Method) >>> at >>> >>> org.apache.activemq.artemis.nativo.jlibaio.LibaioContext.<init>(LibaioContext.java:161) >>> at >>> >>> org.apache.activemq.artemis.core.io.aio.AIOSequentialFileFactory.start(AIOSequentialFileFactory.java:260) >>> at >>> >>> org.apache.activemq.artemis.core.journal.impl.JournalImpl.start(JournalImpl.java:2456) >>> at >>> >>> org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.start(AbstractJournalStorageManager.java:1586) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.initialisePart1(ActiveMQServerImpl.java:2824) >>> at >>> >>> org.apache.activemq.artemis.core.server.impl.LiveOnlyActivation.run(LiveOnlyActivation.java:64) >>> ... 14 more >>> >>> Regards >>> Ehsan >>> >>
