Troubleshooting for /dev/shm issue is described here: https://github.com/real-logic/aeron#troubleshooting
Perhaps you are trying to run too much on a single machine (virtual machine, docker, or whatever) so that it's overloaded? If you have a constrained environment it might be worth trying Artery with TCP instead, see docs. /Patrik PS. Next time it might be better to ask at the new forum at https://discuss.akka.io/ On Tue, Apr 24, 2018 at 11:17 PM, Iman <ielgh...@gmail.com> wrote: > Hello, > I recently switched to using Artery from netty.tcp. > I ran into this error : Insufficient usable storage for new log of > length=50335744 in /dev/shm (tmpfs). > After searching for a solution, I understood that this is because I am > using the defaults to start the media driver, which will be inside the same > JVM of the actor. As described at AKKA documentation > <https://doc.akka.io/docs/akka/current/remoting-artery.html?language=scala#external-shared-aeron-media-driver>, > it is better to start the driver externally and have it shared among actors. > I followed the instructions in the documentation, I even used the same > example configuration. > > Now the problem is that my actors start and then after a little time they > all fail with an exception: > > [ERROR] [04/24/2018 20:49:49.606] [aeron-client-conductor] > [akka.remote.artery.aeron.ArteryAeronUdpTransport(akka://ActorClusterSystemName)] > Fatal Aeron error ConductorServiceTimeoutException. Have to terminate > ActorSystem because it lost contact with the external Aeron media driver. > Possible configuration properties to mitigate the problem are > 'client-liveness-timeout' or 'driver-timeout'. io.aeron.exceptions. > ConductorServiceTimeoutException: Exceeded (ns): 5000000000 > > io.aeron.exceptions.ConductorServiceTimeoutException: Exceeded (ns): > 5000000000 > > at io.aeron.ClientConductor.checkServiceInterval(ClientConductor.java:745) > > at io.aeron.ClientConductor.onCheckTimeouts(ClientConductor.java:720) > > at io.aeron.ClientConductor.service(ClientConductor.java:659) > > at io.aeron.ClientConductor.doWork(ClientConductor.java:151) > > at org.agrona.concurrent.AgentRunner.doDutyCycle(AgentRunner.java:233) > > at org.agrona.concurrent.AgentRunner.run(AgentRunner.java:159) > > at java.lang.Thread.run(Thread.java:748) > > > [ERROR] [04/24/2018 20:49:49.963] [aeron-client-conductor] > [akka.remote.artery.aeron.ArteryAeronUdpTransport(akka://ActorClusterSystemName)] > Aeron error, org.agrona.concurrent.AgentTerminationException > > org.agrona.concurrent.AgentTerminationException > > at io.aeron.ClientConductor.doWork(ClientConductor.java:148) > > at org.agrona.concurrent.AgentRunner.doDutyCycle(AgentRunner.java:233) > > at org.agrona.concurrent.AgentRunner.run(AgentRunner.java:159) > > at java.lang.Thread.run(Thread.java:748) > > > and then after many undelivered messages: > > [INFO] [04/24/2018 20:49:52.270] [ActorClusterSystemName-akka. > remote.default-remote-dispatcher-8] [akka://actorclustersystemn...@xx.xx. > xx.xx:2559/system/remoting-terminator] Remote daemon shut down; > proceeding with flushing remote transports. > > > Other actors in the cluster display this error message: > > [ERROR] [04/24/2018 20:52:27.468] > [ActorClusterSystemName-akka.actor.default-dispatcher-18] > [akka://actorclustersystemn...@xx.xx.xx.xx:2557/] swallowing exception > during message send > > java.lang.IllegalStateException: Aeron client conductor is closed > > at io.aeron.ClientConductor.ensureOpen(ClientConductor.java:635) > > at io.aeron.ClientConductor.addPublication(ClientConductor.java:367) > > at io.aeron.Aeron.addPublication(Aeron.java:247) > > at akka.remote.artery.aeron.AeronSink$$anon$1.<init>(AeronSink.scala:103) > > at akka.remote.artery.aeron.AeronSink.createLogicAndMaterializedValu > e(AeronSink.scala:100) > > at akka.stream.impl.GraphStageIsland.materializeAtomic( > PhasedFusingActorMaterializer.scala:630) > > at akka.stream.impl.PhasedFusingActorMaterializer.materialize( > PhasedFusingActorMaterializer.scala:450) > > at akka.stream.impl.PhasedFusingActorMaterializer.materialize( > PhasedFusingActorMaterializer.scala:415) > > at akka.stream.impl.PhasedFusingActorMaterializer.materialize( > PhasedFusingActorMaterializer.scala:406) > > at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:588) > > at akka.remote.artery.Association.runOutboundOrdinaryMessagesStr > eam(Association.scala:710) > > at akka.remote.artery.Association.$anonfun$runOutboundOrdinaryMessagesStr > eam$3(Association.scala:720) > > at akka.remote.artery.Association.$anonfun$attachOutboundStreamRestart$1( > Association.scala:814) > > at akka.remote.artery.Association$LazyQueueWrapper. > runMaterialize(Association.scala:89) > > at akka.remote.artery.Association$LazyQueueWrapper. > offer(Association.scala:93) > > at akka.remote.artery.Association$LazyQueueWrapper. > offer(Association.scala:84) > > at akka.remote.artery.Association.send(Association.scala:379) > > at akka.remote.artery.ArteryTransport.send(ArteryTransport.scala:714) > > at akka.remote.RemoteActorRef.$bang(RemoteActorRefProvider.scala:574) > > at akka.actor.ActorRef.tell(ActorRef.scala:124) > > at akka.actor.ActorSelection$.rec$1(ActorSelection.scala:265) > > at akka.actor.ActorSelection$.deliverSelection(ActorSelection.scala:269) > > at akka.actor.ActorSelection.tell(ActorSelection.scala:46) > > at akka.actor.ScalaActorSelection.$bang(ActorSelection.scala:280) > > at akka.actor.ScalaActorSelection.$bang$(ActorSelection.scala:280) > > at akka.actor.ActorSelection$$anon$1.$bang(ActorSelection.scala:198) > > at akka.cluster.ClusterCoreDaemon.gossipTo(ClusterDaemon.scala:1285) > > at akka.cluster.ClusterCoreDaemon.gossip(ClusterDaemon.scala:1009) > > at akka.cluster.ClusterCoreDaemon.gossipTick(ClusterDaemon.scala:972) > > at akka.cluster.ClusterCoreDaemon$$anonfun$initialized$1.applyOrElse( > ClusterDaemon.scala:484) > > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > > at akka.actor.Actor.aroundReceive(Actor.scala:517) > > at akka.actor.Actor.aroundReceive$(Actor.scala:515) > > at akka.cluster.ClusterCoreDaemon.aroundReceive(ClusterDaemon.scala:288) > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:588) > > at akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:557) > > at akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1) > > at org.aspectj.runtime.reflect.JoinPointImpl.proceed( > JoinPointImpl.java:149) > > at akka.kamon.instrumentation.ActorMonitors$$anon$1.$ > anonfun$processMessage$1(ActorMonitor.scala:123) > > at kamon.Kamon$.withContext(Kamon.scala:120) > > at akka.kamon.instrumentation.ActorMonitors$$anon$1. > processMessage(ActorMonitor.scala:123) > > at akka.kamon.instrumentation.ActorCellInstrumentation. > aroundBehaviourInvoke(ActorInstrumentation.scala:45) > > at akka.actor.ActorCell.invoke(ActorCell.scala:550) > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > > at kamon.executors.Executors$InstrumentedExecutorService$$ > anon$7.run(Executors.scala:270) > > at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec( > ForkJoinExecutorConfigurator.scala:44) > > at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) > > at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue. > runTask(ForkJoinPool.java:1339) > > at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) > > at akka.dispatch.forkjoin.ForkJoinWorkerThread.run( > ForkJoinWorkerThread.java:107) > > > > The media driver seem to be working fine! I checked the log files in the > shm directory and there are no errors in the loss-report.dat or other > files. > > I have increased both net.core.rmem_max and net.core.wmem_max to 4194304, > I also set java Xms to 1024M. I am running 7 actors in the cluster. > > This is my artery configuration: > > akka.remote { > log-remote-lifecycle-events = off > maximum-payload-bytes = 15 MiB > artery { > enabled = on > transport = aeron-udp > canonical.hostname = "127.0.0.1" > canonical.hostname = ${?HOST} > canonical.port = ${PORT} > advanced { > maximum-large-frame-size = 15 MiB > send-buffer-size = 15 MiB > receive-buffer-size = 15 MiB > maximum-frame-size = 15 MiB > outbound-message-queue-size = 2480000 > aeron-dir = /dev/shm/aeron > embedded-media-driver = off > } > > } > > } > > > AKKA version: 2.5.12. Scala version: 2.12.5. Linux distribution: debian > v9. I added aeron-driver-1.7.0.jar, aeron-client-1.7.0.jar, and > agrona-0.9.12.jar > to classpath when I start the external shared MediaDriver. > > I would appreciate if you can point me to how I can fix these errors. > Thanks > Iman > > -- > ************************************************************ > ***************************************** > ** New discussion forum: https://discuss.akka.io/ replacing akka-user > google-group soon. > ** This group will soon be put into read-only mode, and replaced by > discuss.akka.io > ** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io- > announced > ************************************************************ > ***************************************** > >>>>>>>>>> > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: http://doc.akka.io/docs/akka/ > current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > -- Patrik Nordwall Akka Tech Lead Lightbend <http://www.lightbend.com/> - Reactive apps on the JVM Twitter: @patriknw -- ***************************************************************************************************** ** New discussion forum: https://discuss.akka.io/ replacing akka-user google-group soon. ** This group will soon be put into read-only mode, and replaced by discuss.akka.io ** More details: https://akka.io/blog/news/2018/03/13/discuss.akka.io-announced ***************************************************************************************************** >>>>>>>>>> >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
