Ah, you need to share the Aeron driver among JVMs, don't start Aeron embedded per system JVM as it will exponentially suck all your shared memory, to calculate how to tune Media Driver so that it can fit all your JVMs see here:
https://github.com/akka/akka/issues/21923 HTH, Guido. On Wednesday, December 14, 2016 at 11:09:53 PM UTC, Allan Brighton wrote: > > I tested on Linux (Ubuntu-16.04) and akka-2.4.14 and managed to get the > same error, however it seems to happen only after another error: > > 14:54:18.034 ERROR akka.actor.RepointableActorRef -- Error in stage > [akka.remote.artery.InboundHandshake@6d1b82d5]: Insufficient usable storage > for new log of length=50332096 in /dev/shm (tm > pfs) > io.aeron.exceptions.RegistrationException: Insufficient usable storage for > new log of length=50332096 in /dev/shm (tmpfs) > at io.aeron.ClientConductor.onError(ClientConductor.java:264) > ~[io.aeron.aeron-client-1.0.4.jar:na] > at > io.aeron.DriverListenerAdapter.onMessage(DriverListenerAdapter.java:141) > ~[io.aeron.aeron-client-1.0.4.jar:na] > at > org.agrona.concurrent.broadcast.CopyBroadcastReceiver.receive(CopyBroadcastReceiver.java:99) > > ~[org.agrona.Agrona-0.9.0.jar:na] > at > io.aeron.DriverListenerAdapter.pollMessage(DriverListenerAdapter.java:59) > ~[io.aeron.aeron-client-1.0.4.jar:na] > at io.aeron.ClientConductor.doWork(ClientConductor.java:331) > ~[io.aeron.aeron-client-1.0.4.jar:na] > at io.aeron.ClientConductor.awaitResponse(ClientConductor.java:350) > ~[io.aeron.aeron-client-1.0.4.jar:na] > at > io.aeron.ClientConductor.addPublication(ClientConductor.java:166) > ~[io.aeron.aeron-client-1.0.4.jar:na] > at io.aeron.Aeron.addPublication(Aeron.java:178) > ~[io.aeron.aeron-client-1.0.4.jar:na] > > It seems to be a bug that I have seen reported for akka on Linux, where > files are not removed from shared memory: > > > du -sh /dev/shm > 16G /dev/shm > > I think this was reported here <https://github.com/akka/akka/issues/20779>. > This happens when I run tests on a large project with lots of actors. > /dev/shm fills up and eventually gives this error. > > > On Tuesday, December 13, 2016 at 4:44:25 PM UTC-8, Allan Brighton wrote: >> >> Using akka-2.4.11 on Ubuntu-16.04: If I pull the network plug for a few >> seconds while an actor is running I get the errors below. >> It looks like akka kills the ActorSystem. How do you prevent this or >> recover from it? >> (I think it was the same under 2.4.14 when I tested it) >> >> 16:16:57.508 ERROR a.r.a.ArteryTransport(akka://ConfigService) -- Aeron >> error: 24473 observations from 2016-12-13 16:16:51.027-0800 to 2016-12-13 >> 16:16:57.507-0800 for: >> .io.IOException: Invalid argument >> at sun.nio.ch.DatagramDispatcher.write0(Native Method) >> at sun.nio.ch.DatagramDispatcher.write(DatagramDispatcher.java:51) >> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) >> at sun.nio.ch.IOUtil.write(IOUtil.java:51) >> at >> sun.nio.ch.DatagramChannelImpl.write(DatagramChannelImpl.java:605) >> at >> io.aeron.driver.media.SendChannelEndpoint.send(SendChannelEndpoint.java:160) >> >> at >> io.aeron.driver.NetworkPublication.sendData(NetworkPublication.java:375) >> at >> io.aeron.driver.NetworkPublication.send(NetworkPublication.java:192) >> at io.aeron.driver.Sender.doSend(Sender.java:117) >> at io.aeron.driver.Sender.doWork(Sender.java:60) >> at >> org.agrona.concurrent.CompositeAgent.doWork(CompositeAgent.java:79) >> at org.agrona.concurrent.AgentRunner.run(AgentRunner.java:122) >> at java.lang.Thread.run(Thread.java:745) >> >> 16:16:58.251 ERROR a.r.a.ArteryTransport(akka://ConfigService) -- Fatal >> Aeron error DriverTimeoutException. Have to terminate ActorSystem because >> it lost contact with the embedded Aeron medi >> a driver. Possible configuration properties to mitigate the problem are >> 'client-liveness-timeout' or 'driver-timeout'. Driver has been inactive for >> over 20000ms >> io.aeron.exceptions.DriverTimeoutException: Driver has been inactive for >> over 20000ms >> at >> io.aeron.ClientConductor.checkDriverHeartbeat(ClientConductor.java:291) >> ~[io.aeron.aeron-client-1.0.1.jar:na] >> at >> io.aeron.ClientConductor.onCheckTimeouts(ClientConductor.java:378) >> ~[io.aeron.aeron-client-1.0.1.jar:na] >> at io.aeron.ClientConductor.doWork(ClientConductor.java:309) >> ~[io.aeron.aeron-client-1.0.1.jar:na] >> at io.aeron.ClientConductor.doWork(ClientConductor.java:123) >> ~[io.aeron.aeron-client-1.0.1.jar:na] >> at org.agrona.concurrent.AgentRunner.run(AgentRunner.java:122) >> ~[org.agrona.Agrona-0.5.4.jar:na] >> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111] >> 16:16:59.281 ERROR akka.actor.RepointableActorRef -- Error during >> postStop in [akka.remote.artery.AeronSink@53af6ad1]: Driver is inactive >> io.aeron.exceptions.DriverTimeoutException: Driver is inactive >> at >> io.aeron.ClientConductor.verifyDriverIsActive(ClientConductor.java:299) >> ~[io.aeron.aeron-client-1.0.1.jar:na] >> at >> io.aeron.ClientConductor.releasePublication(ClientConductor.java:151) >> ~[io.aeron.aeron-client-1.0.1.jar:na] >> at io.aeron.Publication.release(Publication.java:229) >> ~[io.aeron.aeron-client-1.0.1.jar:na] >> at io.aeron.Publication.close(Publication.java:206) >> ~[io.aeron.aeron-client-1.0.1.jar:na] >> at >> akka.remote.artery.AeronSink$$anon$1.postStop(AeronSink.scala:132) >> ~[com.typesafe.akka.akka-remote_2.11-2.4.11.jar:na] >> at >> akka.stream.impl.fusing.GraphInterpreter.finalizeStage(GraphInterpreter.scala:801) >> >> ~[com.typesafe.akka.akka-stream_2.11-2.4.11.jar:na] >> at >> akka.stream.impl.fusing.GraphInterpreter.afterStageHasRun(GraphInterpreter.scala:780) >> >> ~[com.typesafe.akka.akka-stream_2.11-2.4.11.jar:na] >> at >> akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:620) >> ~[com.typesafe.akka.akka-stream_2.11-2.4.11.jar:na] >> at >> akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:471) >> >> ~[com.typesafe.akka.akka-stream_2.11-2.4.11.jar:na] >> at >> akka.stream.impl.fusing.GraphInterpreterShell.receive(ActorGraphInterpreter.scala:423) >> >> ~[com.typesafe.akka.akka-stream_2.11-2.4.11.jar:na] >> at >> akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:603) >> >> ~[com.typesafe.akka.akka-stream_2.11-2.4 >> .11.jar:na] >> at >> akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:618) >> >> ~[com.typesafe.akka.akka-stream_2.11-2.4.11.jar:na] >> at akka.actor.Actor$class.aroundReceive(Actor.scala:484) >> ~[com.typesafe.akka.akka-actor_2.11-2.4.11.jar:na] >> at >> akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:529) >> >> ~[com.typesafe.akka.akka-stream_2.11-2.4.11.jar:na] >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526) >> [com.typesafe.akka.akka-actor_2.11-2.4.11.jar:na] >> at akka.actor.ActorCell.invoke(ActorCell.scala:495) >> [com.typesafe.akka.akka-actor_2.11-2.4.11.jar:na] >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257) >> [com.typesafe.akka.akka-actor_2.11-2.4.11.jar:na] >> at akka.dispatch.Mailbox.run(Mailbox.scala:224) >> [com.typesafe.akka.akka-actor_2.11-2.4.11.jar:na] >> at akka.dispatch.Mailbox.exec(Mailbox.scala:234) >> [com.typesafe.akka.akka-actor_2.11-2.4.11.jar:na] >> at >> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> [org.scala-lang.scala-library-2.11.8.jar:na] >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> >> [org.scala-lang.scala-library-2.11.8.jar:na] >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> [org.scala-lang.scala-library-2.11.8.jar:na] >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> [org.scala-lang.scala-library-2.11.8.jar:na] >> 16:16:59.285 ERROR akka.actor.RepointableActorRef -- Error during >> postStop in [akka.remote.artery.AeronSink@7a726acb]: Driver is inactive >> >> -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
