Re: [akka-user] Cluster failure in single JVM
Hi James, independent of what the actual problem is, there should be no difference in expectations concerning running physically distributed or on the same node, since the most frequent problem we observe is that load is applied without control (flow control, bounded queues, etc.) leading to long GC pauses, out-of-memory, overwhelmed network connections (even on the local host), etc. Keeping your systems responsive is not something that can be magically achieved at the runtime or library level. Regards, Roland 22 maj 2014 kl. 16:42 skrev James Bunch james.x.bu...@gmail.com: Can anyone point in the right direction with this one, probably something obvious I'm missing! I'm trying to debug a strange Cluster issue… I have 3 actor systems in a cluster (all in the same JVM) communicating via the distributed bus. (It's just a test app at the moment, and in production will run with each actor system in its own JVM on different servers). Akka v2.3.0 It works fine on my local machine, and on the dev server. On the test server, it works fine until put under load. At that point actor system 2 seems to disappear – it suddenly doesn’t send or receive any heartbeats and is quickly marked as unreachable (I'm basing this on the log output switched on by log-remote-lifecycle-events) No idea why! I was expecting to see this in networked setup (say on EC2) but not within a single app. I've tried changing the failure detection config settings based on blindly copying in config from other posts here, but it doesn’t affect the problem. Can anyone offer any suggestions? -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. Dr. Roland Kuhn Akka Tech Lead Typesafe – Reactive apps on the JVM. twitter: @rolandkuhn -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
[akka-user] scheduled events deadlocking
Hello, I'm trying to set up a simulation where two devices (FSMs) cooperate. The two devices exchange messages, and each is guided by an instance of another actor to drive the usage pattern. The devices have a timer to change behaviour (attempt to exchange a bit of info with the other device), and the usagePattern actors have a timer to simulate the user behaviour. The device timer is scheduled by: context.system.scheduler.schedule(delay milliseconds, delay milliseconds, self, MyMessage()) and the usagePattern is repeatedly scheduling by: context.system.scheduler.scheduleOnce(123 milliseconds, self, SendNext()) from within the message receive My problem is that this deadlocks fairly quickly, seemingly when the two Device instances tries to send messages to each other (all sends are done with ! ). Does anyone have any thoughts on what is causing this, the jvm stacktrace is below: Found one Java-level deadlock: = ForkJoinPool-2-worker-7: waiting for ownable synchronizer 0x0007eeb94470, (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by ForkJoinPool-2-worker-11 ForkJoinPool-2-worker-11: waiting for ownable synchronizer 0x0007eef60c40, (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by ForkJoinPool-2-worker-9 ForkJoinPool-2-worker-9: waiting for ownable synchronizer 0x0007eeb94470, (a java.util.concurrent.locks.ReentrantLock$NonfairSync), which is held by ForkJoinPool-2-worker-11 Java stack information for the threads listed above: === ForkJoinPool-2-worker-7: at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0007eeb94470 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:929) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1245) at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:445) at akka.testkit.CallingThreadDispatcher.runQueue(CallingThreadDispatcher.scala:275) at akka.testkit.CallingThreadDispatcher.dispatch(CallingThreadDispatcher.scala:208) at akka.actor.dungeon.Dispatch$class.sendMessage(Dispatch.scala:123) at akka.actor.ActorCell.sendMessage(ActorCell.scala:369) at akka.actor.Cell$class.sendMessage(ActorCell.scala:290) at akka.actor.ActorCell.sendMessage(ActorCell.scala:369) at akka.actor.LocalActorRef.$bang(ActorRef.scala:384) at agents.UsagePattern.handleSendNext(UsagePattern.scala:68) at agents.UsagePattern$$anonfun$receive$1.applyOrElse(UsagePattern.scala:55) at akka.actor.Actor$class.aroundReceive(Actor.scala:465) at agents.UsagePattern.aroundReceive(UsagePattern.scala:21) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) at akka.actor.ActorCell.invoke(ActorCell.scala:487) at akka.testkit.CallingThreadDispatcher.process$1(CallingThreadDispatcher.scala:251) at akka.testkit.CallingThreadDispatcher.runQueue(CallingThreadDispatcher.scala:284) at akka.testkit.CallingThreadDispatcher.dispatch(CallingThreadDispatcher.scala:208) at akka.actor.dungeon.Dispatch$class.sendMessage(Dispatch.scala:123) at akka.actor.ActorCell.sendMessage(ActorCell.scala:369) at akka.actor.Cell$class.sendMessage(ActorCell.scala:290) at akka.actor.ActorCell.sendMessage(ActorCell.scala:369) at akka.actor.LocalActorRef.$bang(ActorRef.scala:384) at akka.actor.Scheduler$$anon$6.run(Scheduler.scala:106) at scala.concurrent.impl.ExecutionContextImpl$AdaptedForkJoinTask.exec(ExecutionContextImpl.scala:121) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) ForkJoinPool-2-worker-11: at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0007eef60c40 (a java.util.concurrent.locks.ReentrantLock$NonfairSync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:929) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1245) at java.util.concurrent.locks.ReentrantLock.tryLock(ReentrantLock.java:445) at akka.testkit.CallingThreadDispatcher.runQueue(CallingThreadDispatcher.scala:275) at akka.testkit.CallingThreadDispatcher.dispatch(CallingThreadDispatcher.scala:208) at akka.actor.dungeon.Dispatch$class.sendMessage(Dispatch.scala:123) at akka.actor.ActorCell.sendMessage(ActorCell.scala:369) at akka.actor.Cell$class.sendMessage(ActorCell.scala:290) at akka.actor.ActorCell.sendMessage(ActorCell.scala:369) at
Re: [akka-user] scheduled events deadlocking
Hello Oskar, based on the two key datapoints from your email: FSM testing + at akka.testkit.CallingThreadDispatcher.dispatch(CallingThreadDispatcher.scala:208) I deduce that you are using TestActorRef (well you’re probably using TestFSMRef as suggested by the FSM docs, which also IS-A TestActorRef), which is cool for testing one actor in isolation, yet it won’t work nicely when you test interaction between actors, because it’s using the CallingThreadDispatcher. These abstractions allow to very easily test “one actor”, yet at the trade of of effectively being single threaded (sic!), which is why you’re running into deadlocks. So while these patterns and classes are good for synchronous testing, they won’t play well if you have any kind of bigger interaction going on between actors. Please use the plain TestKit instead, like explained in the “Asynchronous Integration Testing with TestKit section of the docs. I hope this helps, happy hakking! -- Konrad 'ktoso' Malawski hAkker @ typesafe -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
