I've got an application that is using akka and I'm seeing some really strange behavior. I'm assuming this is something dumb that I'm doing in my configuration.
The setup is CentOS 6.7 Java 1.8.0_71-b15 Scala 2.11.7 Akka 2.4.2 Here is what I'm seeing. First noticed the issue because the application hit the system thread limit today and crashed (CentOS defaults to 1024 processes/threads per user). That seemed very odd to me since it is pretty lightweight and there is only one actor in the application. So I started monitoring it. From the Java side, it appears that nothing is amiss. Using jstack I see the expected number of threads and nothing grows over time. However, watching the process on the OS side, I see a fairly rapidly growing number of threads being created. After about 8 hours, I'm up to around 600 native threads. It seems like threads are being created and not being destroyed, but Java is losing track of them somehow. I've got some other akka applications with the same basic stack that are not exhibiting this behavior. That leads me to conclude that either my configuration on this particular service is just wrong or my configuration is tickling a bug somewhere. Here is what seems to be the relevant config pieces: I'm creating a config for my actor system so that it will use a really basic custom mailbox I wrote to make sure messages get ordered: private[this] val actorConfig = ConfigFactory.parseString(""" prio-dispatcher { mailbox-type = "com.company.server.PriorityMailbox" } """).withFallback(conf) Then I make my actor system with this config private[this] val system = ActorSystem("mySystem", actorConfig) Mailbox is here: class PriorityMailbox(settings: ActorSystem.Settings, config: Config) extends UnboundedStablePriorityMailbox(PriorityMailbox.priority) I create my actor with: private[this] val priorityActor = system.actorOf(PriorityActor.props(db).withDispatcher( "prio-dispatcher")) The only other thing I do that in anyway really interacts with the actor system at a thread/scheduling level is in the preStart for the priorityActor, where I set up a schedule to send a message: override def preStart() { import context.dispatcher context.system.scheduler.schedule(1.5 minutes, 1.5 minutes, self, NEWCALLBACK) } And that's basically it. And yet, I get what appears to be an unbounded number of OS threads over time. I tried tweaking my actorConfig to a fixed-sized thread pool private[this] val actorConfig = ConfigFactory.parseString(""" prio-dispatcher { mailbox-type = "com.company.server.PriorityMailbox" type = PinnedDispatcher executor = "thread-pool-executor" thread-pool-executor { fixed-pool-size = 2 } throughput = 1 } """).withFallback(conf) And that did not change anything. So here is what else I've been noticing. When I use jstack to print out the current threads I usually see something like this at the top "mySystem-prio-dispatcher-395" #3305 prio=5 os_prio=0 tid=0x00007f64e1642000 nid=0x6c49 waiting on condition [0x00007f6668fd3000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000080177350> (a akka.dispatch. ForkJoinExecutorConfigurator$AkkaForkJoinPool) at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java: 2075) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool. java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run( ForkJoinWorkerThread.java:107) This isn't unexpected, since you expect a ForkJoinPool to clean-up old threads and spawn new ones over time. But it does appear to be the only thing in the system that is actually spawning new threads. Which would seem to indicate to me that this the most likely source of the thread leak. And that number after the thread, which I'm pretty sure is basically a running count of the total number of threads that have been made in the process, keeps going up. Anyone have any thoughts on what I might be doing wrong or places I can look for more information. cheers, Jesse C -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.