Re: [akka-user] Apparent thread leak with non-default akka dispatcher

[Viktor Klang]

Great story, Jesse. Thanks for sharing!

On Wed, Mar 23, 2016 at 3:12 PM, Jesse C  wrote:

> Here is what we've figured out. We've got a native library that contains a
> thread local member that includes a ZMQ socket. That socket is apparently
> not being correctly cleaned-up on thread death. ZMQ sockets are created
> from a ZMQ context, which creates 2 helper threads upon construction. Those
> are the threads we were leaking.
>
> The actor in Akka was invoking this native library. In our old code, using
> Scala's now-deprecated, built-in actor library, we had locked that actor to
> a single thread, so there was never any thread leakage. In Akka, we were
> using a Pinned dispatcher, with the understanding that that would result in
> the same behavior. The scalaDocs say "Dedicates a unique thread for each
> actor passed in as reference."
>
> The akka manual also adds "Note that it's not guaranteed that the *same*
> thread is used over time, since the core pool timeout is used for
> PinnedDispatcher to keep resource usage down in case of idle actors. To
> use the same thread all the time you need to add
> thread-pool-executor.allow-core-timeout=off to the configuration of the
> PinnedDispatcher."
>
> We missed that part. It might help others to have that added to the
> scaladocs.
>
> The solution that we've found to work is to set the allow-core-timeout to
> off and the thread-pool-executor.keep-alive-time to a really high number.
> With those settings we've now got the behavior we thought we were
> originally getting, where the actor is locked into a single thread for its
> entire lifetime.
>
> We're also looking at fixing the thread leak issue in the native library.
>
> Thanks for the responses.
>
> cheers
> jesse c
>
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Cheers,
√

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Apparent thread leak with non-default akka dispatcher

[Jesse C]

Here is what we've figured out. We've got a native library that contains a 
thread local member that includes a ZMQ socket. That socket is apparently 
not being correctly cleaned-up on thread death. ZMQ sockets are created 
from a ZMQ context, which creates 2 helper threads upon construction. Those 
are the threads we were leaking. 

The actor in Akka was invoking this native library. In our old code, using 
Scala's now-deprecated, built-in actor library, we had locked that actor to 
a single thread, so there was never any thread leakage. In Akka, we were 
using a Pinned dispatcher, with the understanding that that would result in 
the same behavior. The scalaDocs say "Dedicates a unique thread for each 
actor passed in as reference."  

The akka manual also adds "Note that it's not guaranteed that the *same* 
thread is used over time, since the core pool timeout is used for 
PinnedDispatcher to keep resource usage down in case of idle actors. To use 
the same thread all the time you need to add 
thread-pool-executor.allow-core-timeout=off to the configuration of the 
PinnedDispatcher."

We missed that part. It might help others to have that added to the 
scaladocs.

The solution that we've found to work is to set the allow-core-timeout to 
off and the thread-pool-executor.keep-alive-time to a really high number. 
With those settings we've now got the behavior we thought we were 
originally getting, where the actor is locked into a single thread for its 
entire lifetime.

We're also looking at fixing the thread leak issue in the native library.

Thanks for the responses.

cheers
jesse c


-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Apparent thread leak with non-default akka dispatcher

[Roland Kuhn]

Hi Jesse,

the prime suspect for threads that are known to the O/S but not to the JVM is 
any sort of native library—Akka does not do anything of this kind.

Regards,

Roland

> 21 mar 2016 kl. 15:12 skrev Jesse C :
> 
> the 600 threads do not show up in JStack. JStack shows the expected number of 
> threads. However, looking in the 'task' folder for the process in the '/proc' 
> directory shows all 600 or so, which is why this feels like some sort of 
> leak. The thread is being cleaned up in Java, but not at the OS level. I was 
> wondering if it was a garbage collection issue, but I used jcmd to force a 
> full collection for the process and that did not result in any of the native 
> threads being removed.
> 
> 
> -- 
> >> Read the docs: http://akka.io/docs/ 
> >> Check the FAQ: 
> >> http://doc.akka.io/docs/akka/current/additional/faq.html 
> >> 
> >> Search the archives: https://groups.google.com/group/akka-user 
> >> 
> --- 
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+unsubscr...@googlegroups.com 
> .
> To post to this group, send email to akka-user@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/akka-user 
> .
> For more options, visit https://groups.google.com/d/optout 
> .



Dr. Roland Kuhn
Akka Tech Lead
Typesafe  – Reactive apps on the JVM.
twitter: @rolandkuhn
 

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Apparent thread leak with non-default akka dispatcher

[Jesse C]

the 600 threads do not show up in JStack. JStack shows the expected number 
of threads. However, looking in the 'task' folder for the process in the 
'/proc' directory shows all 600 or so, which is why this feels like some 
sort of leak. The thread is being cleaned up in Java, but not at the OS 
level. I was wondering if it was a garbage collection issue, but I used 
jcmd to force a full collection for the process and that did not result in 
any of the native threads being removed.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Apparent thread leak with non-default akka dispatcher

[Patrik Nordwall]

Do you say that you have 600 live threads? Have you verified that with
jstack or a profiler?

The count doesn't mean that the threads are alive. Threads are stopped when
idle for some time. E.g. the fork-join pool has an idle timeout of around 2
seconds. For fun you can try to schedule something every second instead of
1.5.minutes.

The thread-pool-executor has a allow-core-timeout setting that you can try
to set to off.

PinnedDispatcher with fixed-pool-size = 2 does not make sense.

On Sat, Mar 19, 2016 at 2:47 AM, Jesse C  wrote:

> I've got an application that is using akka and I'm seeing some really
> strange behavior. I'm assuming this is something dumb that I'm doing in my
> configuration.
>
> The setup is
> CentOS 6.7
> Java 1.8.0_71-b15
> Scala 2.11.7
> Akka 2.4.2
>
> Here is what I'm seeing. First noticed the issue because the application
> hit the system thread limit today and crashed (CentOS defaults to 1024
> processes/threads per user). That seemed very odd to me since it is pretty
> lightweight and there is only one actor in the application. So I started
> monitoring it.  From the Java side, it appears that nothing is amiss. Using
> jstack I see the expected number of threads and nothing grows over time.
> However, watching the process on the OS side, I see a fairly rapidly
> growing number of threads being created. After about 8 hours, I'm up to
> around 600 native threads. It seems like threads are being created and not
> being destroyed, but Java is losing track of them somehow.
>
> I've got some other akka applications with the same basic stack that are
> not exhibiting this behavior. That leads me to conclude that either my
> configuration on this particular service is just wrong or my configuration
> is tickling a bug somewhere.
>
> Here is what seems to be the relevant config pieces:
>
> I'm creating a config for my actor system so that it will use a really
> basic custom mailbox I wrote to make sure messages get ordered:
>   private[this]
>   val actorConfig = ConfigFactory.parseString("""
> prio-dispatcher {
>   mailbox-type = "com.company.server.PriorityMailbox"
> }
>   """).withFallback(conf)
>
> Then I make my actor system with this config
>
>  private[this]
>  val system = ActorSystem("mySystem", actorConfig)
>
>
>
> Mailbox is here:
> class PriorityMailbox(settings: ActorSystem.Settings, config: Config)
>   extends UnboundedStablePriorityMailbox(PriorityMailbox.priority)
>
> I create my actor with:
>
> private[this]
> val priorityActor = system.actorOf(PriorityActor.props(db).withDispatcher(
> "prio-dispatcher"))
>
> The only other thing I do that in anyway really interacts with the actor
> system at a thread/scheduling level is in the preStart for the
> priorityActor, where I set up a schedule to send a message:
>
>   override
>   def preStart()
>   {
> import context.dispatcher
> context.system.scheduler.schedule(1.5 minutes, 1.5 minutes, self,
> NEWCALLBACK)
>   }
>
> And that's basically it. And yet, I get what appears to be an unbounded
> number of OS threads over time.
>
> I tried tweaking my actorConfig to a fixed-sized thread pool
>
>   private[this]
>   val actorConfig = ConfigFactory.parseString("""
> prio-dispatcher {
>   mailbox-type = "com.company.server.PriorityMailbox"
>   type = PinnedDispatcher
>   executor = "thread-pool-executor"
>   thread-pool-executor {
> fixed-pool-size = 2
>   }
>   throughput = 1
> }
>   """).withFallback(conf)
>
> And that did not change anything.
>
> So here is what else I've been noticing. When I use jstack to print out
> the current threads I usually see something like this at the top
> "mySystem-prio-dispatcher-395" #3305 prio=5 os_prio=0
> tid=0x7f64e1642000 nid=0x6c49 waiting on condition [0x7f6668fd3000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x80177350> (a akka.dispatch.
> ForkJoinExecutorConfigurator$AkkaForkJoinPool)
> at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:
> 2075)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.
> java:1979)
> at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
> ForkJoinWorkerThread.java:107)
>
>
> This isn't unexpected, since you expect a ForkJoinPool to clean-up old
> threads and spawn new ones over time. But it does appear to be the only
> thing in the system that is actually spawning new threads. Which would seem
> to indicate to me that this the most likely source of the thread leak. And
> that number after the thread, which I'm pretty sure is basically a running
> count of the total number of threads that have been made in the process,
> keeps going up.
>
>
> Anyone have any thoughts on what I might be doing wrong or places I can
> look for more information.
>
> cheers,
> Jesse C
>
>
> --
> >> Read the docs: http://akka.io/docs/
> 

[akka-user] Apparent thread leak with non-default akka dispatcher

[Jesse C]

I've got an application that is using akka and I'm seeing some really 
strange behavior. I'm assuming this is something dumb that I'm doing in my 
configuration.

The setup is
CentOS 6.7
Java 1.8.0_71-b15
Scala 2.11.7
Akka 2.4.2

Here is what I'm seeing. First noticed the issue because the application 
hit the system thread limit today and crashed (CentOS defaults to 1024 
processes/threads per user). That seemed very odd to me since it is pretty 
lightweight and there is only one actor in the application. So I started 
monitoring it.  From the Java side, it appears that nothing is amiss. Using 
jstack I see the expected number of threads and nothing grows over time. 
However, watching the process on the OS side, I see a fairly rapidly 
growing number of threads being created. After about 8 hours, I'm up to 
around 600 native threads. It seems like threads are being created and not 
being destroyed, but Java is losing track of them somehow.

I've got some other akka applications with the same basic stack that are 
not exhibiting this behavior. That leads me to conclude that either my 
configuration on this particular service is just wrong or my configuration 
is tickling a bug somewhere.

Here is what seems to be the relevant config pieces:

I'm creating a config for my actor system so that it will use a really 
basic custom mailbox I wrote to make sure messages get ordered:
  private[this]
  val actorConfig = ConfigFactory.parseString("""
prio-dispatcher {
  mailbox-type = "com.company.server.PriorityMailbox"
}
  """).withFallback(conf)

Then I make my actor system with this config

 private[this]
 val system = ActorSystem("mySystem", actorConfig)



Mailbox is here:
class PriorityMailbox(settings: ActorSystem.Settings, config: Config)
  extends UnboundedStablePriorityMailbox(PriorityMailbox.priority)

I create my actor with:

private[this]
val priorityActor = system.actorOf(PriorityActor.props(db).withDispatcher(
"prio-dispatcher"))

The only other thing I do that in anyway really interacts with the actor 
system at a thread/scheduling level is in the preStart for the 
priorityActor, where I set up a schedule to send a message:

  override
  def preStart()
  {
import context.dispatcher
context.system.scheduler.schedule(1.5 minutes, 1.5 minutes, self, 
NEWCALLBACK)
  }

And that's basically it. And yet, I get what appears to be an unbounded 
number of OS threads over time.

I tried tweaking my actorConfig to a fixed-sized thread pool

  private[this]
  val actorConfig = ConfigFactory.parseString("""
prio-dispatcher {
  mailbox-type = "com.company.server.PriorityMailbox"
  type = PinnedDispatcher
  executor = "thread-pool-executor"
  thread-pool-executor {
fixed-pool-size = 2
  }
  throughput = 1
}
  """).withFallback(conf)

And that did not change anything.

So here is what else I've been noticing. When I use jstack to print out the 
current threads I usually see something like this at the top
"mySystem-prio-dispatcher-395" #3305 prio=5 os_prio=0 
tid=0x7f64e1642000 nid=0x6c49 waiting on condition [0x7f6668fd3000]
   java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for  <0x80177350> (a akka.dispatch.
ForkJoinExecutorConfigurator$AkkaForkJoinPool)
at scala.concurrent.forkjoin.ForkJoinPool.scan(ForkJoinPool.java:
2075)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.
java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(
ForkJoinWorkerThread.java:107)


This isn't unexpected, since you expect a ForkJoinPool to clean-up old 
threads and spawn new ones over time. But it does appear to be the only 
thing in the system that is actually spawning new threads. Which would seem 
to indicate to me that this the most likely source of the thread leak. And 
that number after the thread, which I'm pretty sure is basically a running 
count of the total number of threads that have been made in the process, 
keeps going up.


Anyone have any thoughts on what I might be doing wrong or places I can 
look for more information.

cheers,
Jesse C


-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.