Hi,
are you running this application on Windows? There is a known situation
<https://github.com/akka/akka/issues/16942> where high CPU usage might
occur when using TCP with akka IO.
On Tue, Apr 14, 2015 at 7:28 AM, Akara Sucharitakul <
akara.sucharita...@gmail.com> wrote:
> Scala 2.11.6, Akka 2.3.9, Spray 1.3.2.
>
> When testing a particular workload, we ran into CPU spikes. The avg CPU
> load is less than 5% but about 6-7 minutes into the run, we start to see
> CPU spiking to near 100% lasting for several seconds. This repeats itself
> every 6-7 minutes. We can't correlate this with a particular GC activity
> just yet. We tested against Java 7_45 but tried Java 8 without much change.
>
> Spray is configured on the default-dispatcher (Spray default).
>
> Trying to capture thread dumps at the time spikes happening reveal 9
> threads spinning on the followings:
>
> "default-akka.actor.default-dispatcher-11" prio=10 tid=0x0000000001b2b800
> nid=0x3752 runnable [0x00007f1191d40000]
> java.lang.Thread.State: RUNNABLE
> at
> akka.io.DirectByteBufferPool.takeBufferFromPool(DirectByteBufferPool.scala:41)
> at akka.io.DirectByteBufferPool.acquire(DirectByteBufferPool.scala:31)
> at akka.io.TcpConnection.doRead(TcpConnection.scala:231)
> at
> akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(TcpConnection.scala:87)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
> at akka.io.TcpConnection.aroundReceive(TcpConnection.scala:28)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
> at akka.dispatch.Mailbox.run(Mailbox.scala:221)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> 2 threads show this spinning:
>
> "default-akka.actor.default-dispatcher-8" prio=10 tid=0x0000000001d6f000
> nid=0x374f runnable [0x00007f11928d2000]
> java.lang.Thread.State: RUNNABLE
> at
> akka.io.DirectByteBufferPool.offerBufferToPool(DirectByteBufferPool.scala:66)
> at akka.io.DirectByteBufferPool.release(DirectByteBufferPool.scala:34)
> at akka.io.TcpConnection.doRead(TcpConnection.scala:245)
> at
> akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(TcpConnection.scala:87)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
> at akka.io.TcpConnection.aroundReceive(TcpConnection.scala:28)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
> at akka.dispatch.Mailbox.run(Mailbox.scala:221)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> 1 thread is on I/O:
>
> "default-akka.actor.default-dispatcher-4" prio=10 tid=0x0000000001be2800
> nid=0x36ff runnable [0x00007f11924b0000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295)
> - locked <0x00000007d3b7e940> (a java.lang.Object)
> at akka.io.TcpConnection.innerRead$1(TcpConnection.scala:216)
> at akka.io.TcpConnection.doRead(TcpConnection.scala:232)
> at
> akka.io.TcpConnection$$anonfun$connected$1.applyOrElse(TcpConnection.scala:87)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
> at akka.io.TcpConnection.aroundReceive(TcpConnection.scala:28)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
> at akka.actor.ActorCell.invoke(ActorCell.scala:487)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254)
> at akka.dispatch.Mailbox.run(Mailbox.scala:221)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:231)
> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
> All in all, during this time there is no app logic being executed on the
> default dispatcher at all (12 threads on the default-dispatcher).
>
> For reference, here's the takeBufferFromPool implementation:
>
> @tailrec
> private final def takeBufferFromPool(): ByteBuffer =
> if (locked.compareAndSet(false, true)) {
> val buffer =
> try if (buffersInPool > 0) {
> buffersInPool -= 1
> pool(buffersInPool)
> } else null
> finally locked.set(false)
>
> // allocate new and clear outside the lock
> if (buffer == null)
> allocate(defaultBufferSize)
> else {
> buffer.clear()
> buffer
> }
> } else takeBufferFromPool() // spin while locked
>
>
> It seems like some thread is holding *locked* (AtomicBoolean) to true
> causing the other threads to spin, but I do not seem to find the offending
> code.
>
> Now to the set of questions:
>
> 1. Using thread-pool-executor instead of fork-join-executor prevents
> the spikes from happening. Anybody running into similar issues with
> fork-join-executor? Any know bug in the Scala ForkJoinPool?
> 2. I don't seem to find a way to use the Java 8 or Java 7 ForkJoinPool
> directly in the Akka docs. Is there a way I can test it out?
> 3. Any other thoughts?
>
> Thanks in advance!
> -Akara
>
>
> --
> >>>>>>>>>> Read the docs: http://akka.io/docs/
> >>>>>>>>>> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
--
Martynas Mickevičius
Typesafe <http://typesafe.com/> – Reactive
<http://www.reactivemanifesto.org/> Apps on the JVM
--
>>>>>>>>>> Read the docs: http://akka.io/docs/
>>>>>>>>>> Check the FAQ:
>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
