akka version: 2.3.12 akka-http version: 1.0 Hi all, I am running a service which is receiving small to medium payloads and uploading them to S3 type cloud storage. The load I am applying is about 200 requests/sec. Occasionally, I see the below stack trace and the service is pausing, temporarily not able to process any further requests. It will eventually resume. The stack trace does not point to any of my classes/namespace, rather show akka-stream classes.. Has anyone seen this exception and/or can help pin point the source of the error? Thanks.
=== Aug 28 04:57:33: 2015-08-28T04:57:33.773+0000 level=ERROR logger=a.a.ActorSystemImpl Outgoing request stream error sourceThread=casper-akka.actor.default-dispatcher-206, akkaTimestamp=04:57:33.767UTC, akkaSource=ActorSystem(casper) Aug 28 04:57:33: java.lang.IllegalStateException: oneHundredContinueRef unavailable Aug 28 04:57:33: at akka.http.impl.engine.parsing.HttpRequestParser$$anonfun$expect100continueHandling$1$1$$anonfun$apply$1$$anon$1$$anonfun$1.apply(HttpRequestParser.scala:148) ~[com.typesafe.akka.akka-http-core-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.http.impl.engine.parsing.HttpRequestParser$$anonfun$expect100continueHandling$1$1$$anonfun$apply$1$$anon$1$$anonfun$1.apply(HttpRequestParser.scala:148) ~[com.typesafe.akka.akka-http-core-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.Option.getOrElse(Option.scala:121) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.http.impl.engine.parsing.HttpRequestParser$$anonfun$expect100continueHandling$1$1$$anonfun$apply$1$$anon$1.onPull(HttpRequestParser.scala:148) ~[com.typesafe.akka.akka-http-core-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.http.impl.engine.parsing.HttpRequestParser$$anonfun$expect100continueHandling$1$1$$anonfun$apply$1$$anon$1.onPull(HttpRequestParser.scala:143) ~[com.typesafe.akka.akka-http-core-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.fusing.OneBoundedInterpreter$$anon$3.run(Interpreter.scala:473) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.fusing.OneBoundedInterpreter$State$class.progress(Interpreter.scala:245) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.fusing.OneBoundedInterpreter$$anon$3.progress(Interpreter.scala:467) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.fusing.OneBoundedInterpreter.akka$stream$impl$fusing$OneBoundedInterpreter$$execute(Interpreter.scala:580) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.fusing.OneBoundedInterpreter$State$class.execute(Interpreter.scala:241) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.fusing.OneBoundedInterpreter$EntryState.execute(Interpreter.scala:666) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.stage.AbstractStage.enterAndPull(Stage.scala:76) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.fusing.ActorOutputBoundary.akka$stream$impl$fusing$ActorOutputBoundary$$tryPutBallIn(ActorInterpreter.scala:217) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.fusing.ActorOutputBoundary$$anonfun$downstreamRunning$1.applyOrElse(ActorInterpreter.scala:305) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.stream.impl.SubReceive.apply(Transfer.scala:16) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.SubReceive.apply(Transfer.scala:12) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.stream.impl.SubReceive.applyOrElse(Transfer.scala:12) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.actor.Actor$class.aroundReceive(Actor.scala:467) ~[com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.stream.impl.fusing.ActorInterpreter.aroundReceive(ActorInterpreter.scala:366) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.actor.ActorCell.invoke(ActorCell.scala:487) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.dispatch.Mailbox.run(Mailbox.scala:220) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: 2015-08-28T04:57:33.789+0000 level=ERROR logger=a.a.ActorSystemImpl Outgoing request stream error sourceThread=casper-akka.actor.default-dispatcher-222, akkaTimestamp=04:57:33.772UTC, akkaSource=ActorSystem(casper) Aug 28 04:57:33: java.lang.IllegalStateException: Substream publisher only supports one subscriber (which is allowed, see reactive-streams specification, rule 1.12) Aug 28 04:57:33: at akka.stream.impl.ReactiveStreamsCompliance$.rejectAdditionalSubscriber(ReactiveStreamsCompliance.scala:57) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MultiStreamOutputProcessor$SubstreamOutput.subscribe(StreamOfStreamProcessors.scala:104) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.assignPort(StreamLayout.scala:721) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.ActorMaterializerImpl$$anon$3.materializeAtomic(ActorMaterializerImpl.scala:99) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:688) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.collection.immutable.Set$Set4.foreach(Set.scala:181) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.materializeComposite(StreamLayout.scala:704) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:694) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.collection.immutable.Set$Set1.foreach(Set.scala:79) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:691) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.collection.immutable.Set$Set3.foreach(Set.scala:145) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.materializeComposite(StreamLayout.scala:704) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:694) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.collection.immutable.Set$Set1.foreach(Set.scala:79) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.materializeComposite(StreamLayout.scala:704) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:694) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession$$anonfun$materializeModule$1.apply(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.collection.immutable.Set$Set2.foreach(Set.scala:111) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.materializeModule(StreamLayout.scala:679) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.MaterializerSession.materialize(StreamLayout.scala:655) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.ActorMaterializerImpl.materialize(ActorMaterializerImpl.scala:201) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.scaladsl.RunnableGraph.run(Flow.scala:365) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.scaladsl.Source.runWith(Source.scala:93) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.ConcatAllImpl$$anonfun$1.apply$mcV$sp(ConcatAllImpl.scala:28) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.Pump$class.pump(Transfer.scala:199) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.ActorProcessorImpl.pump(ActorProcessor.scala:250) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.BatchingInputBuffer.enqueueInputElement(ActorProcessor.scala:91) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.BatchingInputBuffer$$anonfun$upstreamRunning$1.applyOrElse(ActorProcessor.scala:142) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.stream.impl.SubReceive.apply(Transfer.scala:16) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.stream.impl.SubReceive.apply(Transfer.scala:12) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.stream.impl.SubReceive.applyOrElse(Transfer.scala:12) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) ~[org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at akka.actor.Actor$class.aroundReceive(Actor.scala:467) ~[com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.stream.impl.ActorProcessorImpl.aroundReceive(ActorProcessor.scala:250) ~[com.typesafe.akka.akka-stream-experimental_2.11-1.0.jar:na] Aug 28 04:57:33: at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.actor.ActorCell.invoke(ActorCell.scala:487) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.dispatch.Mailbox.run(Mailbox.scala:220) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) [com.typesafe.akka.akka-actor_2.11-2.3.12.jar:na] Aug 28 04:57:33: at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [org.scala-lang.scala-library-2.11.6.jar:na] Aug 28 04:57:33: at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [org.scala-lang.scala-library-2.11.6.jar:na] -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
