Hi Konrad, thanks for you answer. This explains a lot and makes sense. Configuring 'infinite' fixes my issue. The new error description makes it a lot easier.
One thing I do notice is that the CPU keeps running high whenever I kill 'curl'. Is there something I should do to close the stream? Suspending curl works fine though. Thanks, Jeroen Op vrijdag 18 december 2015 14:17:47 UTC+1 schreef Konrad Malawski: > > It's a feature. (yes, really) :-) > > Allow me to explain; Akka HTTP always opts on the safe side of things. > For example, if you write an endpoint that can get POST data, and someone > (an attacker) sends you data and it never ends sending... > You do want to kill that connection as soon as you notice something fishy > is going on (i.e. perhaps an attack, someone sending loads of data > and you never expected them to). > > So the default is to play safe, and limit any HTTP Entity to 8M (which is > pretty huge anyway, if you think about it – we've heard > people say "whoa, that's a huge default!", but of course they'd trim it > down to a lower setting for production, suiting their needs). > > If you know that all calls you do will use streaming instead of > "accumulate that infinite stream as String" (you can do that, right), > you can disable this check by using the setting described here: > > http://doc.akka.io/docs/akka-stream-and-http-experimental/snapshot/scala/http/common/http-model.html#Limiting_message_entity_length > > So the value of: > akka.http.[server|client].parsing.max-content-length > (Depending if you want it for server or client). > > Last week I also improved that exception to be more self-explanatory, how > it looks like this ( https://github.com/akka/akka/pull/19158 ): > s"EntityStreamSizeException: actual entity size ($actualSize) exceeded > content length limit ($limit bytes)! " + > s"You can configure this by setting > `akka.http.[server|client].parsing.max-content-length` or calling > `HttpEntity.withSizeLimit` " + > s"before materializing the dataBytes stream." > > So your question will have been answered by the exception itself hopefully > :-) > > > I also strongly recommend you have a look at this workshop I did on Scala > Exchange a week ago: > > https://skillsmatter.com/skillscasts/6869-workshop-end-to-end-asynchronous-back-pressure-with-akka-streams > It goes into depth who and how these things work the way they work. > > Thanks for trying out the early 2.0 milestones, we'll be releasing a "2.0" > very soon, please upgrade to it then! :-) > Hope this helps, happy hakking! > > -- > Cheers, > Konrad 'ktoso’ Malawski > Akka <http://akka.io> @ Typesafe <http://typesafe.com> > > On 18 December 2015 at 14:02:30, Jeroen Gordijn (jeroen....@gmail.com > <javascript:>) wrote: > > Same code with akka-streams 1.0 works fine. > > Is this a bug? > > Regards, > Jeroen > > Op donderdag 17 december 2015 22:16:17 UTC+1 schreef Jeroen Gordijn: >> >> Hi all, >> >> I'm running into an EntityStreamSizeException when streaming data from a >> streaming response I got by calling another endpoint.. It is a little bit >> like presented in the talk by Mathias & Johannes at scalaworld: >> https://www.youtube.com/watch?v=6VBn9V3S2aQ >> >> I'm using with akka-http 2.0-M2 and created my problem in isolation. See >> the route (and link to full gist below). When I call `curl -i >> http://localhost:8080/endless` <http://localhost:8080/endless> the >> stream will continue indefinitely. However, when I call `curl -i >> http://localhost:8080/pipe` <http://localhost:8080/pipe> it takes a few >> seconds to get "curl: (56) Recv failure: Connection reset by peer" on the >> client an the exception below on the server. The source below is just an >> example to isolate the problem. >> >> Am I doing something wrong? I would expect an endless stream and no >> limit. I'm using Chunked as stated in >> http://doc.akka.io/docs/akka-stream-and-http-experimental/snapshot/scala/http/common/http-model.html#HttpEntity >> >> Thanks! >> Jeroen >> >> val source: Source[Int, Unit] = Source(Stream.from(1)) >> >> val route = (path("endless") & get) { >> complete { >> HttpResponse( >> entity = HttpEntity.Chunked( >> MediaTypes.`text/plain`, >> source.map(nr ⇒ ByteString((nr.toString * 10) + "\n", "UTF-8")) >> ) >> ) >> } >> } ~ >> (path("pipe") & get) { >> val s = Http().singleRequest(HttpRequest(uri = >> "http://localhost:8080/endless";)).map { >> _.entity.dataBytes >> .via(Framing.delimiter(ByteString("\n"), >> maximumFrameLength = 10000, allowTruncation = true)) >> .map(entry ⇒ entry.utf8String) >> } >> onSuccess(s) { x ⇒ >> complete(HttpResponse( >> entity = HttpEntity.Chunked( >> MediaTypes.`text/plain`, >> x.map(x ⇒ ByteString(x + "----\n", "UTF-8") >> ) >> ))) >> } >> } >> >> >> >> *Full gist*: https://gist.github.com/jgordijn/390c9022062cfb9fce8c >> >> *Exception:* >> [ERROR] [12/17/2015 22:06:10.493] [Test-akka.actor.default-dispatcher-4] >> [ActorSystem(Test)] Outgoing request stream error >> EntityStreamSizeException(8388608, None) >> at >> akka.http.scaladsl.model.HttpEntity$$anonfun$limitable$1$$anon$1.onPush(HttpEntity.scala:469) >> at >> akka.http.scaladsl.model.HttpEntity$$anonfun$limitable$1$$anon$1.onPush(HttpEntity.scala:451) >> at >> akka.stream.stage.AbstractStage$PushPullGraphLogic$$anon$1.onPush(Stage.scala:54) >> at >> akka.stream.impl.fusing.GraphInterpreter.processElement$1(GraphInterpreter.scala:535) >> at >> akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:546) >> at >> akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:509) >> at >> akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$runBatch(ActorGraphInterpreter.scala:408) >> at >> akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:346) >> at akka.actor.Actor$class.aroundReceive(Actor.scala:467) >> at >> akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:292) >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >> at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) >> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> [ERROR] [12/17/2015 22:06:10.491] [Test-akka.actor.default-dispatcher-9] >> [ActorSystem(Test)] Outgoing response stream error >> EntityStreamSizeException(8388608, None) >> at >> akka.http.scaladsl.model.HttpEntity$$anonfun$limitable$1$$anon$1.onPush(HttpEntity.scala:469) >> at >> akka.http.scaladsl.model.HttpEntity$$anonfun$limitable$1$$anon$1.onPush(HttpEntity.scala:451) >> at >> akka.stream.stage.AbstractStage$PushPullGraphLogic$$anon$1.onPush(Stage.scala:54) >> at >> akka.stream.impl.fusing.GraphInterpreter.processElement$1(GraphInterpreter.scala:535) >> at >> akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:546) >> at >> akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:509) >> at >> akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$runBatch(ActorGraphInterpreter.scala:408) >> at >> akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:346) >> at akka.actor.Actor$class.aroundReceive(Actor.scala:467) >> at >> akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:292) >> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) >> at akka.dispatch.Mailbox.run(Mailbox.scala:220) >> at >> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397) >> at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >> at >> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >> at >> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >> at >> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >> >> [ERROR] [12/17/2015 22:06:10.500] [Test-akka.actor.default-dispatcher-8] >> [ActorSystem(Test)] Outgoing response stream error >> (akka.stream.StreamTcpException) >> [INFO] [12/17/2015 22:06:10.502] [Test-akka.actor.default-dispatcher-3] >> [akka://Test/system/IO-TCP/selectors/$a/4] Message >> [akka.io.Tcp$ResumeReading$] from >> Actor[akka://Test/user/StreamSupervisor-0/StageActorRef-4] to >> Actor[akka://Test/system/IO-TCP/selectors/$a/4#-1123967977] was not >> delivered. [1] dead letters encountered. This logging can be turned off or >> adjusted with configuration settings 'akka.log-dead-letters' and >> 'akka.log-dead-letters-during-shutdown'. >> >> -- > >>>>>>>>>> Read the docs: http://akka.io/docs/ > >>>>>>>>>> Check the FAQ: > http://doc.akka.io/docs/akka/current/additional/faq.html > >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to the Google Groups > "Akka User List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to akka-user+...@googlegroups.com <javascript:>. > To post to this group, send email to akka...@googlegroups.com > <javascript:>. > Visit this group at https://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. > > -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at https://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
