[akka-user] Re: Size limitation for data upload in akka http

[André]

Hi Daniel,

what error msg do you get? Have you tried 
increasing akka.http.server.parsing.max-content-length (reference 

)

https://github.com/akka/akka/issues/18533 could be related here and was 
recently fixed.

Cheers
André

On Wednesday, October 7, 2015 at 9:30:08 AM UTC+2, Daniel Bauer wrote:
>
> I'm using akka http to implement a REST service and that works fine except 
> for one case. The HTTP message parser has a limit on the message size. This 
> means that I cannot upload "large" data using a simple POST message, 
> extract the "entity.dataBytes" and write that to a file. Is there a way how 
> I can get access to a "raw" stream such that it also works with arbitrary 
> file sizes? The simplistic solution below doesn't work for large data (due 
> to the message size limitation):
>
>   val routes = {
> pathSingleSlash {
>   (post & extractRequest) { 
> request => {
>   val source = request.entity.dataBytes
>   val outFile = new File("/tmp/outfile.dat")
>   val sink = SynchronousFileSink.create(outFile)
>   val repl = source.runWith(sink).map(x => s"Finished uploading 
> ${x} bytes!")
>   onSuccess(repl) { repl =>
> complete(HttpResponse(status = StatusCodes.OK, entity = repl))
>   }
> }
>   }
> }
>   }
>
>   Http().bindAndHandle(routes, config.getString("http.interface"), 
> config.getInt("http.port"))
>
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Re: Size limitation for data upload in akka http

[Daniel Bauer]

Hi André.

Thanks for your reply. The error I get is this:

Illegal request, responding with status '413 Request Entity Too Large': 
Request Content-Length 24090970 exceeds the configured limit of 8388608

While I could increase the max-content-length, this wouldn't work in 
general. I'd like to prevent that the entire file is first buffered in 
memory. I'm expecting that some files might be tens of gigabytes. Is there 
a way to circumvent that and get the content as an input stream?

-Daniel

On Wednesday, October 7, 2015 at 11:39:12 AM UTC+2, André wrote:
>
> Hi Daniel,
>
> what error msg do you get? Have you tried 
> increasing akka.http.server.parsing.max-content-length (reference 
> 
> )
>
> https://github.com/akka/akka/issues/18533 could be related here and was 
> recently fixed.
>
> Cheers
> André
>
> On Wednesday, October 7, 2015 at 9:30:08 AM UTC+2, Daniel Bauer wrote:
>>
>> I'm using akka http to implement a REST service and that works fine 
>> except for one case. The HTTP message parser has a limit on the message 
>> size. This means that I cannot upload "large" data using a simple POST 
>> message, extract the "entity.dataBytes" and write that to a file. Is there 
>> a way how I can get access to a "raw" stream such that it also works with 
>> arbitrary file sizes? The simplistic solution below doesn't work for large 
>> data (due to the message size limitation):
>>
>>   val routes = {
>> pathSingleSlash {
>>   (post & extractRequest) { 
>> request => {
>>   val source = request.entity.dataBytes
>>   val outFile = new File("/tmp/outfile.dat")
>>   val sink = SynchronousFileSink.create(outFile)
>>   val repl = source.runWith(sink).map(x => s"Finished uploading 
>> ${x} bytes!")
>>   onSuccess(repl) { repl =>
>> complete(HttpResponse(status = StatusCodes.OK, entity = repl))
>>   }
>> }
>>   }
>> }
>>   }
>>
>>   Http().bindAndHandle(routes, config.getString("http.interface"), 
>> config.getInt("http.port"))
>>
>>
>>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: Size limitation for data upload in akka http

[Konrad Malawski]

Hi there,

Yes, please increase that setting, it does the limiting by counting how many 
bytes where sent and if it's exceeded aborts.

No, increasing this setting to a large number will *not* cause the file be 
buffered in completeness in memory (!) - while using Akka HTTP you're safe from 
those problems.

You don't need an input stream - just keep using the Source[ByteString, _] that 
Akka HTTP exposes and the file-uploading client will be back-pressured
and will never send more than the server allows it to. This "allows it to" is 
driven by how much data you've consumed in your Akka Stream - from the 
entity.dataBytes :-)

This works because we're using akka-streams to read from the TCP socket, and if 
you're using `dataBytes` to stream somewhere else,
only this certain chunk of the file fill be in memory at any given time.

-- 
Cheers,
Konrad 'ktoso’ Malawski
Akka @ Typesafe

On 8 October 2015 at 14:44:23, Daniel Bauer (dan13ba...@gmail.com) wrote:

Hi André.

Thanks for your reply. The error I get is this:

Illegal request, responding with status '413 Request Entity Too Large': Request 
Content-Length 24090970 exceeds the configured limit of 8388608

While I could increase the max-content-length, this wouldn't work in general. 
I'd like to prevent that the entire file is first buffered in memory. I'm 
expecting that some files might be tens of gigabytes. Is there a way to 
circumvent that and get the content as an input stream?

-Daniel

On Wednesday, October 7, 2015 at 11:39:12 AM UTC+2, André wrote:
Hi Daniel,

what error msg do you get? Have you tried increasing 
akka.http.server.parsing.max-content-length (reference)

https://github.com/akka/akka/issues/18533 could be related here and was 
recently fixed.

Cheers
André

On Wednesday, October 7, 2015 at 9:30:08 AM UTC+2, Daniel Bauer wrote:
I'm using akka http to implement a REST service and that works fine except for 
one case. The HTTP message parser has a limit on the message size. This means 
that I cannot upload "large" data using a simple POST message, extract the 
"entity.dataBytes" and write that to a file. Is there a way how I can get 
access to a "raw" stream such that it also works with arbitrary file sizes? The 
simplistic solution below doesn't work for large data (due to the message size 
limitation):

  val routes = {
    pathSingleSlash {
  (post & extractRequest) {
    request => {
  val source = request.entity.dataBytes
  val outFile = new File("/tmp/outfile.dat")
  val sink = SynchronousFileSink.create(outFile)
  val repl = source.runWith(sink).map(x => s"Finished uploading ${x} 
bytes!")
  onSuccess(repl) { repl =>
    complete(HttpResponse(status = StatusCodes.OK, entity = repl))
  }
    }
  }
    }
  }

  Http().bindAndHandle(routes, config.getString("http.interface"), 
config.getInt("http.port"))


--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.