Re: [akka-user] akka-http receiving file upload without copying the file to memory

2016-07-13 Thread Simao Mata 
Alright, thanks for your help, now I see that the problem is in parsing
the request I will look into using the `fileUpload` directive.
 
Thanks!
 
On Wed, Jul 13, 2016, at 16:04, Simao Mata wrote:
> Ah, that explains it then.
>
> I am using `formFields('file.as[StrictForm.FileData])` to extract the
> file from the form. So is there a way I can get that data and not put
> it in memory?
>
> Thanks!
>
>
> On Wed, Jul 13, 2016, at 16:00, Konrad Malawski wrote:
>> `Strict` indicates "the entire thing is in memory. (run away before
>> it blows up!!!)".
>>
>> How did you get this entity? Normally we don't expose Strict
>> entities, you likely have converted to it explicitly => thus, loading
>> the entire upload into memory.
>> Don't do toStrict or any of these operations, just stream directly
>> into a file.
>>
>> --
>> Konrad `ktoso` Malawski
>> Akka[1] @ Lightbend[2]
>>
>> On 13 July 2016 at 15:58:22, Simão Mata (sima...@gmail.com) wrote:
>>>
>>>
>>>
>>> Hi,
>>>
>>>  I would like to ask for a clarification on how
>>>  `mapMaterializedValue` works, I think this might be the cause for
>>>  this problem.
>>>
>>>  I am trying to receive a large file upload, and I am copying to the
>>>  file system using something like:
>>>
>>>  val sink = FileIO .toPath(path) .mapMaterializedValue(_.map(result
>>>  => (uri, result.count)))
>>>
>>>  val source = fileData.entity.dataBytes // fileData is a
>>>  StrictForm.FileData
>>>
>>>  val f = source.to(sink).run()
>>>
>>>  This somehow causes the file to be copied to memory, which causes
>>>  the jvm to stop with an out of memory exception.
>>>
>>>  Can `mapMaterializedValue` be the cause for this behaviour?
>>>
>>>  Thank you, Simao
>>> --
>>>  >> Read the docs: http://akka.io/docs/ Check the FAQ:
>>>  >> http://doc.akka.io/docs/akka/current/additional/faq.html
>>>  >> Search the archives:
>>>  >> https://groups.google.com/group/akka-user
>>>  ---
>>>  You received this message because you are subscribed to the Google
>>>  Groups "Akka User List" group. To unsubscribe from this group and
>>>  stop receiving emails from it, send an email to akka-
>>>  user+unsubscr...@googlegroups.com. To post to this group, send
>>>  email to akka-user@googlegroups.com. Visit this group at
>>>  https://groups.google.com/group/akka-user. For more options, visit
>>>  https://groups.google.com/d/optout.
>>>
>
>
> --
>  >>  Read the docs: http://akka.io/docs/
>  >>  Check the FAQ:
>  >>  http://doc.akka.io/docs/akka/current/additional/faq.html
>  >>  Search the archives:
>  >>  https://groups.google.com/group/akka-user
>  ---
>  You received this message because you are subscribed to a topic in
>  the Google Groups "Akka User List" group.
>  To unsubscribe from this topic, visit
>  https://groups.google.com/d/topic/akka-user/PZlTYMydfZk/unsubscribe.
>  To unsubscribe from this group and all its topics, send an email to
>  akka-user+unsubscr...@googlegroups.com.
>  To post to this group, send email to akka-user@googlegroups.com.
>  Visit this group at https://groups.google.com/group/akka-user.
>  For more options, visit https://groups.google.com/d/optout.
 

Links:

  1. http://akka.io
  2. http://lightbend.com

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] akka-http receiving file upload without copying the file to memory

2016-07-13 Thread Konrad Malawski 
Please check the docs:
http://doc.akka.io/docs/akka/2.4/scala/http/routing-dsl/directives/file-upload-directives/fileUpload.html
And the impl of this directive to gain some more understanding how it works.

-- 
Konrad `ktoso` Malawski
Akka  @ Lightbend 

On 13 July 2016 at 16:04:14, Simao Mata (sima...@gmail.com) wrote:

Ah, that explains it then.

I am using `formFields('file.as[StrictForm.FileData])` to extract the file
from the form. So is there a way I can get that data and not put it in
memory?

Thanks!


On Wed, Jul 13, 2016, at 16:00, Konrad Malawski wrote:

`Strict` indicates "the entire thing is in memory. (run away before it
blows up!!!)".

How did you get this entity? Normally we don't expose Strict entities, you
likely have converted to it explicitly => thus, loading the entire upload
into memory.
Don't do toStrict or any of these operations, just stream directly into a
file.

-- 
Konrad `ktoso` Malawski
Akka  @ Lightbend 


On 13 July 2016 at 15:58:22, Simão Mata (sima...@gmail.com) wrote:




Hi,

I would like to ask for a clarification on how `mapMaterializedValue`
works, I think this might be the cause for this problem.

I am trying to receive a large file upload, and I am copying to the file
system using something like:

val sink = FileIO
.toPath(path)
.mapMaterializedValue(_.map(result => (uri, result.count)))

val source = fileData.entity.dataBytes // fileData is a StrictForm.FileData

val f = source.to(sink).run()

This somehow causes the file to be copied to memory, which causes the jvm
to stop with an out of memory exception.

Can `mapMaterializedValue` be the cause for this behaviour?

Thank you,
Simao
--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] akka-http receiving file upload without copying the file to memory

2016-07-13 Thread Simao Mata 
Ah, that explains it then.
 
I am using `formFields('file.as[StrictForm.FileData])` to extract the
file from the form. So is there a way I can get that data and not put it
in memory?
 
Thanks!
 
 
On Wed, Jul 13, 2016, at 16:00, Konrad Malawski wrote:
> `Strict` indicates "the entire thing is in memory. (run away before it
> blows up!!!)".
>
> How did you get this entity? Normally we don't expose Strict entities,
> you likely have converted to it explicitly => thus, loading the entire
> upload into memory.
> Don't do toStrict or any of these operations, just stream directly
> into a file.
>
> --
> Konrad `ktoso` Malawski
> Akka[1] @ Lightbend[2]
>
> On 13 July 2016 at 15:58:22, Simão Mata (sima...@gmail.com) wrote:
>>
>>
>>
>> Hi,
>>
>>  I would like to ask for a clarification on how
>>  `mapMaterializedValue` works, I think this might be the cause for
>>  this problem.
>>
>>  I am trying to receive a large file upload, and I am copying to the
>>  file system using something like:
>>
>>  val sink = FileIO .toPath(path) .mapMaterializedValue(_.map(result
>>  => (uri, result.count)))
>>
>>  val source = fileData.entity.dataBytes // fileData is a
>>  StrictForm.FileData
>>
>>  val f = source.to(sink).run()
>>
>>  This somehow causes the file to be copied to memory, which causes
>>  the jvm to stop with an out of memory exception.
>>
>>  Can `mapMaterializedValue` be the cause for this behaviour?
>>
>>  Thank you, Simao
>> --
>>  >> Read the docs: http://akka.io/docs/ Check the FAQ:
>>  >> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  >> Search the archives:
>>  >> https://groups.google.com/group/akka-user
>>  ---
>>  You received this message because you are subscribed to the Google
>>  Groups "Akka User List" group. To unsubscribe from this group and
>>  stop receiving emails from it, send an email to akka-
>>  user+unsubscr...@googlegroups.com. To post to this group, send email
>>  to akka-user@googlegroups.com. Visit this group at
>>  https://groups.google.com/group/akka-user. For more options, visit
>>  https://groups.google.com/d/optout.
>>
 

Links:

  1. http://akka.io
  2. http://lightbend.com

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] akka-http receiving file upload without copying the file to memory

2016-07-13 Thread Simão Mata 
Hi,

I would like to ask for a clarification on how `mapMaterializedValue` 
works, I think this might be the cause for this problem.

I am trying to receive a large file upload, and I am copying to the file 
system using something like:

val sink = FileIO
  .toPath(path)
  .mapMaterializedValue(_.map(result => (uri, result.count)))

val source = fileData.entity.dataBytes // fileData is a 
StrictForm.FileData

val f = source.to(sink).run()

This somehow causes the file to be copied to memory, which causes the jvm 
to stop with an out of memory exception.

Can `mapMaterializedValue` be the cause for this behaviour?

Thank you,
Simao

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.