Re: [akka-user] Using Akka http for large uploads not uploading entire file

[Konrad Malawski]

I see, thanks for double-checking on your end then!

I hope you're enjoying Akka Http :-)
Happy hakking!

-- 
Cheers,
Konrad `ktoso` Malawski
Akka @ Typesafe

On 2 September 2015 at 16:08:55, Charlie Evans (charlesdevan...@gmail.com) 
wrote:

Hi Konrad,

Thanks so much in getting back to me, I'm starting to think it may be an issue 
outside of akka. We have Node proxying the request over to the akka server and 
it might be something funky going on there. 

Sorry to have wasted your time.

Charlie

On Wednesday, September 2, 2015 at 3:02:28 PM UTC+1, Konrad Malawski wrote:
Actually, follow up question – since the snippet seems incomplete.
How are you running the mapAsync?

In the snippet you posted there is no materialization of the 
formdata.parts.mapAsync(1)
part. You should have some run() at the end of the stream, otherwise it does 
nothing.

For example you could runWith(Sink.ignore).map(_ => "Upload completed!")

Would you mind posting the complete snippet so we can verify it?

// I quickly did so but was not able to reproduce the problem described so far.

-- 
Cheers,
Konrad `ktoso` Malawski
Akka @ Typesafe

On 2 September 2015 at 14:46:11, Charlie Evans (charles...@gmail.com) wrote:

Hi all,

I'm trying to use Akka http for large files (300MB is what I'm currently 
testing) coming in as multi-part form data. The file gets written straight to 
disk. The file ends up being a few MB short of complete and the upload hangs 
indefinitely. I had a look at the files and the top and bottom match so it must 
have missed a chunk somewhere in between? How would I prevent this?

The code of interest is below:

val file = new File(localPath)

val sink = SynchronousFileSink(file)


entity(as[Multipart.FormData]) { (formdata: Multipart.FormData) =>

val fileNamesFuture = formdata.parts.mapAsync(1) { p =>

                    if (p.filename.isDefined) {

                      println(s"Got part. name: ${p.name} filename: 
${p.filename}")

                      p.entity.dataBytes.runWith(sink).map { bytesWritten => 

                        """{"result":"Finished upload"}""".parseJson

}

}



Thanks



--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+...@googlegroups.com.
To post to this group, send email to akka...@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.
--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Using Akka http for large uploads not uploading entire file

[Charlie Evans]

Hi Konrad,

Thanks so much in getting back to me, I'm starting to think it may be an 
issue outside of akka. We have Node proxying the request over to the akka 
server and it might be something funky going on there. 

Sorry to have wasted your time.

Charlie

On Wednesday, September 2, 2015 at 3:02:28 PM UTC+1, Konrad Malawski wrote:
>
> Actually, follow up question – since the snippet seems incomplete.
> How are you running the mapAsync?
>
> In the snippet you posted there is no materialization of the 
> formdata.parts.mapAsync(1)
> part. You should have some run() at the end of the stream, otherwise it 
> does nothing.
>
> For example you could runWith(Sink.ignore).map(_ => "Upload completed!")
>
> Would you mind posting the complete snippet so we can verify it?
>
> // I quickly did so but was not able to reproduce the problem described so 
> far.
>
> -- 
> Cheers,
> Konrad `ktoso` Malawski
> Akka  @ Typesafe 
>
> On 2 September 2015 at 14:46:11, Charlie Evans (charles...@gmail.com 
> ) wrote:
>
> Hi all, 
>
> I'm trying to use Akka http for large files (300MB is what I'm currently 
> testing) coming in as multi-part form data. The file gets written straight 
> to disk. The file ends up being a few MB short of complete and the upload 
> hangs indefinitely. I had a look at the files and the top and bottom match 
> so it must have missed a chunk somewhere in between? How would I prevent 
> this?
>
> The code of interest is below:
>
> val file = new File(localPath)
>
> val sink = SynchronousFileSink(file)
>
>
> entity(as[Multipart.FormData]) { (formdata: Multipart.FormData) =>
>
> val fileNamesFuture = formdata.parts.mapAsync(1) { p =>
>
> if (p.filename.isDefined) {
>
>   println(s"Got part. name: ${p.name} filename: ${p.
> filename}")
>
>   p.entity.dataBytes.runWith(sink).map { bytesWritten 
> => 
>
> """{"result":"Finished upload"}""".parseJson
>
> }
>
> }
>
>
> Thanks
>
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: 
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+...@googlegroups.com .
> To post to this group, send email to akka...@googlegroups.com 
> .
> Visit this group at http://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Using Akka http for large uploads not uploading entire file

[Konrad Malawski]

Actually, follow up question – since the snippet seems incomplete.
How are you running the mapAsync?

In the snippet you posted there is no materialization of the 
formdata.parts.mapAsync(1)
part. You should have some run() at the end of the stream, otherwise it does 
nothing.

For example you could runWith(Sink.ignore).map(_ => "Upload completed!")

Would you mind posting the complete snippet so we can verify it?

// I quickly did so but was not able to reproduce the problem described so far.

-- 
Cheers,
Konrad `ktoso` Malawski
Akka @ Typesafe

On 2 September 2015 at 14:46:11, Charlie Evans (charlesdevan...@gmail.com) 
wrote:

Hi all,

I'm trying to use Akka http for large files (300MB is what I'm currently 
testing) coming in as multi-part form data. The file gets written straight to 
disk. The file ends up being a few MB short of complete and the upload hangs 
indefinitely. I had a look at the files and the top and bottom match so it must 
have missed a chunk somewhere in between? How would I prevent this?

The code of interest is below:

val file = new File(localPath)

val sink = SynchronousFileSink(file)



entity(as[Multipart.FormData]) { (formdata: Multipart.FormData) =>

val fileNamesFuture = formdata.parts.mapAsync(1) { p =>

                    if (p.filename.isDefined) {

                      println(s"Got part. name: ${p.name} filename: 
${p.filename}")

                      p.entity.dataBytes.runWith(sink).map { bytesWritten => 

                        """{"result":"Finished upload"}""".parseJson

}

}



Thanks



--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Using Akka http for large uploads not uploading entire file

[Konrad Malawski]

Hi Charlie,
I'll try to reproduce the problem you described - it definitely should not 
happen, so we may be looking at a bug here.
Is there anything else other than the snippet (thanks for pasting it!) that 
might be related to the issue you're seeing?

Does the same happen if you replicate the posted data with curl (I'll use that 
now locally to simulate the same situation,
and would like to confirm it fails for you with curl as well)?

// And a sanity check - you are using Akka Streams 1.0 and Akka 2.1.12 (if not 
please upgrade)?

-- 
Cheers,
Konrad `ktoso` Malawski
Akka @ Typesafe

On 2 September 2015 at 14:46:11, Charlie Evans (charlesdevan...@gmail.com) 
wrote:

Hi all,

I'm trying to use Akka http for large files (300MB is what I'm currently 
testing) coming in as multi-part form data. The file gets written straight to 
disk. The file ends up being a few MB short of complete and the upload hangs 
indefinitely. I had a look at the files and the top and bottom match so it must 
have missed a chunk somewhere in between? How would I prevent this?

The code of interest is below:

val file = new File(localPath)

val sink = SynchronousFileSink(file)



entity(as[Multipart.FormData]) { (formdata: Multipart.FormData) =>

val fileNamesFuture = formdata.parts.mapAsync(1) { p =>

                    if (p.filename.isDefined) {

                      println(s"Got part. name: ${p.name} filename: 
${p.filename}")

                      p.entity.dataBytes.runWith(sink).map { bytesWritten => 

                        """{"result":"Finished upload"}""".parseJson

}

}



Thanks



--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Using Akka http for large uploads not uploading entire file

[Charlie Evans]

Hi all,

I'm trying to use Akka http for large files (300MB is what I'm currently 
testing) coming in as multi-part form data. The file gets written straight 
to disk. The file ends up being a few MB short of complete and the upload 
hangs indefinitely. I had a look at the files and the top and bottom match 
so it must have missed a chunk somewhere in between? How would I prevent 
this?

The code of interest is below:

val file = new File(localPath)

val sink = SynchronousFileSink(file)


entity(as[Multipart.FormData]) { (formdata: Multipart.FormData) =>

val fileNamesFuture = formdata.parts.mapAsync(1) { p =>

if (p.filename.isDefined) {

  println(s"Got part. name: ${p.name} filename: ${p.
filename}")

  p.entity.dataBytes.runWith(sink).map { bytesWritten 
=> 

"""{"result":"Finished upload"}""".parseJson

}

}


Thanks


-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.