Re: [akka-user] Best practice for consuming a Http response entity's data stream

2016-04-13 Thread Chris Baxter 
Bumping this back up to the top to see if I can get an answer from someone 
on the Akka team.  If the intention is to parse the entity's response body 
as json into an object structure, what's the best way to handle that 
without having to eagerly read the entire response into memory?  

On Friday, April 8, 2016 at 9:57:27 AM UTC-4, Chris Baxter wrote:
>
> None of the out of the box Unmarshallers for json (like SprayJsonSupport) 
> support parsing in a lazy and streaming way.  I did find this repo which 
> looks promising:
>
> https://github.com/knutwalker/akka-stream-json
>
> Is this the best kind of approach?  It's certainly nice in that you don't 
> have to read all of the data into memory, but large responses are not the 
> norm and are more of the exception for us.
>
> On Friday, April 8, 2016 at 8:59:10 AM UTC-4, Chris Baxter wrote:
>>
>> Thanks for responding Viktor.
>>
>> 1) I see the flaw in this design (flatMapConcat) now.  If the response is 
>> chunked you only read the first chunk
>> 2) I want to be able to parse the body as json and have the final result 
>> of the flow be a Future for some object that I have mapped the response 
>> json to.  Any suggestions for doing that w/o reading the entire byte string 
>> into memory?  Are you maybe suggesting that instead of feeding something 
>> complete into my json parser (String, Array[Byte]) that I should instead 
>> try and feed in something that is more stream oriented, such as an 
>> InputStream and then find a way to plumb that together with the response 
>> stream?
>>
>> Any suggestions for a Flow that can deal with chunk and feed the data 
>> into a parsing stage w/o having to read it all into memory would be greatly 
>> appreciated?  I don't need perfect code, just an approach so I can take it 
>> from there.
>>
>> On Friday, April 8, 2016 at 7:13:51 AM UTC-4, √ wrote:
>>>
>>>
>>>
>>> On Fri, Apr 8, 2016 at 1:06 PM, Chris Baxter  wrote:
>>>
 If I want to consume a Http service and then do something with the 
 response body, there are a couple of ways to go about doing that.  The two 
 ones that I am trying to decide between are:

 val f:Future[ByteString] =

   Source.single(req).

 via(outgoingConn).

 flatMapConcat(_.entity.dataBytes).

 completionTimeout(timeout).

 runWith(Sink.head)

>>>
>>> This does not return the entire body as a ByteString.
>>>  
>>>
 and


 val f:Future[ByteString] =

   Source.single(req).

 via(pool).

 mapAsync(1){ resp =>

   resp.entity.toStrict(timeout).map(_.data )

 }.

 completionTimeout(timeout).

 runWith(Sink.head)


 I'm thinking the first approach is the better one.  Up until now, my 
 common code for making outbound request has been using the second 
 approach.  I'm about to refactor that code into using the first approach 
 as 
 it seems cleaner and requires less use of Futures.  Just wanted to see 
 what 
 the consensus from the Akka team and others was on this.

>>> My question is: why do you need to eagerly read everything into memory 
>>> to "do something with the repsonse body"?
>>>  
>>>


 -- 
 >> Read the docs: http://akka.io/docs/
 >> Check the FAQ: 
 http://doc.akka.io/docs/akka/current/additional/faq.html
 >> Search the archives: 
 https://groups.google.com/group/akka-user
 --- 
 You received this message because you are subscribed to the Google 
 Groups "Akka User List" group.
 To unsubscribe from this group and stop receiving emails from it, send 
 an email to akka-user+...@googlegroups.com.
 To post to this group, send email to akka...@googlegroups.com.
 Visit this group at https://groups.google.com/group/akka-user.
 For more options, visit https://groups.google.com/d/optout.

>>>
>>>
>>>
>>> -- 
>>> Cheers,
>>> √
>>>
>>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Best practice for consuming a Http response entity's data stream

2016-04-08 Thread Chris Baxter 
None of the out of the box Unmarshallers for json (like SprayJsonSupport) 
support parsing in a lazy and streaming way.  I did find this repo which 
looks promising:

https://github.com/knutwalker/akka-stream-json

Is this the best kind of approach?  It's certainly nice in that you don't 
have to read all of the data into memory, but large responses are not the 
norm and are more of the exception for us.

On Friday, April 8, 2016 at 8:59:10 AM UTC-4, Chris Baxter wrote:
>
> Thanks for responding Viktor.
>
> 1) I see the flaw in this design (flatMapConcat) now.  If the response is 
> chunked you only read the first chunk
> 2) I want to be able to parse the body as json and have the final result 
> of the flow be a Future for some object that I have mapped the response 
> json to.  Any suggestions for doing that w/o reading the entire byte string 
> into memory?  Are you maybe suggesting that instead of feeding something 
> complete into my json parser (String, Array[Byte]) that I should instead 
> try and feed in something that is more stream oriented, such as an 
> InputStream and then find a way to plumb that together with the response 
> stream?
>
> Any suggestions for a Flow that can deal with chunk and feed the data into 
> a parsing stage w/o having to read it all into memory would be greatly 
> appreciated?  I don't need perfect code, just an approach so I can take it 
> from there.
>
> On Friday, April 8, 2016 at 7:13:51 AM UTC-4, √ wrote:
>>
>>
>>
>> On Fri, Apr 8, 2016 at 1:06 PM, Chris Baxter  wrote:
>>
>>> If I want to consume a Http service and then do something with the 
>>> response body, there are a couple of ways to go about doing that.  The two 
>>> ones that I am trying to decide between are:
>>>
>>> val f:Future[ByteString] =
>>>
>>>   Source.single(req).
>>>
>>> via(outgoingConn).
>>>
>>> flatMapConcat(_.entity.dataBytes).
>>>
>>> completionTimeout(timeout).
>>>
>>> runWith(Sink.head)
>>>
>>
>> This does not return the entire body as a ByteString.
>>  
>>
>>> and
>>>
>>>
>>> val f:Future[ByteString] =
>>>
>>>   Source.single(req).
>>>
>>> via(pool).
>>>
>>> mapAsync(1){ resp =>
>>>
>>>   resp.entity.toStrict(timeout).map(_.data )
>>>
>>> }.
>>>
>>> completionTimeout(timeout).
>>>
>>> runWith(Sink.head)
>>>
>>>
>>> I'm thinking the first approach is the better one.  Up until now, my 
>>> common code for making outbound request has been using the second 
>>> approach.  I'm about to refactor that code into using the first approach as 
>>> it seems cleaner and requires less use of Futures.  Just wanted to see what 
>>> the consensus from the Akka team and others was on this.
>>>
>> My question is: why do you need to eagerly read everything into memory to 
>> "do something with the repsonse body"?
>>  
>>
>>>
>>>
>>> -- 
>>> >> Read the docs: http://akka.io/docs/
>>> >> Check the FAQ: 
>>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>> >> Search the archives: 
>>> https://groups.google.com/group/akka-user
>>> --- 
>>> You received this message because you are subscribed to the Google 
>>> Groups "Akka User List" group.
>>> To unsubscribe from this group and stop receiving emails from it, send 
>>> an email to akka-user+...@googlegroups.com.
>>> To post to this group, send email to akka...@googlegroups.com.
>>> Visit this group at https://groups.google.com/group/akka-user.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> -- 
>> Cheers,
>> √
>>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Best practice for consuming a Http response entity's data stream

2016-04-08 Thread Chris Baxter 
Thanks for responding Viktor.

1) I see the flaw in this design (flatMapConcat) now.  If the response is 
chunked you only read the first chunk
2) I want to be able to parse the body as json and have the final result of 
the flow be a Future for some object that I have mapped the response json 
to.  Any suggestions for doing that w/o reading the entire byte string into 
memory?  Are you maybe suggesting that instead of feeding something 
complete into my json parser (String, Array[Byte]) that I should instead 
try and feed in something that is more stream oriented, such as an 
InputStream and then find a way to plumb that together with the response 
stream?

Any suggestions for a Flow that can deal with chunk and feed the data into 
a parsing stage w/o having to read it all into memory would be greatly 
appreciated?  I don't need perfect code, just an approach so I can take it 
from there.

On Friday, April 8, 2016 at 7:13:51 AM UTC-4, √ wrote:
>
>
>
> On Fri, Apr 8, 2016 at 1:06 PM, Chris Baxter  > wrote:
>
>> If I want to consume a Http service and then do something with the 
>> response body, there are a couple of ways to go about doing that.  The two 
>> ones that I am trying to decide between are:
>>
>> val f:Future[ByteString] =
>>
>>   Source.single(req).
>>
>> via(outgoingConn).
>>
>> flatMapConcat(_.entity.dataBytes).
>>
>> completionTimeout(timeout).
>>
>> runWith(Sink.head)
>>
>
> This does not return the entire body as a ByteString.
>  
>
>> and
>>
>>
>> val f:Future[ByteString] =
>>
>>   Source.single(req).
>>
>> via(pool).
>>
>> mapAsync(1){ resp =>
>>
>>   resp.entity.toStrict(timeout).map(_.data )
>>
>> }.
>>
>> completionTimeout(timeout).
>>
>> runWith(Sink.head)
>>
>>
>> I'm thinking the first approach is the better one.  Up until now, my 
>> common code for making outbound request has been using the second 
>> approach.  I'm about to refactor that code into using the first approach as 
>> it seems cleaner and requires less use of Futures.  Just wanted to see what 
>> the consensus from the Akka team and others was on this.
>>
> My question is: why do you need to eagerly read everything into memory to 
> "do something with the repsonse body"?
>  
>
>>
>>
>> -- 
>> >> Read the docs: http://akka.io/docs/
>> >> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> >> Search the archives: https://groups.google.com/group/akka-user
>> --- 
>> You received this message because you are subscribed to the Google Groups 
>> "Akka User List" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to akka-user+...@googlegroups.com .
>> To post to this group, send email to akka...@googlegroups.com 
>> .
>> Visit this group at https://groups.google.com/group/akka-user.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> -- 
> Cheers,
> √
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Best practice for consuming a Http response entity's data stream

2016-04-08 Thread Viktor Klang 
On Fri, Apr 8, 2016 at 1:06 PM, Chris Baxter  wrote:

> If I want to consume a Http service and then do something with the
> response body, there are a couple of ways to go about doing that.  The two
> ones that I am trying to decide between are:
>
> val f:Future[ByteString] =
>
>   Source.single(req).
>
> via(outgoingConn).
>
> flatMapConcat(_.entity.dataBytes).
>
> completionTimeout(timeout).
>
> runWith(Sink.head)
>

This does not return the entire body as a ByteString.


> and
>
>
> val f:Future[ByteString] =
>
>   Source.single(req).
>
> via(pool).
>
> mapAsync(1){ resp =>
>
>   resp.entity.toStrict(timeout).map(_.data )
>
> }.
>
> completionTimeout(timeout).
>
> runWith(Sink.head)
>
>
> I'm thinking the first approach is the better one.  Up until now, my
> common code for making outbound request has been using the second
> approach.  I'm about to refactor that code into using the first approach as
> it seems cleaner and requires less use of Futures.  Just wanted to see what
> the consensus from the Akka team and others was on this.
>
My question is: why do you need to eagerly read everything into memory to
"do something with the repsonse body"?


>
>
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ:
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to akka-user+unsubscr...@googlegroups.com.
> To post to this group, send email to akka-user@googlegroups.com.
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Cheers,
√

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Best practice for consuming a Http response entity's data stream

2016-04-08 Thread Chris Baxter 
If I want to consume a Http service and then do something with the response 
body, there are a couple of ways to go about doing that.  The two ones that 
I am trying to decide between are:

val f:Future[ByteString] =

  Source.single(req).

via(outgoingConn).

flatMapConcat(_.entity.dataBytes).

completionTimeout(timeout).

runWith(Sink.head)


and


val f:Future[ByteString] =

  Source.single(req).

via(pool).

mapAsync(1){ resp =>

  resp.entity.toStrict(timeout).map(_.data )

}.

completionTimeout(timeout).

runWith(Sink.head)


I'm thinking the first approach is the better one.  Up until now, my common 
code for making outbound request has been using the second approach.  I'm 
about to refactor that code into using the first approach as it seems 
cleaner and requires less use of Futures.  Just wanted to see what the 
consensus from the Akka team and others was on this.



-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.