Re: [akka-user] Re: Fetching remote file via akka-http singleRequest results in extremely high allocation rate

2016-10-19 Thread vladyslav . chaban 
Hi,

there's really not too much to add to the gist from the first post: this 
method is called once, almost right in the main class, to fetch some 
resource. For now I managed to work this issue around, but I thought that 
working on rope-like ByteStrings should be faster than on array-backed Java 
strings? Is it possible to split incoming ByteString chunk directly without 
reallocation?

Thanks

On Wednesday, October 19, 2016 at 4:32:09 PM UTC+2, Konrad Malawski wrote:
>
> Hi Vladyslav,
> firstly, no reason to get desperate :-)
> Glad you did some profiling, it would be super helpful next time around if 
> you share more details about both your code an profile you're observing.
> It's hard to give advice what can be improved when given a blank canvas 
> (no idea what your code looks like).
>
> Of course things can be improved, and in fact just a few days ago we 
> merged a significant performance improvement that affects the Framing 
> stages:
> https://github.com/akka/akka/pull/21539
>
> Keep an eye out for 2.4.12.
>
> -- 
> Konrad `ktoso` Malawski
> Akka  @ Lightbend 
>
> On 19 October 2016 at 16:11:32, vladysla...@rtsmunity.com  (
> vladysla...@rtsmunity.com ) wrote:
>
>
> After some playing with JMC and async boundaries it turned out the hottest 
> methods are related to ByteString splitting and delimiting (for example, 
> scala.collection.IndexedSeqOptimized$class.sameElements(IndexedSeqOptimized, 
> GenIterable) takes 73-75% of processor time). The input file is 45M lines, 
> 2-3-characters each. Is it at all possible to make this work?
> --
> >> Read the docs: http://akka.io/docs/
> >> Check the FAQ: 
> http://doc.akka.io/docs/akka/current/additional/faq.html
> >> Search the archives: https://groups.google.com/group/akka-user
> ---
> You received this message because you are subscribed to the Google Groups 
> "Akka User List" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to akka-user+...@googlegroups.com .
> To post to this group, send email to akka...@googlegroups.com 
> .
> Visit this group at https://groups.google.com/group/akka-user.
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Re: Fetching remote file via akka-http singleRequest results in extremely high allocation rate

2016-10-19 Thread johannes . rudolph 
Using mapConcat instead may even be faster ;)

On Wednesday, October 19, 2016 at 5:05:22 PM UTC+2, 
vladysla...@rtsmunity.com wrote:
>
> Okay, so the issue was really in Framing performance. Changing Framing 
> stage for
>
> flatMapConcat(chunk -> Source.from(Arrays.asList(chunk.split("\n".
>
> actually made it reasonably fast.
>
>
> On Wednesday, October 19, 2016 at 3:25:50 PM UTC+2, 
> vladysla...@rtsmunity.com wrote:
>>
>> Hello,
>>
>> I have the following problem. I'm trying to fetch a file via HTTP in 
>> streaming fashion, processing it line by line (code: 
>> https://gist.github.com/anonymous/c737fa388b55181dcdab9fa6cb8cb2bc), but 
>> it is extremely slow (about a minute to fetch a 150MB file to t2.micro 
>> instance from S3). I guess, something is reallocated for each input line 
>> (input files have very short lines), resulting in chainsaw-like heap size 
>> graph and extremely high allocation rate (JMC screenshot: 
>> http://prnt.sc/cw8si6). What am I doing wrong?
>>
>> Thanks
>>
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Re: Fetching remote file via akka-http singleRequest results in extremely high allocation rate

2016-10-19 Thread vladyslav . chaban 
Okay, so the issue was really in Framing performance. Changing Framing 
stage for

flatMapConcat(chunk -> Source.from(Arrays.asList(chunk.split("\n".

actually made it reasonably fast.


On Wednesday, October 19, 2016 at 3:25:50 PM UTC+2, 
vladysla...@rtsmunity.com wrote:
>
> Hello,
>
> I have the following problem. I'm trying to fetch a file via HTTP in 
> streaming fashion, processing it line by line (code: 
> https://gist.github.com/anonymous/c737fa388b55181dcdab9fa6cb8cb2bc), but 
> it is extremely slow (about a minute to fetch a 150MB file to t2.micro 
> instance from S3). I guess, something is reallocated for each input line 
> (input files have very short lines), resulting in chainsaw-like heap size 
> graph and extremely high allocation rate (JMC screenshot: 
> http://prnt.sc/cw8si6). What am I doing wrong?
>
> Thanks
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Re: Fetching remote file via akka-http singleRequest results in extremely high allocation rate

2016-10-19 Thread johannes . rudolph 
Hi Vladyslav,

this sounds like a worst-case scenario for the Framing stages: 45M lines 
and each line 2-3 characters long will put a lot of pressure on the streams 
infrastructure and the framing stage. It might still make sense to 
benchmark and optimize the Framing stage. One optimization could be to 
collect a few elements in one go (the current implementation only looks for 
one match at a time).

Johannes


On Wednesday, October 19, 2016 at 4:11:29 PM UTC+2, 
vladysla...@rtsmunity.com wrote:
>
>
> After some playing with JMC and async boundaries it turned out the hottest 
> methods are related to ByteString splitting and delimiting (for example, 
> scala.collection.IndexedSeqOptimized$class.sameElements(IndexedSeqOptimized, 
> GenIterable) takes 73-75% of processor time). The input file is 45M lines, 
> 2-3-characters each. Is it at all possible to make this work?
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

Re: [akka-user] Re: Fetching remote file via akka-http singleRequest results in extremely high allocation rate

2016-10-19 Thread Konrad Malawski 
Hi Vladyslav,
firstly, no reason to get desperate :-)
Glad you did some profiling, it would be super helpful next time around if
you share more details about both your code an profile you're observing.
It's hard to give advice what can be improved when given a blank canvas (no
idea what your code looks like).

Of course things can be improved, and in fact just a few days ago we merged
a significant performance improvement that affects the Framing stages:
https://github.com/akka/akka/pull/21539

Keep an eye out for 2.4.12.

-- 
Konrad `ktoso` Malawski
Akka  @ Lightbend 

On 19 October 2016 at 16:11:32, vladyslav.cha...@rtsmunity.com (
vladyslav.cha...@rtsmunity.com) wrote:


After some playing with JMC and async boundaries it turned out the hottest
methods are related to ByteString splitting and delimiting (for example,
scala.collection.IndexedSeqOptimized$class.sameElements(IndexedSeqOptimized,
GenIterable) takes 73-75% of processor time). The input file is 45M lines,
2-3-characters each. Is it at all possible to make this work?
--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ:
http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups
"Akka User List" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Re: Fetching remote file via akka-http singleRequest results in extremely high allocation rate

2016-10-19 Thread vladyslav . chaban 

After some playing with JMC and async boundaries it turned out the hottest 
methods are related to ByteString splitting and delimiting (for example, 
scala.collection.IndexedSeqOptimized$class.sameElements(IndexedSeqOptimized, 
GenIterable) takes 73-75% of processor time). The input file is 45M lines, 
2-3-characters each. Is it at all possible to make this work?

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

[akka-user] Re: Fetching remote file via akka-http singleRequest results in extremely high allocation rate

2016-10-19 Thread vladyslav . chaban 
Akka and Akka-HTTP version is 2.4.11.
On Wednesday, October 19, 2016 at 3:25:50 PM UTC+2, 
vladysla...@rtsmunity.com wrote:
>
> Hello,
>
> I have the following problem. I'm trying to fetch a file via HTTP in 
> streaming fashion, processing it line by line (code: 
> https://gist.github.com/anonymous/c737fa388b55181dcdab9fa6cb8cb2bc), but 
> it is extremely slow (about a minute to fetch a 150MB file to t2.micro 
> instance from S3). I guess, something is reallocated for each input line 
> (input files have very short lines), resulting in chainsaw-like heap size 
> graph and extremely high allocation rate (JMC screenshot: 
> http://prnt.sc/cw8si6). What am I doing wrong?
>
> Thanks
>

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at https://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.