[akka-user] Long-running stream with groupBy resource usage

2014-12-06 Thread Denny
Hello,

I have a long-running stream of HTTP requests and I want to group the 
requests by domain. I also want to implement request throttling on a 
per-domain basis. groupBy(_.domain) and then throttle each substream seems 
like the way to go, but I'm worried about the resource usage. There may be 
many thousands of substreams and they will be running over a long period of 
time. Most of them will receive elements only sporadically, in short 
bursts, or maybe just once. Ideally I would like to time out substreams 
that receive no elements for a certain time, and then re-create them when 
new elements for them comes in. But my understanding is that if the 
substream is cancelled it will not be re-created, is that correct? What is 
the recommended way of doing this?

In 0.9 I used the TimerTransformer to implement this myself (without 
groupBy) but it seems like the TimerTransformer is gone from the new 
release.


-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.


Re: [akka-user] Long-running stream with groupBy resource usage

2014-12-11 Thread Björn Antonsson
Hi,

Yes you are right that groupBy is probably not a good solution. The incoming 
connections have already been acepted and if you throttle them by delaying the 
decision about what to do with them, then you are vulnerable to a DOS attack. 
Also you are right that substreams that you cancel will stay cancelled and 
can't be recreated. I have no canned answer for your use case but hopefully 
you'll find inspiration in the cook-book examples that are being written right 
now. The next milestone is a documentation milestone and it will be released 
before Christmas.

The TimerTransformer had unexpected and unwanted behavior so it has been 
removed for now but will be put back in in one form or another in a future 
release.

B/

On 7 December 2014 at 01:32:37, Denny (dennybr...@gmail.com) wrote:

Hello,

I have a long-running stream of HTTP requests and I want to group the requests 
by domain. I also want to implement request throttling on a per-domain basis. 
groupBy(_.domain) and then throttle each substream seems like the way to go, 
but I'm worried about the resource usage. There may be many thousands of 
substreams and they will be running over a long period of time. Most of them 
will receive elements only sporadically, in short bursts, or maybe just once. 
Ideally I would like to time out substreams that receive no elements for a 
certain time, and then re-create them when new elements for them comes in. But 
my understanding is that if the substream is cancelled it will not be 
re-created, is that correct? What is the recommended way of doing this?

In 0.9 I used the TimerTransformer to implement this myself (without groupBy) 
but it seems like the TimerTransformer is gone from the new release.


--
>> Read the docs: http://akka.io/docs/
>> Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>> Search the archives: https://groups.google.com/group/akka-user
---
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.

-- 
Björn Antonsson
Typesafe – Reactive Apps on the JVM
twitter: @bantonsson

-- 
>>  Read the docs: http://akka.io/docs/
>>  Check the FAQ: 
>> http://doc.akka.io/docs/akka/current/additional/faq.html
>>  Search the archives: https://groups.google.com/group/akka-user
--- 
You received this message because you are subscribed to the Google Groups "Akka 
User List" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to akka-user+unsubscr...@googlegroups.com.
To post to this group, send email to akka-user@googlegroups.com.
Visit this group at http://groups.google.com/group/akka-user.
For more options, visit https://groups.google.com/d/optout.