Re: [akka-user] [akka-streams]: actor producers, load balancers
One thing unanswered from the previous email: Is it in the spirit of akka-stream/reactive streams to implement your own producers? Or should all producers (publishers) be created by the framework? In theory reactive streams aim to be usable between frameworks - so an Rx Producer would be consumable in akka-streams and vice-versa. So, yes it's ok to provide your own Producers (like from a database etc). Internally we have some more abstractions which make implementing these easier, but I'm not aware of plans of exposing them any time soon - for starters we need to update akka-streams to the updated reactive-streams interfaces. -- Cheers, Konrad 'ktoso' Malawski hAkker - Typesafe, Inc http://scaladays.org -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] [akka-streams]: actor producers, load balancers
On Monday, May 26, 2014 2:25:52 PM UTC+2, Konrad Malawski wrote: One thing unanswered from the previous email: Is it in the spirit of akka-stream/reactive streams to implement your own producers? Or should all producers (publishers) be created by the framework? In theory reactive streams aim to be usable between frameworks - so an Rx Producer would be consumable in akka-streams and vice-versa. So, yes it's ok to provide your own Producers (like from a database etc). Internally we have some more abstractions which make implementing these easier, but I'm not aware of plans of exposing them any time soon - for starters we need to update akka-streams to the updated reactive-streams interfaces. I see, but then I have to implement the subscription management etc myself - which I guess can be tricky ;) But why not try. Thanks for the help! Adam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] [akka-streams]: actor producers, load balancers
Hi Adam - is it reasonable (thinking about reactive streams in general) to have an actor which produces elements on-demand (instead of providing a collection/iterator/() = as is currently supported)? As far as I understand the current implementation, subscribers explicitly ask publishers for more elements (through Subscription.requestMore) - so it seems it would be possible to pass such a request to an actor and ask for the given amount of elements. Is there any chance to get actor producers in some future releases, or there are no such plans currently? There is already one bridge to build consumers based on actors: https://github.com/akka/akka/pull/15214 The producer side is still missing, but there will be something to support that as well. - another thing is if the streams are thought to be more local, or remote as well? There's currently the TCP stream implementation, which I guess would indicate remote as well (and in such scenarios the need for backpressure arises quite naturally, maybe even more than in locally), but do you plan to develop this somehow? Streaming through TCP is already remote, so in that sense it is already supported. Of course you need to open up the connections yourself now, and there is no registry or naming functionality to obtain remote stream endpoints. E.g. when there would be multiple consumers for a single producer, a useful component would be a load-balancer which takes into account the backpressure information. All stream components take into account backpressure signal independently of fan-in or fan-out behavior, so this is by default available. (we do not have balancing elements yet, but they will be there eventually. Workaround can be a groupBy assigning to a fixed number of output groups randomly, but this is a bit dangerous field). From the stream viewpoint it does not matter whether the output streams are local or TCP connections, the balancing should work exactly the same way. -Endre Thanks! -- Adam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] [akka-streams]: actor producers, load balancers
Hey Adam, Patrik has right now opened a ticket and started exposing the ActorProducer abstraction :-) https://github.com/akka/akka/issues/15288 This will help a lot in implementing external producers. :-) On Mon, May 26, 2014 at 2:52 PM, Endre Varga endre.va...@typesafe.comwrote: Hi Adam - is it reasonable (thinking about reactive streams in general) to have an actor which produces elements on-demand (instead of providing a collection/iterator/() = as is currently supported)? As far as I understand the current implementation, subscribers explicitly ask publishers for more elements (through Subscription.requestMore) - so it seems it would be possible to pass such a request to an actor and ask for the given amount of elements. Is there any chance to get actor producers in some future releases, or there are no such plans currently? There is already one bridge to build consumers based on actors: https://github.com/akka/akka/pull/15214 The producer side is still missing, but there will be something to support that as well. - another thing is if the streams are thought to be more local, or remote as well? There's currently the TCP stream implementation, which I guess would indicate remote as well (and in such scenarios the need for backpressure arises quite naturally, maybe even more than in locally), but do you plan to develop this somehow? Streaming through TCP is already remote, so in that sense it is already supported. Of course you need to open up the connections yourself now, and there is no registry or naming functionality to obtain remote stream endpoints. E.g. when there would be multiple consumers for a single producer, a useful component would be a load-balancer which takes into account the backpressure information. All stream components take into account backpressure signal independently of fan-in or fan-out behavior, so this is by default available. (we do not have balancing elements yet, but they will be there eventually. Workaround can be a groupBy assigning to a fixed number of output groups randomly, but this is a bit dangerous field). From the stream viewpoint it does not matter whether the output streams are local or TCP connections, the balancing should work exactly the same way. -Endre Thanks! -- Adam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, Konrad 'ktoso' Malawski hAkker - Typesafe, Inc http://scaladays.org -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] [akka-streams]: actor producers, load balancers
- another thing is if the streams are thought to be more local, or remote as well? There's currently the TCP stream implementation, which I guess would indicate remote as well (and in such scenarios the need for backpressure arises quite naturally, maybe even more than in locally), but do you plan to develop this somehow? Streaming through TCP is already remote, so in that sense it is already supported. Of course you need to open up the connections yourself now, and there is no registry or naming functionality to obtain remote stream endpoints. Right, but I guess naming/lookup is out of scope for akka-stream? I guess I was asking whether your initial focus is going to be on local or remote, but I think Konrad already answered that :) E.g. when there would be multiple consumers for a single producer, a useful component would be a load-balancer which takes into account the backpressure information. All stream components take into account backpressure signal independently of fan-in or fan-out behavior, so this is by default available. (we do not have balancing elements yet, but they will be there eventually. Workaround can be a groupBy assigning to a fixed number of output groups randomly, but this is a bit dangerous field). From the stream viewpoint it does not matter whether the output streams are local or TCP connections, the balancing should work exactly the same way. So right now you have broadcast communication implemented when there are multiple consumers. The other pattern is point-to-point, where each element goes to one of the attached consumers - where load-balancing would make sense (as you have the demand information, you could do very good guesses :) ). Anyway, I'll be watching the developments here :) Also with Actor producers, I think it would be quite easy to implement such a load-balancer: for each consumer, you would have an actor-producer, which would report the demand to a parent actor, where the load-balancing would take place. Thanks for the info! Adam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] [akka-streams]: actor producers, load balancers
Awesome, subscribed :) Thanks, Adam On Monday, May 26, 2014 3:04:30 PM UTC+2, Konrad Malawski wrote: Hey Adam, Patrik has right now opened a ticket and started exposing the ActorProducer abstraction :-) https://github.com/akka/akka/issues/15288 This will help a lot in implementing external producers. :-) On Mon, May 26, 2014 at 2:52 PM, Endre Varga endre...@typesafe.comjavascript: wrote: Hi Adam - is it reasonable (thinking about reactive streams in general) to have an actor which produces elements on-demand (instead of providing a collection/iterator/() = as is currently supported)? As far as I understand the current implementation, subscribers explicitly ask publishers for more elements (through Subscription.requestMore) - so it seems it would be possible to pass such a request to an actor and ask for the given amount of elements. Is there any chance to get actor producers in some future releases, or there are no such plans currently? There is already one bridge to build consumers based on actors: https://github.com/akka/akka/pull/15214 The producer side is still missing, but there will be something to support that as well. - another thing is if the streams are thought to be more local, or remote as well? There's currently the TCP stream implementation, which I guess would indicate remote as well (and in such scenarios the need for backpressure arises quite naturally, maybe even more than in locally), but do you plan to develop this somehow? Streaming through TCP is already remote, so in that sense it is already supported. Of course you need to open up the connections yourself now, and there is no registry or naming functionality to obtain remote stream endpoints. E.g. when there would be multiple consumers for a single producer, a useful component would be a load-balancer which takes into account the backpressure information. All stream components take into account backpressure signal independently of fan-in or fan-out behavior, so this is by default available. (we do not have balancing elements yet, but they will be there eventually. Workaround can be a groupBy assigning to a fixed number of output groups randomly, but this is a bit dangerous field). From the stream viewpoint it does not matter whether the output streams are local or TCP connections, the balancing should work exactly the same way. -Endre Thanks! -- Adam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com javascript:. To post to this group, send email to akka...@googlegroups.comjavascript: . Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+...@googlegroups.com javascript:. To post to this group, send email to akka...@googlegroups.comjavascript: . Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Cheers, Konrad 'ktoso' Malawski hAkker - Typesafe, Inc http://scaladays.org -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
[akka-user] [akka-streams]: actor producers, load balancers
Hello, I've been looking at the akka-streams, and got two questions: - is it reasonable (thinking about reactive streams in general) to have an actor which produces elements on-demand (instead of providing a collection/iterator/() = as is currently supported)? As far as I understand the current implementation, subscribers explicitly ask publishers for more elements (through Subscription.requestMore) - so it seems it would be possible to pass such a request to an actor and ask for the given amount of elements. Is there any chance to get actor producers in some future releases, or there are no such plans currently? - another thing is if the streams are thought to be more local, or remote as well? There's currently the TCP stream implementation, which I guess would indicate remote as well (and in such scenarios the need for backpressure arises quite naturally, maybe even more than in locally), but do you plan to develop this somehow? E.g. when there would be multiple consumers for a single producer, a useful component would be a load-balancer which takes into account the backpressure information. Thanks! -- Adam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] [akka-streams]: actor producers, load balancers
Cześć Adam :-) - is it reasonable (thinking about reactive streams in general) to have an actor which produces elements on-demand (instead of providing a collection/iterator/() = as is currently supported)? As far as I understand the current implementation, subscribers explicitly ask publishers for more elements (through Subscription.requestMore) - so it seems it would be possible to pass such a request to an actor and ask for the given amount of elements. Is there any chance to get actor producers in some future releases, or there are no such plans currently? Yes, definitely! We currently do support it (on release-2.3-dev, it’s pretty new) via: ``` /** * Define the sequence of elements to be produced by the given closure. * The stream ends normally when evaluation of the closure results in * a [[akka.stream.Stop]] exception being thrown; it ends exceptionally * when any other exception is thrown. */ def apply[T](f: () ⇒ T): Flow[T] ``` Which generates an `Actor` backed producer for you (that will call your function), or if you need complete control you can implement a `Producer[T]` and give it to `Flow`: ``` /** * Construct a transformation of the given producer. The transformation steps * are executed by a series of [[org.reactivestreams.api.Processor]] instances * that mediate the flow of elements downstream and the propagation of * back-pressure upstream. */ def apply[T](producer: Producer[T]): Flow[T] ``` These should be enough to implement what you’re after. Disclaimer Please note that the spec ( https://github.com/reactive-streams/reactive-streams ) is under heavy discussions and development at this moment. Our current impl is still targeting the previous version, differences include for example dropping the Producer interface in favour of only keeping `Publisher` etc. Also known as: This is still is changing a lot :-) - another thing is if the streams are thought to be more local, or remote as well? There's currently the TCP stream implementation, which I guess would indicate remote as well (and in such scenarios the need for backpressure arises quite naturally, maybe even more than in locally), but do you plan to develop this somehow? E.g. when there would be multiple consumers for a single producer, a useful component would be a load-balancer which takes into account the backpressure information. We’re currently focused on in-jvm implementations, though multi-language-and-runtime are definitely on the reactive-streams’ radar: https://github.com/reactive-streams/reactive-streams/issues/45 Let’s first nail the in-vm implementation to then move on to the bigger picture (personal opinion here), but there’s so many people involved and loads of excitement around it, so we’ll see ;-) As for Akka, we’re currently mostly focused on getting akka-http (which will be stream based) out of the door, and optimise it, the rest comes next. I hope this helps! // So... what Producer are you implementing? :-) -- Konrad 'ktoso' Malawski hAkker @ typesafe -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] [akka-streams]: actor producers, load balancers
On Friday, May 23, 2014 4:57:32 PM UTC+2, Konrad Malawski wrote: Cześć Adam :-) - is it reasonable (thinking about reactive streams in general) to have an actor which produces elements on-demand (instead of providing a collection/iterator/() = as is currently supported)? As far as I understand the current implementation, subscribers explicitly ask publishers for more elements (through Subscription.requestMore) - so it seems it would be possible to pass such a request to an actor and ask for the given amount of elements. Is there any chance to get actor producers in some future releases, or there are no such plans currently? Yes, definitely! We currently do support it (on release-2.3*-dev*, it’s pretty new) via: ``` /** * Define the sequence of elements to be produced by the given closure. * The stream ends normally when evaluation of the closure results in * a [[akka.stream.Stop]] exception being thrown; it ends exceptionally * when any other exception is thrown. */ def apply[T](f: () ⇒ T): Flow[T] ``` Which generates an `Actor` backed producer for you (that will call your function), or if you need complete control you can implement a `Producer[T]` and give it to `Flow`: Is it in the spirit of akka-stream/reactive streams to implement your own producers? Or should all producers (publishers) be created by the framework? *Disclaimer* Please note that the spec ( https://github.com/reactive-streams/reactive-streams ) is under heavy discussions and development at this moment. Our current impl is still targeting the previous version, differences include for example dropping the Producer interface in favour of only keeping `Publisher` etc. Ah, good to know, I was trying to understand today what is exactly the difference between Producer and Publisher and why do you need that distinction ;) - another thing is if the streams are thought to be more local, or remote as well? There's currently the TCP stream implementation, which I guess would indicate remote as well (and in such scenarios the need for backpressure arises quite naturally, maybe even more than in locally), but do you plan to develop this somehow? E.g. when there would be multiple consumers for a single producer, a useful component would be a load-balancer which takes into account the backpressure information. We’re currently focused on in-jvm implementations, though multi-language-and-runtime are definitely on the reactive-streams’ radar: https://github.com/reactive-streams/reactive-streams/issues/45https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Freactive-streams%2Freactive-streams%2Fissues%2F45sa=Dsntz=1usg=AFQjCNEPPvBVRxCWoDYTDt7ggdEt1ljBbA Let’s first nail the in-vm implementation to then move on to the bigger picture (personal opinion here), but there’s so many people involved and loads of excitement around it, so we’ll see ;-) Sure, it's good to focus on one goal initially, thought the initial akka-streams does contain TCP based streams, which is an open invitation to use it ;) As for Akka, we’re currently mostly focused on getting akka-http (which will be stream based) out of the door, and optimise it, the rest comes next. stream-based as in reactive-stream-based? I hope this helps! // So... what Producer are you implementing? :-) Just playing around ... to rule the world, as always ;) Adam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
Re: [akka-user] [akka-streams]: actor producers, load balancers
Which generates an `Actor` backed producer for you (that will call your function), or if you need complete control you can implement a `Producer[T]` and give it to `Flow`: I don't think that is enough. It assumes that the elements are available when calling the closure, otherwise blocking will happen. We have one ticket https://github.com/akka/akka/issues/15135 for creating a producer from a closure that returns a future. I think we should have support for a producer that is an actor also. Stay tuned. Right, I was thinking about a Future-based producer initially as well (I could then use ? to get the data from the actor), but then I thought it would be actually more efficient if I knew how many elements I can produce in the actor. Adam -- Read the docs: http://akka.io/docs/ Check the FAQ: http://doc.akka.io/docs/akka/current/additional/faq.html Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups Akka User List group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
