Thanks Patrik, Well it sounds like everything is heading in the right direction, if it is not already there yet. I'm very excited to see what will be in the next milestone :-D
On 22 April 2014 12:08, Patrik Nordwall <patrik.nordw...@gmail.com> wrote: > > > > On Mon, Apr 21, 2014 at 3:59 PM, Sam Halliday <sam.halli...@gmail.com> > wrote: >>> >>> On Monday, 21 April 2014 12:32:57 UTC+1, Patrik Nordwall wrote: >>>> >>>> I intend to read the documentation fully, but I was a little >>>> disappointed that the activator examples did not have a simple example with >>>> an (effectively) infinite data source that can only be polled in serial, >>>> with parallel (but controllably finite) consumers. >>>> >>>> Isn't that demonstrated with the random number generator source, and its >>>> slow consumers? >>> >>> I missed that one. How many consumers are there at any given moment? >>> >>> It has one consumer but two filter steps that can execute pipelined. You >>> can attach several consumers with toProducer, and then start several flows >>> from that. Backpressure works with multiple consumers also. >> >> >> >> OK great. I did actually see this example and that's not what I mean. I'd >> really like to be able to specify (e.g. in runtime config files) how many >> maximum threads can be running in the "filter(rnd => isPrime(rnd))" block. >> >> Say we want to do the filtering in parallel, using 2 cores. Imagine the >> first random number that we get is really big and takes a few seconds to >> check if it is prime. The second number is "3" and we instantly accept >> it.... it would be preferable if this result were held back until the first >> answer became available, but the free core still goes on to check the third >> number. >> >> Alternatively, I can imagine situations where order does not matter at >> all. This is all considered in the Observable pattern, so I should imagine >> you have also included it :-) >> >> Does that make sense? Would it be tricky to update the primes example in >> this way? > > > This should be possible when we have all operators in place. Please create a > ticket, so that it's not forgotten. > >> >> >> Adding a second flow is a very different thing. I think I'd need to read >> the docs (and source code) in a lot more detail before understanding the >> consequences for a particular Producer (e.g. does it replay from the start, >> is it sending the same results to all flows, are all flows getting the same >> order of events, etc). This is of less interest to me at the moment, but I >> can see it being very important. >> >> >> >>> >>> But Iterator[T] is a little too ill-defined near the end of the stream >>> (that's why I created my own Producer in the RxJava playground). For >>> example, does it block on hasNext or on next if it knows there are more >>> elements that are not yet available, or does it close up and go home? >>> Traditional Java APIs (such as Queues) would actually return early if a >>> queue was exhausted, instead of waiting for a poison pill. In any case, if >>> Flow can handle an Iterator that blocks (e.g. for I/O), it's probably good >>> enough for most situations. >>> >>> Ah, I see what you mean. Blocking hasNext/next doesn't sound attractive >>> to me. That should probably be another Producer, that can do the polling. >>> >> >> >> Now I'm confused whether Producer is a pull or push based source... in the >> examples, I was getting the impression that it was very much a pull based >> API (and would therefore have to block on some level, if data is not >> available yet). Is it also a pusher? > > > Yes. It gets a request from downstream consumer of X more elements. That > doesn't mean that it replies immediately with X elements, but it > asynchronously pushes up to X elements whenever it has them. Exactly how it > "waits" for the elements itself is up to the implementation of the Producer. > >> >> >> The brain scanner project is an example of a pusher source... throttling >> it doesn't make any sense unless it is acceptable to throw results away >> (i.e. not collect them in time). So, yes, you are absolutely correct that a >> blocking Iterator is not good here. > > > Sure, it could be implemented as an actor that periodically poll the device, > but it must still not send more elements downstream than what was requested. > If it can't slow down the polling, it must buffer and finally throw away > elements. > >> >> However, for datasources (e.g. reading from a really big query result over >> a SQL connection), the "next" or "hasNext" in an equivalent Iterator may >> very well block and there is no way to get around this. > > > Async database driver? Yeah, I know there are not many of those, and then > you have to use blocking somewhere. Nothing wrong with that, just keep it > managed. Akka streams are implemented with Actors, and thereby you can use a > dedicated dispatcher for such things. > >> >> Indeed, you will have the same problem with >> Source.fromFile(...).readLines, exaggerated if the file is on a really slow >> hard drive (or a network drive). > > > Yes, there will most likely be special producers for working with files. > > /Patrik > >> >> >> Best regards, >> Sam >> >> -- >> >>>>>>>>>> Read the docs: http://akka.io/docs/ >> >>>>>>>>>> Check the FAQ: >> >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >> >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user >> --- >> You received this message because you are subscribed to the Google Groups >> "Akka User List" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to akka-user+unsubscr...@googlegroups.com. >> >> To post to this group, send email to akka-user@googlegroups.com. >> Visit this group at http://groups.google.com/group/akka-user. >> For more options, visit https://groups.google.com/d/optout. > > > > > -- > > Patrik Nordwall > Typesafe - Reactive apps on the JVM > Twitter: @patriknw > > JOIN US. REGISTER TODAY! > Scala > Days > June 16th-18th, > Berlin > > -- >>>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>>> Check the FAQ: >>>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user > --- > You received this message because you are subscribed to a topic in the > Google Groups "Akka User List" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/akka-user/1TlAy-oqOk8/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > akka-user+unsubscr...@googlegroups.com. > To post to this group, send email to akka-user@googlegroups.com. > Visit this group at http://groups.google.com/group/akka-user. > For more options, visit https://groups.google.com/d/optout. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to akka-user+unsubscr...@googlegroups.com. To post to this group, send email to akka-user@googlegroups.com. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
