On Thursday, October 9, 2014 2:22:50 PM UTC+2, Nahuel Greco wrote:
>
> Fluid: as you said, backpressure on the outgoing channel will be
> sufficient to avoid producer to overrun consumer (without using any extra
> ack-channel), but the producer will compute a new not-yet-used value before
> blocking on sending it to consumer. That's what I want to avoid.
>
You can avoid this very easily. Everytime before the producer expensively
calculates its next value to put it on the channel, make it put a cheaply
calculated value, e. g. the keyword :useless, on the channel instead.
Now on the consumer side you can do this:
0. Blocking (immediate take): Consume the keyword :useless = The producer
will begin production
1. *This is what you want from peek! *Blocking take = Waiting for the
producer to have produced the expensive value, consume it *but the producer
won't calculate the next value (because he puts the :useless keyword which
won't be consumed until later)*
2. Wait for external service to be ready, put value there
3. Go to step 1 trigger production of the next value
Assuming the producer and the consumer run on different threads, what have
you won by introducing step 0?
1. The producer thread can't begin to work until the consumer thread did
his work
2. The consumer thread can't begin to work until the producer thread did
his work
= At every point in time only one thread performs. Entirely :useless
multi-threading.
I can also use alts! on the producer to check if he can write to the
> channel before computing the value, [...]
>
You can't
> [...]
>
> Fregal: If middleman is managing all the communication with the external
> service for *multiple* producers, then it can't be reduced to a simple
> synchronous function call from a producer, hence the necessity of
> core.async (the alternative is to descend to locks world).
>
If I understand you right, you would want to start production of multiple
values in parallel, wait for the first to be ready, then put that to the
external service and loop. Here core.async can be of help. E. g.
(defn exp-calcs
[]
#{(a/thread calc-exp-val-1)
(a/thread calc-exp-val-2)
(a/thread calc-exp-val-3)
(a/thread calc-exp-val-4)
(a/thread calc-exp-val-5)
; ...
})
(loop [chans (exp-calcs)]
(let [[val ch] (a/alts!! (seq exp-calcs))]
;; wait for external service, put val there
;; also have the cancellation ch in chans and use it to guard the
;; following form
(let [chans (disj chans ch)]
(recur (if (seq chans)
chans
(exp-calcs))))))
Again, no peek required.
>
>
>
> Saludos,
> Nahuel Greco.
>
> On Thu, Oct 9, 2014 at 9:07 AM, Fergal Byrne <fergalby...@gmail.com
> <javascript:>> wrote:
>
>> Hi Nahuel,
>>
>> I think it's worth stepping back from discussing solutions and have a
>> look at the problem. If I'm reading things right you need the following:
>>
>> 1. Producer produces values one at a time - should not step until last
>> value has been handled correctly elsewhere.
>> 2. Middleman needs to consume one value and (potentially) retry an
>> external service until it can verify correct handling.
>> 3. Producer should not interpret the consumption of a value as a signal
>> to proceed.
>>
>> You also seem to want the following:
>>
>> a. Use core.async
>> b. Make the solution "elegant"
>>
>> That's all fine, but it seems like a bit of a golden hammer. Your problem
>> is precisely not the kind of problem core.async channels are designed to
>> solve. Channels are there to provide decoupling of producer and consumer,
>> whereas you, in fact, need the kind of coupling provide by function calls.
>>
>> Channels are for when you don't want to know how or when your product is
>> handled. You're finished, you throw the product on a channel, and you move
>> on to making your next product. All you ever find out is if/that your
>> product has been picked off the channel.
>>
>> Function calls are for when you do need to know what happened to your
>> product, and when it has all happened, because you can't move on until the
>> last thing has been done.
>>
>> Just call the middleman in synchronous code in your producer, with one
>> value, and look at its return value to tell you what to do next. You get
>> your blocking semantics (1-3) and you also get information about success or
>> failure.
>>
>> Using channels, consumption is the only information the producer hears
>> back. To tell the producer you've completed your mission as middleman,
>> you'll need another channel. If you really want to use core.async:
>>
>> (>! c v) ; blocks until consumed
>> (respond-to (<! ackchan)) ; blocks until handled
>> ...do expensive stuff
>>
>> but this is just an expensive function call.
>>
>> Regards,
>>
>> Fergal Byrne
>>
>>
>> On Thu, Oct 9, 2014 at 11:16 AM, Eduard Bondarenko <edb...@gmail.com
>> <javascript:>> wrote:
>>
>>> I think the point is to not generate values by producer if external
>>> service is not available
>>> "The producer only unparks when the value is effectively consumed by
>>> the external service. That's my objective."
>>>
>>> "external ready" channel here serves as a latch that stops producer
>>> generating value.
>>>
>>> On Thu, Oct 9, 2014 at 6:39 AM, Fluid Dynamics <a209...@trbvm.com
>>> <javascript:>> wrote:
>>> > On Monday, October 6, 2014 9:36:59 AM UTC-4, edbond wrote:
>>> >>
>>> >> Add one more chan, "external ready".
>>> >> Put :ok there to let producer generate new value.
>>> >>
>>> >> producer:
>>> >> - read from "external ready"
>>> >> - generate value
>>> >> - put into "outgoing" chan
>>> >>
>>> >> client:
>>> >> - contact external server, put in "external ready" if ok
>>> >> - read from "outgoing" chan
>>> >> - send to external
>>> >>
>>> >> Handle exceptions and loop where you need.
>>> >
>>> >
>>> > If you're not reading from "outgoing" until the external server is
>>> known to
>>> > be ready, you don't need the "external ready" channel. Backpressure on
>>> the
>>> > "outgoing" channel will suffice to keep the producer from overrunning
>>> the
>>> > consumer in this case.
>>> >
>>> > --
>>> > You received this message because you are subscribed to the Google
>>> > Groups "Clojure" group.
>>> > To post to this group, send email to clo...@googlegroups.com
>>> <javascript:>
>>> > Note that posts from new members are moderated - please be patient
>>> with your
>>> > first post.
>>> > To unsubscribe from this group, send email to
>>> > clojure+u...@googlegroups.com <javascript:>
>>> > For more options, visit this group at
>>> > http://groups.google.com/group/clojure?hl=en
>>> > ---
>>> > You received this message because you are subscribed to a topic in the
>>> > Google Groups "Clojure" group.
>>> > To unsubscribe from this topic, visit
>>> > https://groups.google.com/d/topic/clojure/QbiwXYDw6oA/unsubscribe.
>>> > To unsubscribe from this group and all its topics, send an email to
>>> > clojure+u...@googlegroups.com <javascript:>.
>>> > For more options, visit https://groups.google.com/d/optout.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clo...@googlegroups.com
>>> <javascript:>
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+u...@googlegroups.com <javascript:>
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+u...@googlegroups.com <javascript:>.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>>
>> --
>>
>> Fergal Byrne, Brenter IT
>>
>> http://inbits.com - Better Living through Thoughtful Technology
>> http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne
>>
>> Founder of Clortex: HTM in Clojure -
>> https://github.com/nupic-community/clortex
>>
>> Author, Real Machine Intelligence with Clortex and NuPIC
>> Read for free or buy the book at https://leanpub.com/realsmartmachines
>>
>> Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014:
>> http://euroclojure.com/2014/
>> and at LambdaJam Chicago, July 2014: http://www.lambdajam.com
>>
>> e:fergalby...@gmail.com <javascript:> t:+353 83 4214179
>> Join the quest for Machine Intelligence at http://numenta.org
>> Formerly of Adnet edi...@adnet.ie <javascript:> http://www.adnet.ie
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com
>> <javascript:>
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <javascript:>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com <javascript:>.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
