yes, that's the idea exactly
also, you might want more fine grained control of how much parallelism
occurs (eg. if every thread is writing to the same physical device, you can
often get better throughput by not parallelizing at all, or keeping the
parallelism quite limited - it's worth experimenting) - there are good ways
to control that using ThreadPoolExecutor directly, or using
clojure.lang.PersistentQueue/EMPTY as a control construct, or core.async
channels, or ztellman's manifold library, or the claypoole threading library
On Thu, Feb 1, 2018, 03:44 Jacek Grzebyta <grzebyta....@gmail.com> wrote:
> Thanks folks. I see now! It should be a list of agents not list of futures
> within agent..... Also any task sent to a agent is processed within a
> thread anyway so I do not need to add future...
>
> On 1 February 2018 at 02:17, John Newman <john...@gmail.com> wrote:
>
>> Ah, he's using one agent, I see.
>>
>> On Jan 31, 2018 9:15 PM, "John Newman" <john...@gmail.com> wrote:
>>
>>> Multiple sen-doffs to one agent will serialize it's calls, but spawning
>>> agents on each new task will spawn threads on a bounded thread pool, I
>>> believe.
>>>
>>> On Jan 31, 2018 8:32 PM, "Justin Smith" <noisesm...@gmail.com> wrote:
>>>
>>>> Doing all the actions via one agent means that the actions are
>>>> serialized though - you end up with no performance improvement over doing
>>>> them all in a doseq in one future - the right way to do this tends to be
>>>> trickier than it looks at first glance, and depends on your requirements.
>>>> agents, the claypoole library, and reducers are all potentially useful. If
>>>> parallelization leads to complex coordination needs, core.async can help
>>>> too.
>>>>
>>>> On Wed, Jan 31, 2018 at 5:18 PM John Newman <john...@gmail.com> wrote:
>>>>
>>>>> Agents manage a pool of threads for you. Try doing it without the
>>>>> future call and see if that works (unless you're trying to do something
>>>>> else).
>>>>>
>>>>> John
>>>>>
>>>>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <
>>>>> grzebyta....@gmail.com> wrote:
>>>>>
>>>>>> Thanks a lot. I will check it tomorrow.
>>>>>>
>>>>>> J
>>>>>>
>>>>>> On 1 Feb 2018 12:12 a.m., "Justin Smith" <noisesm...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> this is exactly the kind of problem code I was describing - there's
>>>>>>> no backpressure on existing future tasks to hold up the launching of
>>>>>>> more
>>>>>>> futures - the work done by the agent calling conj is negligible. You
>>>>>>> need
>>>>>>> to control the size of the pool of threads used, and you need to impose
>>>>>>> back-pressure.
>>>>>>>
>>>>>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <
>>>>>>> grzebyta....@gmail.com> wrote:
>>>>>>>
>>>>>>>> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> On 31 January 2018 at 17:59, Jacek Grzebyta <
>>>>>>>>> grzebyta....@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> I have application with quite intense tripe store populating
>>>>>>>>>> ~30/40 k records per chunk (139 portions). The data are wrapped
>>>>>>>>>> within the
>>>>>>>>>> future:
>>>>>>>>>>
>>>>>>>>>> (conj agent (future (apply task args)))
>>>>>>>>>>
>>>>>>>>>> and that all together is send-off into (agent []).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What is "agent"? The first line of code indicates that it's a
>>>>>>>>> local collection shadowing the code function, while the second code
>>>>>>>>> snippet
>>>>>>>>> indicates that you're using the core agent function.
>>>>>>>>>
>>>>>>>>> Also why are you sending off to an agent?
>>>>>>>>>
>>>>>>>>
>>>>>>>> I have ~8sec computing task for each input dataset which generates
>>>>>>>> those records. After that I write it into disk (in software-specific
>>>>>>>> transaction). I just wanted to separate hard computing and io
>>>>>>>> operations. I
>>>>>>>> created a side-effect method which is injected together with the
>>>>>>>> dataset
>>>>>>>> into a future. The futures are async collected within a list wrapped in
>>>>>>>> agent. After the computing the main thread is waiting until all io
>>>>>>>> tasks
>>>>>>>> will be finished.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> At the end of the main thread function I just use await-for and
>>>>>>>>>> after that:
>>>>>>>>>>
>>>>>>>>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>>>>>>>>
>>>>>>>>>
>>>>>>>> As a control, tasks return number of written records.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>> For some reason I see the happy collecting (see attached
>>>>>>>>>> screenshot of jconsole).
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "happy" = "heap"?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Both. As you can see on attached screenshot the heap usage grows
>>>>>>>> easy until aver. ~2 1/4 G than keep that for a few minutes. In that
>>>>>>>> moment
>>>>>>>> I stopped. After that starts grow till ~4G with tendency to do jumps a
>>>>>>>> bit
>>>>>>>> more that 4G.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> After seeing the source code of future I suppose that the memory
>>>>>>>>>> (data are kept as #{} set) is not released. The task returns only
>>>>>>>>>> integer
>>>>>>>>>> so I do not think that might cause the problem.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Can you provide more detail? You keep alluding to things that you
>>>>>>>>> don't provide code for, such as the sets of data.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> The code is attached. However the important code is
>>>>>>>>
>>>>>>>> L123 .
>>>>>>>> (let [;; keeps all data loading futures.
>>>>>>>> ;; waiting until all futures are finished
>>>>>>>> ;; should be done outside the main loop
>>>>>>>> data-loading-tasks (agent [])]
>>>>>>>>
>>>>>>>> L128
>>>>>>>> (doseq
>>>>>>>> (let [r1 (long operation)] L133
>>>>>>>> (doseq
>>>>>>>> (let [r2 (v.v. long)] L155
>>>>>>>>
>>>>>>>> L163 (send-off data-loading-task conj-task)
>>>>>>>>
>>>>>>>> )
>>>>>>>> )
>>>>>>>> )
>>>>>>>> )
>>>>>>>>
>>>>>>>>
>>>>>>>> I guess first I will move data-loading-tasks list into one of inner
>>>>>>>> lets. Also I will create within an injecting function a separate
>>>>>>>> abstract
>>>>>>>> function let inside. The task will populate tmp variable which will be
>>>>>>>> returned as a future result:
>>>>>>>>
>>>>>>>>
>>>>>>>> L114 (conj agent (future (apply (fn [] (let [result (apply task
>>>>>>>> args)] result)))))
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> James Reeves
>>>>>>>>> booleanknot.com
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "Clojure" group.
>>>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>>>> Note that posts from new members are moderated - please be patient
>>>>>>>>> with your first post.
>>>>>>>>> To unsubscribe from this group, send email to
>>>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>>>> For more options, visit this group at
>>>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>>>> ---
>>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>>> Groups "Clojure" group.
>>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Clojure" group.
>>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>>> Note that posts from new members are moderated - please be patient
>>>>>>>> with your first post.
>>>>>>>> To unsubscribe from this group, send email to
>>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>>> For more options, visit this group at
>>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>>> ---
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "Clojure" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Clojure" group.
>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>> Note that posts from new members are moderated - please be patient
>>>>>>> with your first post.
>>>>>>> To unsubscribe from this group, send email to
>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Clojure" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>> Note that posts from new members are moderated - please be patient
>>>>>> with your first post.
>>>>>> To unsubscribe from this group, send email to
>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>> Note that posts from new members are moderated - please be patient
>>>>> with your first post.
>>>>> To unsubscribe from this group, send email to
>>>>> clojure+unsubscr...@googlegroups.com
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/clojure?hl=en
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clojure@googlegroups.com
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+unsubscr...@googlegroups.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/clojure?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
