Ah, he's using one agent, I see.
On Jan 31, 2018 9:15 PM, "John Newman" <john...@gmail.com> wrote:
> Multiple sen-doffs to one agent will serialize it's calls, but spawning
> agents on each new task will spawn threads on a bounded thread pool, I
> believe.
>
> On Jan 31, 2018 8:32 PM, "Justin Smith" <noisesm...@gmail.com> wrote:
>
>> Doing all the actions via one agent means that the actions are serialized
>> though - you end up with no performance improvement over doing them all in
>> a doseq in one future - the right way to do this tends to be trickier than
>> it looks at first glance, and depends on your requirements. agents, the
>> claypoole library, and reducers are all potentially useful. If
>> parallelization leads to complex coordination needs, core.async can help
>> too.
>>
>> On Wed, Jan 31, 2018 at 5:18 PM John Newman <john...@gmail.com> wrote:
>>
>>> Agents manage a pool of threads for you. Try doing it without the future
>>> call and see if that works (unless you're trying to do something else).
>>>
>>> John
>>>
>>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <grzebyta....@gmail.com>
>>> wrote:
>>>
>>>> Thanks a lot. I will check it tomorrow.
>>>>
>>>> J
>>>>
>>>> On 1 Feb 2018 12:12 a.m., "Justin Smith" <noisesm...@gmail.com> wrote:
>>>>
>>>>> this is exactly the kind of problem code I was describing - there's no
>>>>> backpressure on existing future tasks to hold up the launching of more
>>>>> futures - the work done by the agent calling conj is negligible. You need
>>>>> to control the size of the pool of threads used, and you need to impose
>>>>> back-pressure.
>>>>>
>>>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <grzebyta....@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On 31 January 2018 at 17:59, Jacek Grzebyta <grzebyta....@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I have application with quite intense tripe store populating ~30/40
>>>>>>>> k records per chunk (139 portions). The data are wrapped within the
>>>>>>>> future:
>>>>>>>>
>>>>>>>> (conj agent (future (apply task args)))
>>>>>>>>
>>>>>>>> and that all together is send-off into (agent []).
>>>>>>>>
>>>>>>>
>>>>>>> What is "agent"? The first line of code indicates that it's a local
>>>>>>> collection shadowing the code function, while the second code snippet
>>>>>>> indicates that you're using the core agent function.
>>>>>>>
>>>>>>> Also why are you sending off to an agent?
>>>>>>>
>>>>>>
>>>>>> I have ~8sec computing task for each input dataset which generates
>>>>>> those records. After that I write it into disk (in software-specific
>>>>>> transaction). I just wanted to separate hard computing and io
>>>>>> operations. I
>>>>>> created a side-effect method which is injected together with the dataset
>>>>>> into a future. The futures are async collected within a list wrapped in
>>>>>> agent. After the computing the main thread is waiting until all io tasks
>>>>>> will be finished.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> At the end of the main thread function I just use await-for and
>>>>>>>> after that:
>>>>>>>>
>>>>>>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>>>>>>
>>>>>>>
>>>>>> As a control, tasks return number of written records.
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>> For some reason I see the happy collecting (see attached screenshot
>>>>>>>> of jconsole).
>>>>>>>>
>>>>>>>
>>>>>>> "happy" = "heap"?
>>>>>>>
>>>>>>
>>>>>> Both. As you can see on attached screenshot the heap usage grows easy
>>>>>> until aver. ~2 1/4 G than keep that for a few minutes. In that moment I
>>>>>> stopped. After that starts grow till ~4G with tendency to do jumps a bit
>>>>>> more that 4G.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> After seeing the source code of future I suppose that the memory
>>>>>>>> (data are kept as #{} set) is not released. The task returns only
>>>>>>>> integer
>>>>>>>> so I do not think that might cause the problem.
>>>>>>>>
>>>>>>>
>>>>>>> Can you provide more detail? You keep alluding to things that you
>>>>>>> don't provide code for, such as the sets of data.
>>>>>>>
>>>>>>
>>>>>>
>>>>>> The code is attached. However the important code is
>>>>>>
>>>>>> L123 .
>>>>>> (let [;; keeps all data loading futures.
>>>>>> ;; waiting until all futures are finished
>>>>>> ;; should be done outside the main loop
>>>>>> data-loading-tasks (agent [])]
>>>>>>
>>>>>> L128
>>>>>> (doseq
>>>>>> (let [r1 (long operation)] L133
>>>>>> (doseq
>>>>>> (let [r2 (v.v. long)] L155
>>>>>>
>>>>>> L163 (send-off data-loading-task conj-task)
>>>>>>
>>>>>> )
>>>>>> )
>>>>>> )
>>>>>> )
>>>>>>
>>>>>>
>>>>>> I guess first I will move data-loading-tasks list into one of inner
>>>>>> lets. Also I will create within an injecting function a separate abstract
>>>>>> function let inside. The task will populate tmp variable which will be
>>>>>> returned as a future result:
>>>>>>
>>>>>>
>>>>>> L114 (conj agent (future (apply (fn [] (let [result (apply task
>>>>>> args)] result)))))
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> James Reeves
>>>>>>> booleanknot.com
>>>>>>>
>>>>>>> --
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Clojure" group.
>>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>>> Note that posts from new members are moderated - please be patient
>>>>>>> with your first post.
>>>>>>> To unsubscribe from this group, send email to
>>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>>> For more options, visit this group at
>>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>>> ---
>>>>>>> You received this message because you are subscribed to the Google
>>>>>>> Groups "Clojure" group.
>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>>
>>>>>> --
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>>> Note that posts from new members are moderated - please be patient
>>>>>> with your first post.
>>>>>> To unsubscribe from this group, send email to
>>>>>> clojure+unsubscr...@googlegroups.com
>>>>>> For more options, visit this group at
>>>>>> http://groups.google.com/group/clojure?hl=en
>>>>>> ---
>>>>>> You received this message because you are subscribed to the Google
>>>>>> Groups "Clojure" group.
>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>> send an email to clojure+unsubscr...@googlegroups.com.
>>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>>
>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To post to this group, send email to clojure@googlegroups.com
>>>>> Note that posts from new members are moderated - please be patient
>>>>> with your first post.
>>>>> To unsubscribe from this group, send email to
>>>>> clojure+unsubscr...@googlegroups.com
>>>>> For more options, visit this group at
>>>>> http://groups.google.com/group/clojure?hl=en
>>>>> ---
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "Clojure" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>>> For more options, visit https://groups.google.com/d/optout.
>>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clojure@googlegroups.com
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+unsubscr...@googlegroups.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/clojure?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
