this is exactly the kind of problem code I was describing - there's no
backpressure on existing future tasks to hold up the launching of more
futures - the work done by the agent calling conj is negligible. You need
to control the size of the pool of threads used, and you need to impose
back-pressure.
On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <grzebyta....@gmail.com>
wrote:
> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com> wrote:
>
>> On 31 January 2018 at 17:59, Jacek Grzebyta <grzebyta....@gmail.com>
>> wrote:
>>
>>> I have application with quite intense tripe store populating ~30/40 k
>>> records per chunk (139 portions). The data are wrapped within the future:
>>>
>>> (conj agent (future (apply task args)))
>>>
>>> and that all together is send-off into (agent []).
>>>
>>
>> What is "agent"? The first line of code indicates that it's a local
>> collection shadowing the code function, while the second code snippet
>> indicates that you're using the core agent function.
>>
>> Also why are you sending off to an agent?
>>
>
> I have ~8sec computing task for each input dataset which generates those
> records. After that I write it into disk (in software-specific
> transaction). I just wanted to separate hard computing and io operations. I
> created a side-effect method which is injected together with the dataset
> into a future. The futures are async collected within a list wrapped in
> agent. After the computing the main thread is waiting until all io tasks
> will be finished.
>
>
>>
>> At the end of the main thread function I just use await-for and after
>>> that:
>>>
>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>
>>
> As a control, tasks return number of written records.
>
>
>
>>
>>> For some reason I see the happy collecting (see attached screenshot of
>>> jconsole).
>>>
>>
>> "happy" = "heap"?
>>
>
> Both. As you can see on attached screenshot the heap usage grows easy
> until aver. ~2 1/4 G than keep that for a few minutes. In that moment I
> stopped. After that starts grow till ~4G with tendency to do jumps a bit
> more that 4G.
>
>
>>
>>
>>> After seeing the source code of future I suppose that the memory (data
>>> are kept as #{} set) is not released. The task returns only integer so I do
>>> not think that might cause the problem.
>>>
>>
>> Can you provide more detail? You keep alluding to things that you don't
>> provide code for, such as the sets of data.
>>
>
>
> The code is attached. However the important code is
>
> L123 .
> (let [;; keeps all data loading futures.
> ;; waiting until all futures are finished
> ;; should be done outside the main loop
> data-loading-tasks (agent [])]
>
> L128
> (doseq
> (let [r1 (long operation)] L133
> (doseq
> (let [r2 (v.v. long)] L155
>
> L163 (send-off data-loading-task conj-task)
>
> )
> )
> )
> )
>
>
> I guess first I will move data-loading-tasks list into one of inner lets.
> Also I will create within an injecting function a separate abstract
> function let inside. The task will populate tmp variable which will be
> returned as a future result:
>
>
> L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
> result)))))
>
>
>>
>> --
>> James Reeves
>> booleanknot.com
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
