Ah, he's using one agent, I see. On Jan 31, 2018 9:15 PM, "John Newman" <john...@gmail.com> wrote:
> Multiple sen-doffs to one agent will serialize it's calls, but spawning > agents on each new task will spawn threads on a bounded thread pool, I > believe. > > On Jan 31, 2018 8:32 PM, "Justin Smith" <noisesm...@gmail.com> wrote: > >> Doing all the actions via one agent means that the actions are serialized >> though - you end up with no performance improvement over doing them all in >> a doseq in one future - the right way to do this tends to be trickier than >> it looks at first glance, and depends on your requirements. agents, the >> claypoole library, and reducers are all potentially useful. If >> parallelization leads to complex coordination needs, core.async can help >> too. >> >> On Wed, Jan 31, 2018 at 5:18 PM John Newman <john...@gmail.com> wrote: >> >>> Agents manage a pool of threads for you. Try doing it without the future >>> call and see if that works (unless you're trying to do something else). >>> >>> John >>> >>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta <grzebyta....@gmail.com> >>> wrote: >>> >>>> Thanks a lot. I will check it tomorrow. >>>> >>>> J >>>> >>>> On 1 Feb 2018 12:12 a.m., "Justin Smith" <noisesm...@gmail.com> wrote: >>>> >>>>> this is exactly the kind of problem code I was describing - there's no >>>>> backpressure on existing future tasks to hold up the launching of more >>>>> futures - the work done by the agent calling conj is negligible. You need >>>>> to control the size of the pool of threads used, and you need to impose >>>>> back-pressure. >>>>> >>>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <grzebyta....@gmail.com> >>>>> wrote: >>>>> >>>>>> On 31 January 2018 at 18:08, James Reeves <ja...@booleanknot.com> >>>>>> wrote: >>>>>> >>>>>>> On 31 January 2018 at 17:59, Jacek Grzebyta <grzebyta....@gmail.com> >>>>>>> wrote: >>>>>>> >>>>>>>> I have application with quite intense tripe store populating ~30/40 >>>>>>>> k records per chunk (139 portions). The data are wrapped within the >>>>>>>> future: >>>>>>>> >>>>>>>> (conj agent (future (apply task args))) >>>>>>>> >>>>>>>> and that all together is send-off into (agent []). >>>>>>>> >>>>>>> >>>>>>> What is "agent"? The first line of code indicates that it's a local >>>>>>> collection shadowing the code function, while the second code snippet >>>>>>> indicates that you're using the core agent function. >>>>>>> >>>>>>> Also why are you sending off to an agent? >>>>>>> >>>>>> >>>>>> I have ~8sec computing task for each input dataset which generates >>>>>> those records. After that I write it into disk (in software-specific >>>>>> transaction). I just wanted to separate hard computing and io >>>>>> operations. I >>>>>> created a side-effect method which is injected together with the dataset >>>>>> into a future. The futures are async collected within a list wrapped in >>>>>> agent. After the computing the main thread is waiting until all io tasks >>>>>> will be finished. >>>>>> >>>>>> >>>>>>> >>>>>>> At the end of the main thread function I just use await-for and >>>>>>>> after that: >>>>>>>> >>>>>>>> (reduce + (map #(deref %) @data-loading-tasks)) >>>>>>>> >>>>>>> >>>>>> As a control, tasks return number of written records. >>>>>> >>>>>> >>>>>> >>>>>>> >>>>>>>> For some reason I see the happy collecting (see attached screenshot >>>>>>>> of jconsole). >>>>>>>> >>>>>>> >>>>>>> "happy" = "heap"? >>>>>>> >>>>>> >>>>>> Both. As you can see on attached screenshot the heap usage grows easy >>>>>> until aver. ~2 1/4 G than keep that for a few minutes. In that moment I >>>>>> stopped. After that starts grow till ~4G with tendency to do jumps a bit >>>>>> more that 4G. >>>>>> >>>>>> >>>>>>> >>>>>>> >>>>>>>> After seeing the source code of future I suppose that the memory >>>>>>>> (data are kept as #{} set) is not released. The task returns only >>>>>>>> integer >>>>>>>> so I do not think that might cause the problem. >>>>>>>> >>>>>>> >>>>>>> Can you provide more detail? You keep alluding to things that you >>>>>>> don't provide code for, such as the sets of data. >>>>>>> >>>>>> >>>>>> >>>>>> The code is attached. However the important code is >>>>>> >>>>>> L123 . >>>>>> (let [;; keeps all data loading futures. >>>>>> ;; waiting until all futures are finished >>>>>> ;; should be done outside the main loop >>>>>> data-loading-tasks (agent [])] >>>>>> >>>>>> L128 >>>>>> (doseq >>>>>> (let [r1 (long operation)] L133 >>>>>> (doseq >>>>>> (let [r2 (v.v. long)] L155 >>>>>> >>>>>> L163 (send-off data-loading-task conj-task) >>>>>> >>>>>> ) >>>>>> ) >>>>>> ) >>>>>> ) >>>>>> >>>>>> >>>>>> I guess first I will move data-loading-tasks list into one of inner >>>>>> lets. Also I will create within an injecting function a separate abstract >>>>>> function let inside. The task will populate tmp variable which will be >>>>>> returned as a future result: >>>>>> >>>>>> >>>>>> L114 (conj agent (future (apply (fn [] (let [result (apply task >>>>>> args)] result))))) >>>>>> >>>>>> >>>>>>> >>>>>>> -- >>>>>>> James Reeves >>>>>>> booleanknot.com >>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Clojure" group. >>>>>>> To post to this group, send email to clojure@googlegroups.com >>>>>>> Note that posts from new members are moderated - please be patient >>>>>>> with your first post. >>>>>>> To unsubscribe from this group, send email to >>>>>>> clojure+unsubscr...@googlegroups.com >>>>>>> For more options, visit this group at >>>>>>> http://groups.google.com/group/clojure?hl=en >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Clojure" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to clojure+unsubscr...@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Clojure" group. >>>>>> To post to this group, send email to clojure@googlegroups.com >>>>>> Note that posts from new members are moderated - please be patient >>>>>> with your first post. >>>>>> To unsubscribe from this group, send email to >>>>>> clojure+unsubscr...@googlegroups.com >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/clojure?hl=en >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Clojure" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to clojure+unsubscr...@googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To post to this group, send email to clojure@googlegroups.com >>>>> Note that posts from new members are moderated - please be patient >>>>> with your first post. >>>>> To unsubscribe from this group, send email to >>>>> clojure+unsubscr...@googlegroups.com >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/clojure?hl=en >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Clojure" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to clojure+unsubscr...@googlegroups.com. >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To post to this group, send email to clojure@googlegroups.com >>>> Note that posts from new members are moderated - please be patient with >>>> your first post. >>>> To unsubscribe from this group, send email to >>>> clojure+unsubscr...@googlegroups.com >>>> For more options, visit this group at >>>> http://groups.google.com/group/clojure?hl=en >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "Clojure" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to clojure+unsubscr...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clojure@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.