Re: what does future do after fn finish ?
On 31 January 2018 at 17:59, Jacek Grzebyta wrote: > I have application with quite intense tripe store populating ~30/40 k > records per chunk (139 portions). The data are wrapped within the future: > > (conj agent (future (apply task args))) > > and that all together is send-off into (agent []). > What is "agent"? The first line of code indicates that it's a local collection shadowing the code function, while the second code snippet indicates that you're using the core agent function. Also why are you sending off to an agent? At the end of the main thread function I just use await-for and after that: > > (reduce + (map #(deref %) @data-loading-tasks)) > > For some reason I see the happy collecting (see attached screenshot of > jconsole). > "happy" = "heap"? > After seeing the source code of future I suppose that the memory (data are > kept as #{} set) is not released. The task returns only integer so I do not > think that might cause the problem. > Can you provide more detail? You keep alluding to things that you don't provide code for, such as the sets of data. -- James Reeves booleanknot.com -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: what does future do after fn finish ?
As a shot in the dark, a common problem with memory usage and futures that I have seen is the antipattern of launching a future for each piece of data in a collection. The problem that occurs is that the code works for small input collections and a small load of running tasks / requests, but for a larger input and more requests it uses up the heap easily. Then, a dev assumes that somehow the future itself is leaking or otherwise taking up space it shouldn't, when the true problem is that the undbounded creation of futures happens to exhaust your available memory space. If you aren't doing such a thing feel free to disregard. On Wed, Jan 31, 2018 at 10:09 AM James Reeves wrote: > On 31 January 2018 at 17:59, Jacek Grzebyta > wrote: > >> I have application with quite intense tripe store populating ~30/40 k >> records per chunk (139 portions). The data are wrapped within the future: >> >> (conj agent (future (apply task args))) >> >> and that all together is send-off into (agent []). >> > > What is "agent"? The first line of code indicates that it's a local > collection shadowing the code function, while the second code snippet > indicates that you're using the core agent function. > > Also why are you sending off to an agent? > > At the end of the main thread function I just use await-for and after that: >> >> (reduce + (map #(deref %) @data-loading-tasks)) >> >> For some reason I see the happy collecting (see attached screenshot of >> jconsole). >> > > "happy" = "heap"? > > >> After seeing the source code of future I suppose that the memory (data >> are kept as #{} set) is not released. The task returns only integer so I do >> not think that might cause the problem. >> > > Can you provide more detail? You keep alluding to things that you don't > provide code for, such as the sets of data. > > -- > James Reeves > booleanknot.com > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: what does future do after fn finish ?
On 31 January 2018 at 18:08, James Reeves wrote: > On 31 January 2018 at 17:59, Jacek Grzebyta > wrote: > >> I have application with quite intense tripe store populating ~30/40 k >> records per chunk (139 portions). The data are wrapped within the future: >> >> (conj agent (future (apply task args))) >> >> and that all together is send-off into (agent []). >> > > What is "agent"? The first line of code indicates that it's a local > collection shadowing the code function, while the second code snippet > indicates that you're using the core agent function. > > Also why are you sending off to an agent? > I have ~8sec computing task for each input dataset which generates those records. After that I write it into disk (in software-specific transaction). I just wanted to separate hard computing and io operations. I created a side-effect method which is injected together with the dataset into a future. The futures are async collected within a list wrapped in agent. After the computing the main thread is waiting until all io tasks will be finished. > > At the end of the main thread function I just use await-for and after that: >> >> (reduce + (map #(deref %) @data-loading-tasks)) >> > As a control, tasks return number of written records. > >> For some reason I see the happy collecting (see attached screenshot of >> jconsole). >> > > "happy" = "heap"? > Both. As you can see on attached screenshot the heap usage grows easy until aver. ~2 1/4 G than keep that for a few minutes. In that moment I stopped. After that starts grow till ~4G with tendency to do jumps a bit more that 4G. > > >> After seeing the source code of future I suppose that the memory (data >> are kept as #{} set) is not released. The task returns only integer so I do >> not think that might cause the problem. >> > > Can you provide more detail? You keep alluding to things that you don't > provide code for, such as the sets of data. > The code is attached. However the important code is L123 . (let [;; keeps all data loading futures. ;; waiting until all futures are finished ;; should be done outside the main loop data-loading-tasks (agent [])] L128 (doseq (let [r1 (long operation)] L133 (doseq (let [r2 (v.v. long)] L155 L163 (send-off data-loading-task conj-task) ) ) ) ) I guess first I will move data-loading-tasks list into one of inner lets. Also I will create within an injecting function a separate abstract function let inside. The task will populate tmp variable which will be returned as a future result: L114 (conj agent (future (apply (fn [] (let [result (apply task args)] result) > > -- > James Reeves > booleanknot.com > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. experiment_result.clj Description: Binary data
Re: what does future do after fn finish ?
this is exactly the kind of problem code I was describing - there's no backpressure on existing future tasks to hold up the launching of more futures - the work done by the agent calling conj is negligible. You need to control the size of the pool of threads used, and you need to impose back-pressure. On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta wrote: > On 31 January 2018 at 18:08, James Reeves wrote: > >> On 31 January 2018 at 17:59, Jacek Grzebyta >> wrote: >> >>> I have application with quite intense tripe store populating ~30/40 k >>> records per chunk (139 portions). The data are wrapped within the future: >>> >>> (conj agent (future (apply task args))) >>> >>> and that all together is send-off into (agent []). >>> >> >> What is "agent"? The first line of code indicates that it's a local >> collection shadowing the code function, while the second code snippet >> indicates that you're using the core agent function. >> >> Also why are you sending off to an agent? >> > > I have ~8sec computing task for each input dataset which generates those > records. After that I write it into disk (in software-specific > transaction). I just wanted to separate hard computing and io operations. I > created a side-effect method which is injected together with the dataset > into a future. The futures are async collected within a list wrapped in > agent. After the computing the main thread is waiting until all io tasks > will be finished. > > >> >> At the end of the main thread function I just use await-for and after >>> that: >>> >>> (reduce + (map #(deref %) @data-loading-tasks)) >>> >> > As a control, tasks return number of written records. > > > >> >>> For some reason I see the happy collecting (see attached screenshot of >>> jconsole). >>> >> >> "happy" = "heap"? >> > > Both. As you can see on attached screenshot the heap usage grows easy > until aver. ~2 1/4 G than keep that for a few minutes. In that moment I > stopped. After that starts grow till ~4G with tendency to do jumps a bit > more that 4G. > > >> >> >>> After seeing the source code of future I suppose that the memory (data >>> are kept as #{} set) is not released. The task returns only integer so I do >>> not think that might cause the problem. >>> >> >> Can you provide more detail? You keep alluding to things that you don't >> provide code for, such as the sets of data. >> > > > The code is attached. However the important code is > > L123 . > (let [;; keeps all data loading futures. > ;; waiting until all futures are finished > ;; should be done outside the main loop > data-loading-tasks (agent [])] > > L128 > (doseq > (let [r1 (long operation)] L133 > (doseq > (let [r2 (v.v. long)] L155 > > L163 (send-off data-loading-task conj-task) > > ) > ) > ) > ) > > > I guess first I will move data-loading-tasks list into one of inner lets. > Also I will create within an injecting function a separate abstract > function let inside. The task will populate tmp variable which will be > returned as a future result: > > > L114 (conj agent (future (apply (fn [] (let [result (apply task args)] > result) > > >> >> -- >> James Reeves >> booleanknot.com >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send e
Re: what does future do after fn finish ?
Thanks a lot. I will check it tomorrow. J On 1 Feb 2018 12:12 a.m., "Justin Smith" wrote: > this is exactly the kind of problem code I was describing - there's no > backpressure on existing future tasks to hold up the launching of more > futures - the work done by the agent calling conj is negligible. You need > to control the size of the pool of threads used, and you need to impose > back-pressure. > > On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta > wrote: > >> On 31 January 2018 at 18:08, James Reeves wrote: >> >>> On 31 January 2018 at 17:59, Jacek Grzebyta >>> wrote: >>> I have application with quite intense tripe store populating ~30/40 k records per chunk (139 portions). The data are wrapped within the future: (conj agent (future (apply task args))) and that all together is send-off into (agent []). >>> >>> What is "agent"? The first line of code indicates that it's a local >>> collection shadowing the code function, while the second code snippet >>> indicates that you're using the core agent function. >>> >>> Also why are you sending off to an agent? >>> >> >> I have ~8sec computing task for each input dataset which generates those >> records. After that I write it into disk (in software-specific >> transaction). I just wanted to separate hard computing and io operations. I >> created a side-effect method which is injected together with the dataset >> into a future. The futures are async collected within a list wrapped in >> agent. After the computing the main thread is waiting until all io tasks >> will be finished. >> >> >>> >>> At the end of the main thread function I just use await-for and after that: (reduce + (map #(deref %) @data-loading-tasks)) >>> >> As a control, tasks return number of written records. >> >> >> >>> For some reason I see the happy collecting (see attached screenshot of jconsole). >>> >>> "happy" = "heap"? >>> >> >> Both. As you can see on attached screenshot the heap usage grows easy >> until aver. ~2 1/4 G than keep that for a few minutes. In that moment I >> stopped. After that starts grow till ~4G with tendency to do jumps a bit >> more that 4G. >> >> >>> >>> After seeing the source code of future I suppose that the memory (data are kept as #{} set) is not released. The task returns only integer so I do not think that might cause the problem. >>> >>> Can you provide more detail? You keep alluding to things that you don't >>> provide code for, such as the sets of data. >>> >> >> >> The code is attached. However the important code is >> >> L123 . >> (let [;; keeps all data loading futures. >> ;; waiting until all futures are finished >> ;; should be done outside the main loop >> data-loading-tasks (agent [])] >> >> L128 >> (doseq >> (let [r1 (long operation)] L133 >> (doseq >> (let [r2 (v.v. long)] L155 >> >> L163 (send-off data-loading-task conj-task) >> >> ) >> ) >> ) >> ) >> >> >> I guess first I will move data-loading-tasks list into one of inner lets. >> Also I will create within an injecting function a separate abstract >> function let inside. The task will populate tmp variable which will be >> returned as a future result: >> >> >> L114 (conj agent (future (apply (fn [] (let [result (apply task args)] >> result) >> >> >>> >>> -- >>> James Reeves >>> booleanknot.com >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clojure@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+unsubscr...@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message becaus
Re: what does future do after fn finish ?
Agents manage a pool of threads for you. Try doing it without the future call and see if that works (unless you're trying to do something else). John On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta wrote: > Thanks a lot. I will check it tomorrow. > > J > > On 1 Feb 2018 12:12 a.m., "Justin Smith" wrote: > >> this is exactly the kind of problem code I was describing - there's no >> backpressure on existing future tasks to hold up the launching of more >> futures - the work done by the agent calling conj is negligible. You need >> to control the size of the pool of threads used, and you need to impose >> back-pressure. >> >> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta >> wrote: >> >>> On 31 January 2018 at 18:08, James Reeves wrote: >>> On 31 January 2018 at 17:59, Jacek Grzebyta wrote: > I have application with quite intense tripe store populating ~30/40 k > records per chunk (139 portions). The data are wrapped within the future: > > (conj agent (future (apply task args))) > > and that all together is send-off into (agent []). > What is "agent"? The first line of code indicates that it's a local collection shadowing the code function, while the second code snippet indicates that you're using the core agent function. Also why are you sending off to an agent? >>> >>> I have ~8sec computing task for each input dataset which generates those >>> records. After that I write it into disk (in software-specific >>> transaction). I just wanted to separate hard computing and io operations. I >>> created a side-effect method which is injected together with the dataset >>> into a future. The futures are async collected within a list wrapped in >>> agent. After the computing the main thread is waiting until all io tasks >>> will be finished. >>> >>> At the end of the main thread function I just use await-for and after > that: > > (reduce + (map #(deref %) @data-loading-tasks)) > >>> As a control, tasks return number of written records. >>> >>> >>> > For some reason I see the happy collecting (see attached screenshot of > jconsole). > "happy" = "heap"? >>> >>> Both. As you can see on attached screenshot the heap usage grows easy >>> until aver. ~2 1/4 G than keep that for a few minutes. In that moment I >>> stopped. After that starts grow till ~4G with tendency to do jumps a bit >>> more that 4G. >>> >>> > After seeing the source code of future I suppose that the memory (data > are kept as #{} set) is not released. The task returns only integer so I > do > not think that might cause the problem. > Can you provide more detail? You keep alluding to things that you don't provide code for, such as the sets of data. >>> >>> >>> The code is attached. However the important code is >>> >>> L123 . >>> (let [;; keeps all data loading futures. >>> ;; waiting until all futures are finished >>> ;; should be done outside the main loop >>> data-loading-tasks (agent [])] >>> >>> L128 >>> (doseq >>> (let [r1 (long operation)] L133 >>> (doseq >>> (let [r2 (v.v. long)] L155 >>> >>> L163 (send-off data-loading-task conj-task) >>> >>> ) >>> ) >>> ) >>> ) >>> >>> >>> I guess first I will move data-loading-tasks list into one of inner >>> lets. Also I will create within an injecting function a separate abstract >>> function let inside. The task will populate tmp variable which will be >>> returned as a future result: >>> >>> >>> L114 (conj agent (future (apply (fn [] (let [result (apply task args)] >>> result) >>> >>> -- James Reeves booleanknot.com -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clojure@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.goo
Re: what does future do after fn finish ?
Doing all the actions via one agent means that the actions are serialized though - you end up with no performance improvement over doing them all in a doseq in one future - the right way to do this tends to be trickier than it looks at first glance, and depends on your requirements. agents, the claypoole library, and reducers are all potentially useful. If parallelization leads to complex coordination needs, core.async can help too. On Wed, Jan 31, 2018 at 5:18 PM John Newman wrote: > Agents manage a pool of threads for you. Try doing it without the future > call and see if that works (unless you're trying to do something else). > > John > > On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta > wrote: > >> Thanks a lot. I will check it tomorrow. >> >> J >> >> On 1 Feb 2018 12:12 a.m., "Justin Smith" wrote: >> >>> this is exactly the kind of problem code I was describing - there's no >>> backpressure on existing future tasks to hold up the launching of more >>> futures - the work done by the agent calling conj is negligible. You need >>> to control the size of the pool of threads used, and you need to impose >>> back-pressure. >>> >>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta >>> wrote: >>> On 31 January 2018 at 18:08, James Reeves wrote: > On 31 January 2018 at 17:59, Jacek Grzebyta > wrote: > >> I have application with quite intense tripe store populating ~30/40 k >> records per chunk (139 portions). The data are wrapped within the future: >> >> (conj agent (future (apply task args))) >> >> and that all together is send-off into (agent []). >> > > What is "agent"? The first line of code indicates that it's a local > collection shadowing the code function, while the second code snippet > indicates that you're using the core agent function. > > Also why are you sending off to an agent? > I have ~8sec computing task for each input dataset which generates those records. After that I write it into disk (in software-specific transaction). I just wanted to separate hard computing and io operations. I created a side-effect method which is injected together with the dataset into a future. The futures are async collected within a list wrapped in agent. After the computing the main thread is waiting until all io tasks will be finished. > > At the end of the main thread function I just use await-for and after >> that: >> >> (reduce + (map #(deref %) @data-loading-tasks)) >> > As a control, tasks return number of written records. > >> For some reason I see the happy collecting (see attached screenshot >> of jconsole). >> > > "happy" = "heap"? > Both. As you can see on attached screenshot the heap usage grows easy until aver. ~2 1/4 G than keep that for a few minutes. In that moment I stopped. After that starts grow till ~4G with tendency to do jumps a bit more that 4G. > > >> After seeing the source code of future I suppose that the memory >> (data are kept as #{} set) is not released. The task returns only integer >> so I do not think that might cause the problem. >> > > Can you provide more detail? You keep alluding to things that you > don't provide code for, such as the sets of data. > The code is attached. However the important code is L123 . (let [;; keeps all data loading futures. ;; waiting until all futures are finished ;; should be done outside the main loop data-loading-tasks (agent [])] L128 (doseq (let [r1 (long operation)] L133 (doseq (let [r2 (v.v. long)] L155 L163 (send-off data-loading-task conj-task) ) ) ) ) I guess first I will move data-loading-tasks list into one of inner lets. Also I will create within an injecting function a separate abstract function let inside. The task will populate tmp variable which will be returned as a future result: L114 (conj agent (future (apply (fn [] (let [result (apply task args)] result) > > -- > James Reeves > booleanknot.com > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient > with your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google > Groups "Clojure
Re: what does future do after fn finish ?
Multiple sen-doffs to one agent will serialize it's calls, but spawning agents on each new task will spawn threads on a bounded thread pool, I believe. On Jan 31, 2018 8:32 PM, "Justin Smith" wrote: > Doing all the actions via one agent means that the actions are serialized > though - you end up with no performance improvement over doing them all in > a doseq in one future - the right way to do this tends to be trickier than > it looks at first glance, and depends on your requirements. agents, the > claypoole library, and reducers are all potentially useful. If > parallelization leads to complex coordination needs, core.async can help > too. > > On Wed, Jan 31, 2018 at 5:18 PM John Newman wrote: > >> Agents manage a pool of threads for you. Try doing it without the future >> call and see if that works (unless you're trying to do something else). >> >> John >> >> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta >> wrote: >> >>> Thanks a lot. I will check it tomorrow. >>> >>> J >>> >>> On 1 Feb 2018 12:12 a.m., "Justin Smith" wrote: >>> this is exactly the kind of problem code I was describing - there's no backpressure on existing future tasks to hold up the launching of more futures - the work done by the agent calling conj is negligible. You need to control the size of the pool of threads used, and you need to impose back-pressure. On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta wrote: > On 31 January 2018 at 18:08, James Reeves > wrote: > >> On 31 January 2018 at 17:59, Jacek Grzebyta >> wrote: >> >>> I have application with quite intense tripe store populating ~30/40 >>> k records per chunk (139 portions). The data are wrapped within the >>> future: >>> >>> (conj agent (future (apply task args))) >>> >>> and that all together is send-off into (agent []). >>> >> >> What is "agent"? The first line of code indicates that it's a local >> collection shadowing the code function, while the second code snippet >> indicates that you're using the core agent function. >> >> Also why are you sending off to an agent? >> > > I have ~8sec computing task for each input dataset which generates > those records. After that I write it into disk (in software-specific > transaction). I just wanted to separate hard computing and io operations. > I > created a side-effect method which is injected together with the dataset > into a future. The futures are async collected within a list wrapped in > agent. After the computing the main thread is waiting until all io tasks > will be finished. > > >> >> At the end of the main thread function I just use await-for and after >>> that: >>> >>> (reduce + (map #(deref %) @data-loading-tasks)) >>> >> > As a control, tasks return number of written records. > > > >> >>> For some reason I see the happy collecting (see attached screenshot >>> of jconsole). >>> >> >> "happy" = "heap"? >> > > Both. As you can see on attached screenshot the heap usage grows easy > until aver. ~2 1/4 G than keep that for a few minutes. In that moment I > stopped. After that starts grow till ~4G with tendency to do jumps a bit > more that 4G. > > >> >> >>> After seeing the source code of future I suppose that the memory >>> (data are kept as #{} set) is not released. The task returns only >>> integer >>> so I do not think that might cause the problem. >>> >> >> Can you provide more detail? You keep alluding to things that you >> don't provide code for, such as the sets of data. >> > > > The code is attached. However the important code is > > L123 . > (let [;; keeps all data loading futures. > ;; waiting until all futures are finished > ;; should be done outside the main loop > data-loading-tasks (agent [])] > > L128 > (doseq > (let [r1 (long operation)] L133 > (doseq > (let [r2 (v.v. long)] L155 > > L163 (send-off data-loading-task conj-task) > > ) > ) > ) > ) > > > I guess first I will move data-loading-tasks list into one of inner > lets. Also I will create within an injecting function a separate abstract > function let inside. The task will populate tmp variable which will be > returned as a future result: > > > L114 (conj agent (future (apply (fn [] (let [result (apply task > args)] result) > > >> >> -- >> James Reeves >> booleanknot.com >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that p
Re: what does future do after fn finish ?
Ah, he's using one agent, I see. On Jan 31, 2018 9:15 PM, "John Newman" wrote: > Multiple sen-doffs to one agent will serialize it's calls, but spawning > agents on each new task will spawn threads on a bounded thread pool, I > believe. > > On Jan 31, 2018 8:32 PM, "Justin Smith" wrote: > >> Doing all the actions via one agent means that the actions are serialized >> though - you end up with no performance improvement over doing them all in >> a doseq in one future - the right way to do this tends to be trickier than >> it looks at first glance, and depends on your requirements. agents, the >> claypoole library, and reducers are all potentially useful. If >> parallelization leads to complex coordination needs, core.async can help >> too. >> >> On Wed, Jan 31, 2018 at 5:18 PM John Newman wrote: >> >>> Agents manage a pool of threads for you. Try doing it without the future >>> call and see if that works (unless you're trying to do something else). >>> >>> John >>> >>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta >>> wrote: >>> Thanks a lot. I will check it tomorrow. J On 1 Feb 2018 12:12 a.m., "Justin Smith" wrote: > this is exactly the kind of problem code I was describing - there's no > backpressure on existing future tasks to hold up the launching of more > futures - the work done by the agent calling conj is negligible. You need > to control the size of the pool of threads used, and you need to impose > back-pressure. > > On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta > wrote: > >> On 31 January 2018 at 18:08, James Reeves >> wrote: >> >>> On 31 January 2018 at 17:59, Jacek Grzebyta >>> wrote: >>> I have application with quite intense tripe store populating ~30/40 k records per chunk (139 portions). The data are wrapped within the future: (conj agent (future (apply task args))) and that all together is send-off into (agent []). >>> >>> What is "agent"? The first line of code indicates that it's a local >>> collection shadowing the code function, while the second code snippet >>> indicates that you're using the core agent function. >>> >>> Also why are you sending off to an agent? >>> >> >> I have ~8sec computing task for each input dataset which generates >> those records. After that I write it into disk (in software-specific >> transaction). I just wanted to separate hard computing and io >> operations. I >> created a side-effect method which is injected together with the dataset >> into a future. The futures are async collected within a list wrapped in >> agent. After the computing the main thread is waiting until all io tasks >> will be finished. >> >> >>> >>> At the end of the main thread function I just use await-for and after that: (reduce + (map #(deref %) @data-loading-tasks)) >>> >> As a control, tasks return number of written records. >> >> >> >>> For some reason I see the happy collecting (see attached screenshot of jconsole). >>> >>> "happy" = "heap"? >>> >> >> Both. As you can see on attached screenshot the heap usage grows easy >> until aver. ~2 1/4 G than keep that for a few minutes. In that moment I >> stopped. After that starts grow till ~4G with tendency to do jumps a bit >> more that 4G. >> >> >>> >>> After seeing the source code of future I suppose that the memory (data are kept as #{} set) is not released. The task returns only integer so I do not think that might cause the problem. >>> >>> Can you provide more detail? You keep alluding to things that you >>> don't provide code for, such as the sets of data. >>> >> >> >> The code is attached. However the important code is >> >> L123 . >> (let [;; keeps all data loading futures. >> ;; waiting until all futures are finished >> ;; should be done outside the main loop >> data-loading-tasks (agent [])] >> >> L128 >> (doseq >> (let [r1 (long operation)] L133 >> (doseq >> (let [r2 (v.v. long)] L155 >> >> L163 (send-off data-loading-task conj-task) >> >> ) >> ) >> ) >> ) >> >> >> I guess first I will move data-loading-tasks list into one of inner >> lets. Also I will create within an injecting function a separate abstract >> function let inside. The task will populate tmp variable which will be >> returned as a future result: >> >> >> L114 (conj agent (future (apply (fn [] (let [result (apply task >> args)] result) >> >> >>> >>> -- >>> James Reeves >>
Re: what does future do after fn finish ?
Thanks folks. I see now! It should be a list of agents not list of futures within agent. Also any task sent to a agent is processed within a thread anyway so I do not need to add future... On 1 February 2018 at 02:17, John Newman wrote: > Ah, he's using one agent, I see. > > On Jan 31, 2018 9:15 PM, "John Newman" wrote: > >> Multiple sen-doffs to one agent will serialize it's calls, but spawning >> agents on each new task will spawn threads on a bounded thread pool, I >> believe. >> >> On Jan 31, 2018 8:32 PM, "Justin Smith" wrote: >> >>> Doing all the actions via one agent means that the actions are >>> serialized though - you end up with no performance improvement over doing >>> them all in a doseq in one future - the right way to do this tends to be >>> trickier than it looks at first glance, and depends on your requirements. >>> agents, the claypoole library, and reducers are all potentially useful. If >>> parallelization leads to complex coordination needs, core.async can help >>> too. >>> >>> On Wed, Jan 31, 2018 at 5:18 PM John Newman wrote: >>> Agents manage a pool of threads for you. Try doing it without the future call and see if that works (unless you're trying to do something else). John On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta >>> > wrote: > Thanks a lot. I will check it tomorrow. > > J > > On 1 Feb 2018 12:12 a.m., "Justin Smith" wrote: > >> this is exactly the kind of problem code I was describing - there's >> no backpressure on existing future tasks to hold up the launching of more >> futures - the work done by the agent calling conj is negligible. You need >> to control the size of the pool of threads used, and you need to impose >> back-pressure. >> >> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta < >> grzebyta@gmail.com> wrote: >> >>> On 31 January 2018 at 18:08, James Reeves >>> wrote: >>> On 31 January 2018 at 17:59, Jacek Grzebyta >>> > wrote: > I have application with quite intense tripe store populating > ~30/40 k records per chunk (139 portions). The data are wrapped > within the > future: > > (conj agent (future (apply task args))) > > and that all together is send-off into (agent []). > What is "agent"? The first line of code indicates that it's a local collection shadowing the code function, while the second code snippet indicates that you're using the core agent function. Also why are you sending off to an agent? >>> >>> I have ~8sec computing task for each input dataset which generates >>> those records. After that I write it into disk (in software-specific >>> transaction). I just wanted to separate hard computing and io >>> operations. I >>> created a side-effect method which is injected together with the dataset >>> into a future. The futures are async collected within a list wrapped in >>> agent. After the computing the main thread is waiting until all io tasks >>> will be finished. >>> >>> At the end of the main thread function I just use await-for and > after that: > > (reduce + (map #(deref %) @data-loading-tasks)) > >>> As a control, tasks return number of written records. >>> >>> >>> > For some reason I see the happy collecting (see attached > screenshot of jconsole). > "happy" = "heap"? >>> >>> Both. As you can see on attached screenshot the heap usage grows >>> easy until aver. ~2 1/4 G than keep that for a few minutes. In that >>> moment >>> I stopped. After that starts grow till ~4G with tendency to do jumps a >>> bit >>> more that 4G. >>> >>> > After seeing the source code of future I suppose that the memory > (data are kept as #{} set) is not released. The task returns only > integer > so I do not think that might cause the problem. > Can you provide more detail? You keep alluding to things that you don't provide code for, such as the sets of data. >>> >>> >>> The code is attached. However the important code is >>> >>> L123 . >>> (let [;; keeps all data loading futures. >>> ;; waiting until all futures are finished >>> ;; should be done outside the main loop >>> data-loading-tasks (agent [])] >>> >>> L128 >>> (doseq >>> (let [r1 (long operation)] L133 >>> (doseq >>> (let [r2 (v.v. long)] L155 >>> >>> L163 (send-off data-loading-task conj-task) >>> >>> ) >>> ) >>> ) >>> ) >>> >>
Re: what does future do after fn finish ?
yes, that's the idea exactly also, you might want more fine grained control of how much parallelism occurs (eg. if every thread is writing to the same physical device, you can often get better throughput by not parallelizing at all, or keeping the parallelism quite limited - it's worth experimenting) - there are good ways to control that using ThreadPoolExecutor directly, or using clojure.lang.PersistentQueue/EMPTY as a control construct, or core.async channels, or ztellman's manifold library, or the claypoole threading library On Thu, Feb 1, 2018, 03:44 Jacek Grzebyta wrote: > Thanks folks. I see now! It should be a list of agents not list of futures > within agent. Also any task sent to a agent is processed within a > thread anyway so I do not need to add future... > > On 1 February 2018 at 02:17, John Newman wrote: > >> Ah, he's using one agent, I see. >> >> On Jan 31, 2018 9:15 PM, "John Newman" wrote: >> >>> Multiple sen-doffs to one agent will serialize it's calls, but spawning >>> agents on each new task will spawn threads on a bounded thread pool, I >>> believe. >>> >>> On Jan 31, 2018 8:32 PM, "Justin Smith" wrote: >>> Doing all the actions via one agent means that the actions are serialized though - you end up with no performance improvement over doing them all in a doseq in one future - the right way to do this tends to be trickier than it looks at first glance, and depends on your requirements. agents, the claypoole library, and reducers are all potentially useful. If parallelization leads to complex coordination needs, core.async can help too. On Wed, Jan 31, 2018 at 5:18 PM John Newman wrote: > Agents manage a pool of threads for you. Try doing it without the > future call and see if that works (unless you're trying to do something > else). > > John > > On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta < > grzebyta@gmail.com> wrote: > >> Thanks a lot. I will check it tomorrow. >> >> J >> >> On 1 Feb 2018 12:12 a.m., "Justin Smith" >> wrote: >> >>> this is exactly the kind of problem code I was describing - there's >>> no backpressure on existing future tasks to hold up the launching of >>> more >>> futures - the work done by the agent calling conj is negligible. You >>> need >>> to control the size of the pool of threads used, and you need to impose >>> back-pressure. >>> >>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta < >>> grzebyta@gmail.com> wrote: >>> On 31 January 2018 at 18:08, James Reeves wrote: > On 31 January 2018 at 17:59, Jacek Grzebyta < > grzebyta@gmail.com> wrote: > >> I have application with quite intense tripe store populating >> ~30/40 k records per chunk (139 portions). The data are wrapped >> within the >> future: >> >> (conj agent (future (apply task args))) >> >> and that all together is send-off into (agent []). >> > > What is "agent"? The first line of code indicates that it's a > local collection shadowing the code function, while the second code > snippet > indicates that you're using the core agent function. > > Also why are you sending off to an agent? > I have ~8sec computing task for each input dataset which generates those records. After that I write it into disk (in software-specific transaction). I just wanted to separate hard computing and io operations. I created a side-effect method which is injected together with the dataset into a future. The futures are async collected within a list wrapped in agent. After the computing the main thread is waiting until all io tasks will be finished. > > At the end of the main thread function I just use await-for and >> after that: >> >> (reduce + (map #(deref %) @data-loading-tasks)) >> > As a control, tasks return number of written records. > >> For some reason I see the happy collecting (see attached >> screenshot of jconsole). >> > > "happy" = "heap"? > Both. As you can see on attached screenshot the heap usage grows easy until aver. ~2 1/4 G than keep that for a few minutes. In that moment I stopped. After that starts grow till ~4G with tendency to do jumps a bit more that 4G. > > >> After seeing the source code of future I suppose that the memory >> (data are kept as #{} set) is not released. The task returns only >> integer >> so I do not think that m
Re: what does future do after fn finish ?
You may find that using the Claypoole library is the easiest way to handle threadpools: https://github.com/TheClimateCorporation/claypoole Alan On Thu, Feb 1, 2018 at 11:16 AM, Justin Smith wrote: > yes, that's the idea exactly > > also, you might want more fine grained control of how much parallelism > occurs (eg. if every thread is writing to the same physical device, you can > often get better throughput by not parallelizing at all, or keeping the > parallelism quite limited - it's worth experimenting) - there are good ways > to control that using ThreadPoolExecutor directly, or using > clojure.lang.PersistentQueue/EMPTY as a control construct, or core.async > channels, or ztellman's manifold library, or the claypoole threading library > > On Thu, Feb 1, 2018, 03:44 Jacek Grzebyta wrote: > >> Thanks folks. I see now! It should be a list of agents not list of >> futures within agent. Also any task sent to a agent is processed >> within a thread anyway so I do not need to add future... >> >> On 1 February 2018 at 02:17, John Newman wrote: >> >>> Ah, he's using one agent, I see. >>> >>> On Jan 31, 2018 9:15 PM, "John Newman" wrote: >>> Multiple sen-doffs to one agent will serialize it's calls, but spawning agents on each new task will spawn threads on a bounded thread pool, I believe. On Jan 31, 2018 8:32 PM, "Justin Smith" wrote: > Doing all the actions via one agent means that the actions are > serialized though - you end up with no performance improvement over doing > them all in a doseq in one future - the right way to do this tends to be > trickier than it looks at first glance, and depends on your requirements. > agents, the claypoole library, and reducers are all potentially useful. If > parallelization leads to complex coordination needs, core.async can help > too. > > On Wed, Jan 31, 2018 at 5:18 PM John Newman wrote: > >> Agents manage a pool of threads for you. Try doing it without the >> future call and see if that works (unless you're trying to do something >> else). >> >> John >> >> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta < >> grzebyta@gmail.com> wrote: >> >>> Thanks a lot. I will check it tomorrow. >>> >>> J >>> >>> On 1 Feb 2018 12:12 a.m., "Justin Smith" >>> wrote: >>> this is exactly the kind of problem code I was describing - there's no backpressure on existing future tasks to hold up the launching of more futures - the work done by the agent calling conj is negligible. You need to control the size of the pool of threads used, and you need to impose back-pressure. On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta < grzebyta@gmail.com> wrote: > On 31 January 2018 at 18:08, James Reeves > wrote: > >> On 31 January 2018 at 17:59, Jacek Grzebyta < >> grzebyta@gmail.com> wrote: >> >>> I have application with quite intense tripe store populating >>> ~30/40 k records per chunk (139 portions). The data are wrapped >>> within the >>> future: >>> >>> (conj agent (future (apply task args))) >>> >>> and that all together is send-off into (agent []). >>> >> >> What is "agent"? The first line of code indicates that it's a >> local collection shadowing the code function, while the second code >> snippet >> indicates that you're using the core agent function. >> >> Also why are you sending off to an agent? >> > > I have ~8sec computing task for each input dataset which generates > those records. After that I write it into disk (in software-specific > transaction). I just wanted to separate hard computing and io > operations. I > created a side-effect method which is injected together with the > dataset > into a future. The futures are async collected within a list wrapped > in > agent. After the computing the main thread is waiting until all io > tasks > will be finished. > > >> >> At the end of the main thread function I just use await-for and >>> after that: >>> >>> (reduce + (map #(deref %) @data-loading-tasks)) >>> >> > As a control, tasks return number of written records. > > > >> >>> For some reason I see the happy collecting (see attached >>> screenshot of jconsole). >>> >> >> "happy" = "heap"? >> > > Both. As you can see on attached screenshot the heap usage grows > easy until aver. ~2 1/4 G than keep that for a few minutes. In that > moment > I stoppe
Re: what does future do after fn finish ?
+1 for Claypoole, it removed the needs of using agents or futures in 95% of the cases in my code. On Thursday, February 1, 2018 at 9:54:36 PM UTC+1, Alan Thompson wrote: > > You may find that using the Claypoole library is the easiest way to handle > threadpools: https://github.com/TheClimateCorporation/claypoole > Alan > > On Thu, Feb 1, 2018 at 11:16 AM, Justin Smith > wrote: > >> yes, that's the idea exactly >> >> also, you might want more fine grained control of how much parallelism >> occurs (eg. if every thread is writing to the same physical device, you can >> often get better throughput by not parallelizing at all, or keeping the >> parallelism quite limited - it's worth experimenting) - there are good ways >> to control that using ThreadPoolExecutor directly, or using >> clojure.lang.PersistentQueue/EMPTY as a control construct, or core.async >> channels, or ztellman's manifold library, or the claypoole threading library >> >> On Thu, Feb 1, 2018, 03:44 Jacek Grzebyta > > wrote: >> >>> Thanks folks. I see now! It should be a list of agents not list of >>> futures within agent. Also any task sent to a agent is processed >>> within a thread anyway so I do not need to add future... >>> >>> On 1 February 2018 at 02:17, John Newman >>> > wrote: >>> Ah, he's using one agent, I see. On Jan 31, 2018 9:15 PM, "John Newman" > wrote: > Multiple sen-doffs to one agent will serialize it's calls, but > spawning agents on each new task will spawn threads on a bounded thread > pool, I believe. > > On Jan 31, 2018 8:32 PM, "Justin Smith" > wrote: > >> Doing all the actions via one agent means that the actions are >> serialized though - you end up with no performance improvement over >> doing >> them all in a doseq in one future - the right way to do this tends to be >> trickier than it looks at first glance, and depends on your >> requirements. >> agents, the claypoole library, and reducers are all potentially useful. >> If >> parallelization leads to complex coordination needs, core.async can help >> too. >> >> On Wed, Jan 31, 2018 at 5:18 PM John Newman > > wrote: >> >>> Agents manage a pool of threads for you. Try doing it without the >>> future call and see if that works (unless you're trying to do something >>> else). >>> >>> John >>> >>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta >> > wrote: >>> Thanks a lot. I will check it tomorrow. J On 1 Feb 2018 12:12 a.m., "Justin Smith" >>> > wrote: > this is exactly the kind of problem code I was describing - > there's no backpressure on existing future tasks to hold up the > launching > of more futures - the work done by the agent calling conj is > negligible. > You need to control the size of the pool of threads used, and you > need to > impose back-pressure. > > On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta < > grzeby...@gmail.com > wrote: > >> On 31 January 2018 at 18:08, James Reeves > > wrote: >> >>> On 31 January 2018 at 17:59, Jacek Grzebyta >> > wrote: >>> I have application with quite intense tripe store populating ~30/40 k records per chunk (139 portions). The data are wrapped within the future: (conj agent (future (apply task args))) and that all together is send-off into (agent []). >>> >>> What is "agent"? The first line of code indicates that it's a >>> local collection shadowing the code function, while the second code >>> snippet >>> indicates that you're using the core agent function. >>> >>> Also why are you sending off to an agent? >>> >> >> I have ~8sec computing task for each input dataset which >> generates those records. After that I write it into disk (in >> software-specific transaction). I just wanted to separate hard >> computing >> and io operations. I created a side-effect method which is injected >> together with the dataset into a future. The futures are async >> collected >> within a list wrapped in agent. After the computing the main thread >> is >> waiting until all io tasks will be finished. >> >> >>> >>> At the end of the main thread function I just use await-for and after that: (reduce + (map #(deref %) @data-loading-tasks)) >>> >> As a control, tasks return number of written records. >> >> >> >>> For some reas
Re: what does future do after fn finish ?
On 2 February 2018 at 08:34, Niels van Klaveren wrote: > +1 for Claypoole, it removed the needs of using agents or futures in 95% > of the cases in my code. > > Thanks a lot. I modify the code using claypoole. I imagine with-shutdown will close the pool properly after finish all tasks so there is no need to watch them? J -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: what does future do after fn finish ?
OK I found what makes the memory leak. In the project I work with I use a java Model class which is java Collection proxy/facade for a single transaction. Unfortunately it's not thread safe. In a few places I passed single instance of model into several threads Also I requested the instance with -> which makes new thread as well. I was surprised that -> makes trouble but after thinking that might be expected. Especially that internally the wrapper doesn't do simple mapping - it uses some iterator, etc. Anyway the machinery is fragile. It seems I need to rewrite code and replace all multithreading parts by something simpler. If you want see the stacktrace just look at: https://github.com/jgrzebyta/triple-loader/issues/53 On 2 February 2018 at 11:16, Jacek Grzebyta wrote: > > On 2 February 2018 at 08:34, Niels van Klaveren < > niels.vanklave...@gmail.com> wrote: > >> +1 for Claypoole, it removed the needs of using agents or futures in 95% >> of the cases in my code. >> >> > Thanks a lot. I modify the code using claypoole. I imagine with-shutdown > will close the pool properly after finish all tasks so there is no need to > watch them? > > J > > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: what does future do after fn finish ?
-> is just a list transform performed after reading your code into a list data structure containing symbols, and before compiling to byte code - it doesn't do anything directly. On Fri, Feb 2, 2018 at 3:55 PM Jacek Grzebyta wrote: > OK I found what makes the memory leak. > > In the project I work with I use a java Model class which is java > Collection proxy/facade for a single transaction. Unfortunately it's not > thread safe. In a few places I passed single instance of model into several > threads Also I requested the instance with -> which makes new thread as > well. I was surprised that -> makes trouble but after thinking that might > be expected. Especially that internally the wrapper doesn't do simple > mapping - it uses some iterator, etc. Anyway the machinery is fragile. It > seems I need to rewrite code and replace all multithreading parts by > something simpler. > > > > If you want see the stacktrace just look at: > https://github.com/jgrzebyta/triple-loader/issues/53 > > On 2 February 2018 at 11:16, Jacek Grzebyta > wrote: > >> >> On 2 February 2018 at 08:34, Niels van Klaveren < >> niels.vanklave...@gmail.com> wrote: >> >>> +1 for Claypoole, it removed the needs of using agents or futures in 95% >>> of the cases in my code. >>> >>> >> Thanks a lot. I modify the code using claypoole. I imagine with-shutdown >> will close the pool properly after finish all tasks so there is no need to >> watch them? >> >> J >> >> > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: what does future do after fn finish ?
HI, So things works well. In terms of memory usage anyway. I use 50 main threads (in process-data-by-condition-set) and 20 for replications (process-growths). However I found that after good performance in the beginning it slows down rapid and quick. VisualVM shows the threads usage is poor - mainly they are waiting. Only 1 claypoole thread works + some lower level threads managed probably by Clojure, etc. CPU usage is poor as well. I guess the reason is on the lower - Java classes level with IO operations bottleneck. IMO they are working by blocking. That why I wanted to separate them. However IO speed operations go down as well. I've tried async channels but that they were flooded and some data where lost for some reason. Regards, Jacek On 3 February 2018 at 00:14, Justin Smith wrote: > -> is just a list transform performed after reading your code into a list > data structure containing symbols, and before compiling to byte code - it > doesn't do anything directly. > > On Fri, Feb 2, 2018 at 3:55 PM Jacek Grzebyta > wrote: > >> OK I found what makes the memory leak. >> >> In the project I work with I use a java Model class which is java >> Collection proxy/facade for a single transaction. Unfortunately it's not >> thread safe. In a few places I passed single instance of model into several >> threads Also I requested the instance with -> which makes new thread as >> well. I was surprised that -> makes trouble but after thinking that might >> be expected. Especially that internally the wrapper doesn't do simple >> mapping - it uses some iterator, etc. Anyway the machinery is fragile. It >> seems I need to rewrite code and replace all multithreading parts by >> something simpler. >> >> >> >> If you want see the stacktrace just look at: >> https://github.com/jgrzebyta/triple-loader/issues/53 >> >> On 2 February 2018 at 11:16, Jacek Grzebyta >> wrote: >> >>> >>> On 2 February 2018 at 08:34, Niels van Klaveren < >>> niels.vanklave...@gmail.com> wrote: >>> +1 for Claypoole, it removed the needs of using agents or futures in 95% of the cases in my code. >>> Thanks a lot. I modify the code using claypoole. I imagine with-shutdown >>> will close the pool properly after finish all tasks so there is no need to >>> watch them? >>> >>> J >>> >>> >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to the Google Groups >> "Clojure" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to clojure+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.