Re: If Clojure is to blame for the majority of the startup time, why doesn't ClojureScript proportionally slow down the JavaScript startup time also?

[scott stackelhouse]

I was just reading about Docker+CRIU.  

For me, the slow startup isn't much of a bother for normal operations of my 
apps.  They are generally long running services.  But development is where I am 
bothered by it the most.  It can be painful to restart a repl and it's very 
disruptive to my ability to stay focussed.

CRIU seems like it could actually help in this case.  In particular, paired 
with Docker, I can just restart from a known good checkpoint with editor 
loaded, repl up and connected, etc.  That would be very nice.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what does future do after fn finish ?

[John Newman]

Ah, he's using one agent, I see.

On Jan 31, 2018 9:15 PM, "John Newman"  wrote:

> Multiple sen-doffs to one agent will serialize it's calls, but spawning
> agents on each new task will spawn threads on a bounded thread pool, I
> believe.
>
> On Jan 31, 2018 8:32 PM, "Justin Smith"  wrote:
>
>> Doing all the actions via one agent means that the actions are serialized
>> though - you end up with no performance improvement over doing them all in
>> a doseq in one future - the right way to do this tends to be trickier than
>> it looks at first glance, and depends on your requirements. agents, the
>> claypoole library, and reducers are all potentially useful. If
>> parallelization leads to complex coordination needs, core.async can help
>> too.
>>
>> On Wed, Jan 31, 2018 at 5:18 PM John Newman  wrote:
>>
>>> Agents manage a pool of threads for you. Try doing it without the future
>>> call and see if that works (unless you're trying to do something else).
>>>
>>> John
>>>
>>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta 
>>> wrote:
>>>
 Thanks a lot. I will check it tomorrow.

 J

 On 1 Feb 2018 12:12 a.m., "Justin Smith"  wrote:

> this is exactly the kind of problem code I was describing - there's no
> backpressure on existing future tasks to hold up the launching of more
> futures - the work done by the agent calling conj is negligible. You need
> to control the size of the pool of threads used, and you need to impose
> back-pressure.
>
> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta 
> wrote:
>
>> On 31 January 2018 at 18:08, James Reeves 
>> wrote:
>>
>>> On 31 January 2018 at 17:59, Jacek Grzebyta 
>>> wrote:
>>>
 I have application with quite intense tripe store populating ~30/40
 k records per chunk (139 portions). The data are wrapped within the 
 future:

 (conj agent (future (apply task args)))

  and that all together is send-off into (agent []).

>>>
>>> What is "agent"? The first line of code indicates that it's a local
>>> collection shadowing the code function, while the second code snippet
>>> indicates that you're using the core agent function.
>>>
>>> Also why are you sending off to an agent?
>>>
>>
>> I have ~8sec computing task for each input dataset which generates
>> those records. After that I write it into disk (in software-specific
>> transaction). I just wanted to separate hard computing and io 
>> operations. I
>> created a side-effect method which is injected together with the dataset
>> into a future. The futures are async collected within a list wrapped in
>> agent. After the computing the main thread is waiting until all io tasks
>> will be finished.
>>
>>
>>>
>>> At the end of the main thread function I just use await-for and
 after that:

 (reduce + (map #(deref %) @data-loading-tasks))

>>>
>> As a control, tasks return number of written records.
>>
>>
>>
>>>
 For some reason I see the happy collecting (see attached screenshot
 of jconsole).

>>>
>>> "happy" = "heap"?
>>>
>>
>> Both. As you can see on attached screenshot the heap usage grows easy
>> until aver. ~2 1/4 G than keep that  for a few minutes. In that moment I
>> stopped. After that starts grow till ~4G with tendency to do jumps a bit
>> more that 4G.
>>
>>
>>>
>>>
 After seeing the source code of future I suppose that the memory
 (data are kept as #{} set) is not released. The task returns only 
 integer
 so I do not think that might cause the problem.

>>>
>>> Can you provide more detail? You keep alluding to things that you
>>> don't provide code for, such as the sets of data.
>>>
>>
>>
>> The code is attached. However the important code is
>>
>> L123 .
>>   (let [;; keeps all data loading futures.
>> ;; waiting until all futures are finished
>> ;; should be done outside the main loop
>> data-loading-tasks (agent [])]
>>
>> L128
>> (doseq
>>  (let [r1 (long operation)]   L133
>>  (doseq
>> (let [r2 (v.v. long)]   L155
>>
>>   L163   (send-off data-loading-task conj-task)
>>
>>  )
>>  )
>> )
>> )
>>
>>
>> I guess first I will move data-loading-tasks list into one of inner
>> lets. Also I will create within an injecting function a separate abstract
>> function let inside. The task will populate tmp variable which will be
>> returned as a 

Re: what does future do after fn finish ?

[John Newman]

Multiple sen-doffs to one agent will serialize it's calls, but spawning
agents on each new task will spawn threads on a bounded thread pool, I
believe.

On Jan 31, 2018 8:32 PM, "Justin Smith"  wrote:

> Doing all the actions via one agent means that the actions are serialized
> though - you end up with no performance improvement over doing them all in
> a doseq in one future - the right way to do this tends to be trickier than
> it looks at first glance, and depends on your requirements. agents, the
> claypoole library, and reducers are all potentially useful. If
> parallelization leads to complex coordination needs, core.async can help
> too.
>
> On Wed, Jan 31, 2018 at 5:18 PM John Newman  wrote:
>
>> Agents manage a pool of threads for you. Try doing it without the future
>> call and see if that works (unless you're trying to do something else).
>>
>> John
>>
>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta 
>> wrote:
>>
>>> Thanks a lot. I will check it tomorrow.
>>>
>>> J
>>>
>>> On 1 Feb 2018 12:12 a.m., "Justin Smith"  wrote:
>>>
 this is exactly the kind of problem code I was describing - there's no
 backpressure on existing future tasks to hold up the launching of more
 futures - the work done by the agent calling conj is negligible. You need
 to control the size of the pool of threads used, and you need to impose
 back-pressure.

 On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta 
 wrote:

> On 31 January 2018 at 18:08, James Reeves 
> wrote:
>
>> On 31 January 2018 at 17:59, Jacek Grzebyta 
>> wrote:
>>
>>> I have application with quite intense tripe store populating ~30/40
>>> k records per chunk (139 portions). The data are wrapped within the 
>>> future:
>>>
>>> (conj agent (future (apply task args)))
>>>
>>>  and that all together is send-off into (agent []).
>>>
>>
>> What is "agent"? The first line of code indicates that it's a local
>> collection shadowing the code function, while the second code snippet
>> indicates that you're using the core agent function.
>>
>> Also why are you sending off to an agent?
>>
>
> I have ~8sec computing task for each input dataset which generates
> those records. After that I write it into disk (in software-specific
> transaction). I just wanted to separate hard computing and io operations. 
> I
> created a side-effect method which is injected together with the dataset
> into a future. The futures are async collected within a list wrapped in
> agent. After the computing the main thread is waiting until all io tasks
> will be finished.
>
>
>>
>> At the end of the main thread function I just use await-for and after
>>> that:
>>>
>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>
>>
> As a control, tasks return number of written records.
>
>
>
>>
>>> For some reason I see the happy collecting (see attached screenshot
>>> of jconsole).
>>>
>>
>> "happy" = "heap"?
>>
>
> Both. As you can see on attached screenshot the heap usage grows easy
> until aver. ~2 1/4 G than keep that  for a few minutes. In that moment I
> stopped. After that starts grow till ~4G with tendency to do jumps a bit
> more that 4G.
>
>
>>
>>
>>> After seeing the source code of future I suppose that the memory
>>> (data are kept as #{} set) is not released. The task returns only 
>>> integer
>>> so I do not think that might cause the problem.
>>>
>>
>> Can you provide more detail? You keep alluding to things that you
>> don't provide code for, such as the sets of data.
>>
>
>
> The code is attached. However the important code is
>
> L123 .
>   (let [;; keeps all data loading futures.
> ;; waiting until all futures are finished
> ;; should be done outside the main loop
> data-loading-tasks (agent [])]
>
> L128
> (doseq
>  (let [r1 (long operation)]   L133
>  (doseq
> (let [r2 (v.v. long)]   L155
>
>   L163   (send-off data-loading-task conj-task)
>
>  )
>  )
> )
> )
>
>
> I guess first I will move data-loading-tasks list into one of inner
> lets. Also I will create within an injecting function a separate abstract
> function let inside. The task will populate tmp variable which will be
> returned as a future result:
>
>
> L114 (conj agent (future (apply (fn [] (let [result (apply task
> args)]  result)
>
>
>>
>> --
>> James Reeves
>> booleanknot.com
>>
>> --
>> You received this message 

Re: what does future do after fn finish ?

[Justin Smith]

Doing all the actions via one agent means that the actions are serialized
though - you end up with no performance improvement over doing them all in
a doseq in one future - the right way to do this tends to be trickier than
it looks at first glance, and depends on your requirements. agents, the
claypoole library, and reducers are all potentially useful. If
parallelization leads to complex coordination needs, core.async can help
too.

On Wed, Jan 31, 2018 at 5:18 PM John Newman  wrote:

> Agents manage a pool of threads for you. Try doing it without the future
> call and see if that works (unless you're trying to do something else).
>
> John
>
> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta 
> wrote:
>
>> Thanks a lot. I will check it tomorrow.
>>
>> J
>>
>> On 1 Feb 2018 12:12 a.m., "Justin Smith"  wrote:
>>
>>> this is exactly the kind of problem code I was describing - there's no
>>> backpressure on existing future tasks to hold up the launching of more
>>> futures - the work done by the agent calling conj is negligible. You need
>>> to control the size of the pool of threads used, and you need to impose
>>> back-pressure.
>>>
>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta 
>>> wrote:
>>>
 On 31 January 2018 at 18:08, James Reeves 
 wrote:

> On 31 January 2018 at 17:59, Jacek Grzebyta 
> wrote:
>
>> I have application with quite intense tripe store populating ~30/40 k
>> records per chunk (139 portions). The data are wrapped within the future:
>>
>> (conj agent (future (apply task args)))
>>
>>  and that all together is send-off into (agent []).
>>
>
> What is "agent"? The first line of code indicates that it's a local
> collection shadowing the code function, while the second code snippet
> indicates that you're using the core agent function.
>
> Also why are you sending off to an agent?
>

 I have ~8sec computing task for each input dataset which generates
 those records. After that I write it into disk (in software-specific
 transaction). I just wanted to separate hard computing and io operations. I
 created a side-effect method which is injected together with the dataset
 into a future. The futures are async collected within a list wrapped in
 agent. After the computing the main thread is waiting until all io tasks
 will be finished.


>
> At the end of the main thread function I just use await-for and after
>> that:
>>
>> (reduce + (map #(deref %) @data-loading-tasks))
>>
>
 As a control, tasks return number of written records.



>
>> For some reason I see the happy collecting (see attached screenshot
>> of jconsole).
>>
>
> "happy" = "heap"?
>

 Both. As you can see on attached screenshot the heap usage grows easy
 until aver. ~2 1/4 G than keep that  for a few minutes. In that moment I
 stopped. After that starts grow till ~4G with tendency to do jumps a bit
 more that 4G.


>
>
>> After seeing the source code of future I suppose that the memory
>> (data are kept as #{} set) is not released. The task returns only integer
>> so I do not think that might cause the problem.
>>
>
> Can you provide more detail? You keep alluding to things that you
> don't provide code for, such as the sets of data.
>


 The code is attached. However the important code is

 L123 .
   (let [;; keeps all data loading futures.
 ;; waiting until all futures are finished
 ;; should be done outside the main loop
 data-loading-tasks (agent [])]

 L128
 (doseq
  (let [r1 (long operation)]   L133
  (doseq
 (let [r2 (v.v. long)]   L155

   L163   (send-off data-loading-task conj-task)

  )
  )
 )
 )


 I guess first I will move data-loading-tasks list into one of inner
 lets. Also I will create within an injecting function a separate abstract
 function let inside. The task will populate tmp variable which will be
 returned as a future result:


 L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
 result)


>
> --
> James Reeves
> booleanknot.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient
> with your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> 

Re: what does future do after fn finish ?

[John Newman]

Agents manage a pool of threads for you. Try doing it without the future
call and see if that works (unless you're trying to do something else).

John

On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta 
wrote:

> Thanks a lot. I will check it tomorrow.
>
> J
>
> On 1 Feb 2018 12:12 a.m., "Justin Smith"  wrote:
>
>> this is exactly the kind of problem code I was describing - there's no
>> backpressure on existing future tasks to hold up the launching of more
>> futures - the work done by the agent calling conj is negligible. You need
>> to control the size of the pool of threads used, and you need to impose
>> back-pressure.
>>
>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta 
>> wrote:
>>
>>> On 31 January 2018 at 18:08, James Reeves  wrote:
>>>
 On 31 January 2018 at 17:59, Jacek Grzebyta 
 wrote:

> I have application with quite intense tripe store populating ~30/40 k
> records per chunk (139 portions). The data are wrapped within the future:
>
> (conj agent (future (apply task args)))
>
>  and that all together is send-off into (agent []).
>

 What is "agent"? The first line of code indicates that it's a local
 collection shadowing the code function, while the second code snippet
 indicates that you're using the core agent function.

 Also why are you sending off to an agent?

>>>
>>> I have ~8sec computing task for each input dataset which generates those
>>> records. After that I write it into disk (in software-specific
>>> transaction). I just wanted to separate hard computing and io operations. I
>>> created a side-effect method which is injected together with the dataset
>>> into a future. The futures are async collected within a list wrapped in
>>> agent. After the computing the main thread is waiting until all io tasks
>>> will be finished.
>>>
>>>

 At the end of the main thread function I just use await-for and after
> that:
>
> (reduce + (map #(deref %) @data-loading-tasks))
>

>>> As a control, tasks return number of written records.
>>>
>>>
>>>

> For some reason I see the happy collecting (see attached screenshot of
> jconsole).
>

 "happy" = "heap"?

>>>
>>> Both. As you can see on attached screenshot the heap usage grows easy
>>> until aver. ~2 1/4 G than keep that  for a few minutes. In that moment I
>>> stopped. After that starts grow till ~4G with tendency to do jumps a bit
>>> more that 4G.
>>>
>>>


> After seeing the source code of future I suppose that the memory (data
> are kept as #{} set) is not released. The task returns only integer so I 
> do
> not think that might cause the problem.
>

 Can you provide more detail? You keep alluding to things that you don't
 provide code for, such as the sets of data.

>>>
>>>
>>> The code is attached. However the important code is
>>>
>>> L123 .
>>>   (let [;; keeps all data loading futures.
>>> ;; waiting until all futures are finished
>>> ;; should be done outside the main loop
>>> data-loading-tasks (agent [])]
>>>
>>> L128
>>> (doseq
>>>  (let [r1 (long operation)]   L133
>>>  (doseq
>>> (let [r2 (v.v. long)]   L155
>>>
>>>   L163   (send-off data-loading-task conj-task)
>>>
>>>  )
>>>  )
>>> )
>>> )
>>>
>>>
>>> I guess first I will move data-loading-tasks list into one of inner
>>> lets. Also I will create within an injecting function a separate abstract
>>> function let inside. The task will populate tmp variable which will be
>>> returned as a future result:
>>>
>>>
>>> L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
>>> result)
>>>
>>>

 --
 James Reeves
 booleanknot.com

 --
 You received this message because you are subscribed to the Google
 Groups "Clojure" group.
 To post to this group, send email to clojure@googlegroups.com
 Note that posts from new members are moderated - please be patient with
 your first post.
 To unsubscribe from this group, send email to
 clojure+unsubscr...@googlegroups.com
 For more options, visit this group at
 http://groups.google.com/group/clojure?hl=en
 ---
 You received this message because you are subscribed to the Google
 Groups "Clojure" group.
 To unsubscribe from this group and stop receiving emails from it, send
 an email to clojure+unsubscr...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.

>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, 

Re: what does future do after fn finish ?

[Jacek Grzebyta]

Thanks a lot. I will check it tomorrow.

J

On 1 Feb 2018 12:12 a.m., "Justin Smith"  wrote:

> this is exactly the kind of problem code I was describing - there's no
> backpressure on existing future tasks to hold up the launching of more
> futures - the work done by the agent calling conj is negligible. You need
> to control the size of the pool of threads used, and you need to impose
> back-pressure.
>
> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta 
> wrote:
>
>> On 31 January 2018 at 18:08, James Reeves  wrote:
>>
>>> On 31 January 2018 at 17:59, Jacek Grzebyta 
>>> wrote:
>>>
 I have application with quite intense tripe store populating ~30/40 k
 records per chunk (139 portions). The data are wrapped within the future:

 (conj agent (future (apply task args)))

  and that all together is send-off into (agent []).

>>>
>>> What is "agent"? The first line of code indicates that it's a local
>>> collection shadowing the code function, while the second code snippet
>>> indicates that you're using the core agent function.
>>>
>>> Also why are you sending off to an agent?
>>>
>>
>> I have ~8sec computing task for each input dataset which generates those
>> records. After that I write it into disk (in software-specific
>> transaction). I just wanted to separate hard computing and io operations. I
>> created a side-effect method which is injected together with the dataset
>> into a future. The futures are async collected within a list wrapped in
>> agent. After the computing the main thread is waiting until all io tasks
>> will be finished.
>>
>>
>>>
>>> At the end of the main thread function I just use await-for and after
 that:

 (reduce + (map #(deref %) @data-loading-tasks))

>>>
>> As a control, tasks return number of written records.
>>
>>
>>
>>>
 For some reason I see the happy collecting (see attached screenshot of
 jconsole).

>>>
>>> "happy" = "heap"?
>>>
>>
>> Both. As you can see on attached screenshot the heap usage grows easy
>> until aver. ~2 1/4 G than keep that  for a few minutes. In that moment I
>> stopped. After that starts grow till ~4G with tendency to do jumps a bit
>> more that 4G.
>>
>>
>>>
>>>
 After seeing the source code of future I suppose that the memory (data
 are kept as #{} set) is not released. The task returns only integer so I do
 not think that might cause the problem.

>>>
>>> Can you provide more detail? You keep alluding to things that you don't
>>> provide code for, such as the sets of data.
>>>
>>
>>
>> The code is attached. However the important code is
>>
>> L123 .
>>   (let [;; keeps all data loading futures.
>> ;; waiting until all futures are finished
>> ;; should be done outside the main loop
>> data-loading-tasks (agent [])]
>>
>> L128
>> (doseq
>>  (let [r1 (long operation)]   L133
>>  (doseq
>> (let [r2 (v.v. long)]   L155
>>
>>   L163   (send-off data-loading-task conj-task)
>>
>>  )
>>  )
>> )
>> )
>>
>>
>> I guess first I will move data-loading-tasks list into one of inner lets.
>> Also I will create within an injecting function a separate abstract
>> function let inside. The task will populate tmp variable which will be
>> returned as a future result:
>>
>>
>> L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
>> result)
>>
>>
>>>
>>> --
>>> James Reeves
>>> booleanknot.com
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more 

Re: what does future do after fn finish ?

[Justin Smith]

this is exactly the kind of problem code I was describing - there's no
backpressure on existing future tasks to hold up the launching of more
futures - the work done by the agent calling conj is negligible. You need
to control the size of the pool of threads used, and you need to impose
back-pressure.

On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta 
wrote:

> On 31 January 2018 at 18:08, James Reeves  wrote:
>
>> On 31 January 2018 at 17:59, Jacek Grzebyta 
>> wrote:
>>
>>> I have application with quite intense tripe store populating ~30/40 k
>>> records per chunk (139 portions). The data are wrapped within the future:
>>>
>>> (conj agent (future (apply task args)))
>>>
>>>  and that all together is send-off into (agent []).
>>>
>>
>> What is "agent"? The first line of code indicates that it's a local
>> collection shadowing the code function, while the second code snippet
>> indicates that you're using the core agent function.
>>
>> Also why are you sending off to an agent?
>>
>
> I have ~8sec computing task for each input dataset which generates those
> records. After that I write it into disk (in software-specific
> transaction). I just wanted to separate hard computing and io operations. I
> created a side-effect method which is injected together with the dataset
> into a future. The futures are async collected within a list wrapped in
> agent. After the computing the main thread is waiting until all io tasks
> will be finished.
>
>
>>
>> At the end of the main thread function I just use await-for and after
>>> that:
>>>
>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>
>>
> As a control, tasks return number of written records.
>
>
>
>>
>>> For some reason I see the happy collecting (see attached screenshot of
>>> jconsole).
>>>
>>
>> "happy" = "heap"?
>>
>
> Both. As you can see on attached screenshot the heap usage grows easy
> until aver. ~2 1/4 G than keep that  for a few minutes. In that moment I
> stopped. After that starts grow till ~4G with tendency to do jumps a bit
> more that 4G.
>
>
>>
>>
>>> After seeing the source code of future I suppose that the memory (data
>>> are kept as #{} set) is not released. The task returns only integer so I do
>>> not think that might cause the problem.
>>>
>>
>> Can you provide more detail? You keep alluding to things that you don't
>> provide code for, such as the sets of data.
>>
>
>
> The code is attached. However the important code is
>
> L123 .
>   (let [;; keeps all data loading futures.
> ;; waiting until all futures are finished
> ;; should be done outside the main loop
> data-loading-tasks (agent [])]
>
> L128
> (doseq
>  (let [r1 (long operation)]   L133
>  (doseq
> (let [r2 (v.v. long)]   L155
>
>   L163   (send-off data-loading-task conj-task)
>
>  )
>  )
> )
> )
>
>
> I guess first I will move data-loading-tasks list into one of inner lets.
> Also I will create within an injecting function a separate abstract
> function let inside. The task will populate tmp variable which will be
> returned as a future result:
>
>
> L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
> result)
>
>
>>
>> --
>> James Reeves
>> booleanknot.com
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please 

Re: what does future do after fn finish ?

[Jacek Grzebyta]

On 31 January 2018 at 18:08, James Reeves  wrote:

> On 31 January 2018 at 17:59, Jacek Grzebyta 
> wrote:
>
>> I have application with quite intense tripe store populating ~30/40 k
>> records per chunk (139 portions). The data are wrapped within the future:
>>
>> (conj agent (future (apply task args)))
>>
>>  and that all together is send-off into (agent []).
>>
>
> What is "agent"? The first line of code indicates that it's a local
> collection shadowing the code function, while the second code snippet
> indicates that you're using the core agent function.
>
> Also why are you sending off to an agent?
>

I have ~8sec computing task for each input dataset which generates those
records. After that I write it into disk (in software-specific
transaction). I just wanted to separate hard computing and io operations. I
created a side-effect method which is injected together with the dataset
into a future. The futures are async collected within a list wrapped in
agent. After the computing the main thread is waiting until all io tasks
will be finished.


>
> At the end of the main thread function I just use await-for and after that:
>>
>> (reduce + (map #(deref %) @data-loading-tasks))
>>
>
As a control, tasks return number of written records.



>
>> For some reason I see the happy collecting (see attached screenshot of
>> jconsole).
>>
>
> "happy" = "heap"?
>

Both. As you can see on attached screenshot the heap usage grows easy until
aver. ~2 1/4 G than keep that  for a few minutes. In that moment I stopped.
After that starts grow till ~4G with tendency to do jumps a bit more that
4G.


>
>
>> After seeing the source code of future I suppose that the memory (data
>> are kept as #{} set) is not released. The task returns only integer so I do
>> not think that might cause the problem.
>>
>
> Can you provide more detail? You keep alluding to things that you don't
> provide code for, such as the sets of data.
>


The code is attached. However the important code is

L123 .
  (let [;; keeps all data loading futures.
;; waiting until all futures are finished
;; should be done outside the main loop
data-loading-tasks (agent [])]

L128
(doseq
 (let [r1 (long operation)]   L133
 (doseq
(let [r2 (v.v. long)]   L155

  L163   (send-off data-loading-task conj-task)

 )
 )
)
)


I guess first I will move data-loading-tasks list into one of inner lets.
Also I will create within an injecting function a separate abstract
function let inside. The task will populate tmp variable which will be
returned as a future result:


L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
result)


>
> --
> James Reeves
> booleanknot.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


experiment_result.clj
Description: Binary data

Re: what does future do after fn finish ?

[Justin Smith]

As a shot in the dark, a common problem with memory usage and futures that
I have seen is the antipattern of launching a future for each piece of data
in a collection. The problem that occurs is that the code works for small
input collections and a small load of running tasks / requests, but for a
larger input and more requests it uses up the heap easily. Then, a dev
assumes that somehow the future itself is leaking or otherwise taking up
space it shouldn't, when the true problem is that the undbounded creation
of futures happens to exhaust your available memory space. If you aren't
doing such a thing feel free to disregard.

On Wed, Jan 31, 2018 at 10:09 AM James Reeves  wrote:

> On 31 January 2018 at 17:59, Jacek Grzebyta 
> wrote:
>
>> I have application with quite intense tripe store populating ~30/40 k
>> records per chunk (139 portions). The data are wrapped within the future:
>>
>> (conj agent (future (apply task args)))
>>
>>  and that all together is send-off into (agent []).
>>
>
> What is "agent"? The first line of code indicates that it's a local
> collection shadowing the code function, while the second code snippet
> indicates that you're using the core agent function.
>
> Also why are you sending off to an agent?
>
> At the end of the main thread function I just use await-for and after that:
>>
>> (reduce + (map #(deref %) @data-loading-tasks))
>>
>> For some reason I see the happy collecting (see attached screenshot of
>> jconsole).
>>
>
> "happy" = "heap"?
>
>
>> After seeing the source code of future I suppose that the memory (data
>> are kept as #{} set) is not released. The task returns only integer so I do
>> not think that might cause the problem.
>>
>
> Can you provide more detail? You keep alluding to things that you don't
> provide code for, such as the sets of data.
>
> --
> James Reeves
> booleanknot.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what does future do after fn finish ?

[James Reeves]

On 31 January 2018 at 17:59, Jacek Grzebyta  wrote:

> I have application with quite intense tripe store populating ~30/40 k
> records per chunk (139 portions). The data are wrapped within the future:
>
> (conj agent (future (apply task args)))
>
>  and that all together is send-off into (agent []).
>

What is "agent"? The first line of code indicates that it's a local
collection shadowing the code function, while the second code snippet
indicates that you're using the core agent function.

Also why are you sending off to an agent?

At the end of the main thread function I just use await-for and after that:
>
> (reduce + (map #(deref %) @data-loading-tasks))
>
> For some reason I see the happy collecting (see attached screenshot of
> jconsole).
>

"happy" = "heap"?


> After seeing the source code of future I suppose that the memory (data are
> kept as #{} set) is not released. The task returns only integer so I do not
> think that might cause the problem.
>

Can you provide more detail? You keep alluding to things that you don't
provide code for, such as the sets of data.

-- 
James Reeves
booleanknot.com

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

what does future do after fn finish ?

[Jacek Grzebyta]

Hi,

I have application with quite intense tripe store populating ~30/40 k
records per chunk (139 portions). The data are wrapped within the future:

(conj agent (future (apply task args)))

 and that all together is send-off into (agent []).
At the end of the main thread function I just use await-for and after that:

(reduce + (map #(deref %) @data-loading-tasks))

For some reason I see the happy collecting (see attached screenshot of
jconsole). After seeing the source code of future I suppose that the memory
(data are kept as #{} set) is not released. The task returns only integer
so I do not think that might cause the problem.

Thanks a lot,
Jacek

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] lein-jlink: A leiningen plugin for custom JRE creation

[Didier]

This is awesome! I'll definitly give it a shot. Thanks for putting it together. 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] com.walmartlabs/lacinia 0.24.0, com.walmartlabs/lacinia-pedestal 0.6.0

[Howard Lewis Ship]

Thanks for the kind words!

On Tue, Jan 30, 2018 at 5:31 PM, Matching Socks 
wrote:

> Lacinia is well done, and, in combination with Clojure, a terrific
> labor-saving device.
>
> In fact, Lacinia is so well-conceived that it makes GraphQL itself glow
> with labor-saving virtue.
>
> GraphQL is anyway splendidly low-impact for in-the-browser clients.  In
> return, the burdens of GraphQL are borne by the server.  Lacinia bears
> them.  The chores left to your Clojure data service are considerably less
> onerous than GraphQL.
>
> Meanwhile, Lacinia seems to have no overbearing opinions.  For example,
> you may analyze the whole breadth and depth of a Lacinia-parsed GraphQL
> query and optimize retrieval from some back-end database.  But you need
> not.  And Lacinia won't even notice the difference.
>
> Lowering the barrier to providing GraphQL services, Lacinia promotes an
> open and interoperable internet.
>
> Five gold stars.
>
>> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Howard M. Lewis Ship

Senior Mobile Developer at Walmart Labs

Creator of Apache Tapestry

(971) 678-5210
http://howardlewisship.com
@hlship

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [ANN] lein-jlink: A leiningen plugin for custom JRE creation

[Ning Sun]

Hi Laurens,

I haven't got chance to play with proguard. I doubt of it works well with 
non-Android projects. For now I think the best method is to keep you dependency 
tree from bloated.

On January 31, 2018 12:09:57 AM GMT+08:00, Laurens Van Houtven <_...@lvh.io> 
wrote:
>This is great! Thanks Ning! Do you have any
>experience/pointers/projects
>around reducing deployable size on the code side (as opposed to
>runtime)?
>They seem like very related concerns. I've messed with proguard but it
>mostly seems to be great at producing jars that are 10% of the size and
>don't actually work ;-)
>
>On Tue, Jan 30, 2018 at 8:24 AM, Ning Sun 
>wrote:
>
>> Hi clojurians,
>>
>> I just created a Leiningen plugin that creates custom Java Runtime
>> Environment based on you configuration. The custom JRE can be as
>small
>> as 29MB, and it's fully capable to run a Ring web app.
>>
>> With this plugin:
>> 0. You can test your Clojure application against the custom JRE;
>> 1. You can distribute your Clojure application with runtime attached
>and
>> don't have to assume your user has JRE installed;
>> 2. You can create minimal Docker image for your Clojure service, it
>> takes less than 50MB at minimal setup, comparing to others can be
>350MB+;
>> 3. You discover.
>>
>> The plugin is here:
>> https://github.com/sunng87/lein-jlink
>>
>> I have a blog post about it:
>> https://sunng.info/blog/custom-jre-for-clojure-app-distribution.html
>>
>> And I'd like to hear your ideas about it.
>>
>> Don't forget it requires JDK9 or 10. Because jlink is a feature
>> introduced in JDK9.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient
>with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google
>Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it,
>send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>-- 
>You received this message because you are subscribed to the Google
>Groups "Clojure" group.
>To post to this group, send email to clojure@googlegroups.com
>Note that posts from new members are moderated - please be patient with
>your first post.
>To unsubscribe from this group, send email to
>clojure+unsubscr...@googlegroups.com
>For more options, visit this group at
>http://groups.google.com/group/clojure?hl=en
>--- 
>You received this message because you are subscribed to the Google
>Groups "Clojure" group.
>To unsubscribe from this group and stop receiving emails from it, send
>an email to clojure+unsubscr...@googlegroups.com.
>For more options, visit https://groups.google.com/d/optout.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] Logic Workbench lwb Rev1.0.1

['Burt' via Clojure]

The Logic Workbench lwb is a box of tools for the propositional logic, 
predicate logic, and linear temporal logic.
It's a playground.

More about lwb: Intro .

Comments welcome

--
Burkhardt Renz
THM (Technische Hochschule Mittelhessen) 

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.