Re: what does future do after fn finish ?

[Justin Smith]

-> is just a list transform performed after reading your code into a list
data structure containing symbols, and before compiling to byte code - it
doesn't do anything directly.

On Fri, Feb 2, 2018 at 3:55 PM Jacek Grzebyta 
wrote:

> OK I found what makes the memory leak.
>
> In the project I work with I use a  java Model class which is java
> Collection proxy/facade for a single transaction. Unfortunately it's not
> thread safe. In a few places I passed single instance of model into several
> threads Also I requested the instance with -> which makes new thread as
> well. I was surprised that -> makes trouble but after thinking that might
> be expected. Especially that internally the wrapper doesn't do simple
> mapping - it uses some iterator, etc. Anyway the machinery is fragile. It
> seems I need to rewrite code and replace all multithreading parts by
> something simpler.
>
>
>
> If you want see the stacktrace just look at:
> https://github.com/jgrzebyta/triple-loader/issues/53
>
> On 2 February 2018 at 11:16, Jacek Grzebyta 
> wrote:
>
>>
>> On 2 February 2018 at 08:34, Niels van Klaveren <
>> niels.vanklave...@gmail.com> wrote:
>>
>>> +1 for Claypoole, it removed the needs of using agents or futures in 95%
>>> of the cases in my code.
>>>
>>>
>> Thanks a lot. I modify the code using claypoole. I imagine with-shutdown
>> will close the pool properly after finish all tasks so there is no need to
>> watch them?
>>
>> J
>>
>>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what does future do after fn finish ?

[Jacek Grzebyta]

OK I found what makes the memory leak.

In the project I work with I use a  java Model class which is java
Collection proxy/facade for a single transaction. Unfortunately it's not
thread safe. In a few places I passed single instance of model into several
threads Also I requested the instance with -> which makes new thread as
well. I was surprised that -> makes trouble but after thinking that might
be expected. Especially that internally the wrapper doesn't do simple
mapping - it uses some iterator, etc. Anyway the machinery is fragile. It
seems I need to rewrite code and replace all multithreading parts by
something simpler.



If you want see the stacktrace just look at:
https://github.com/jgrzebyta/triple-loader/issues/53

On 2 February 2018 at 11:16, Jacek Grzebyta  wrote:

>
> On 2 February 2018 at 08:34, Niels van Klaveren <
> niels.vanklave...@gmail.com> wrote:
>
>> +1 for Claypoole, it removed the needs of using agents or futures in 95%
>> of the cases in my code.
>>
>>
> Thanks a lot. I modify the code using claypoole. I imagine with-shutdown
> will close the pool properly after finish all tasks so there is no need to
> watch them?
>
> J
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] walmartlabs/system-viz 0.3.0

[Howard Lewis Ship]

system-viz is a tiny, simple library to visualize a system, constructed
using Stuart Sierra's component library.

This release adds limited ability to customize node attributes in the
output graph, along with improved documentation.
[image: Inline image 1]
https://github.com/walmartlabs/system-viz

-- 
Howard M. Lewis Ship

Senior Mobile Developer at Walmart Labs

Creator of Apache Tapestry

(971) 678-5210
http://howardlewisship.com
@hlship

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ANN] martian 0.1.7 - UUID support

[Oliver Hine]

Hi everyone,

I am pleased to announce I have just released martian 0.1.7 
 

Martian provides an abstraction allowing you to describe the HTTP endpoints 
that you call without methods, urls and query parameters leaking into your 
application code, allowing you to work with pure Clojure data. It builds on 
the flexible interceptor pattern familiar to users of pedestal and supports 
APIs described with Swagger.

This release adds support for:

   - UUIDs as added by @markdingram  in #47 
    - thank you
   - A subscription to find out what requests are pending - #39 
   

This release fixes:

   - Reflection warnings in the JVM libraries

Enjoy!

Cheers,
Oliy

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what does future do after fn finish ?

[Jacek Grzebyta]

On 2 February 2018 at 08:34, Niels van Klaveren  wrote:

> +1 for Claypoole, it removed the needs of using agents or futures in 95%
> of the cases in my code.
>
>
Thanks a lot. I modify the code using claypoole. I imagine with-shutdown
will close the pool properly after finish all tasks so there is no need to
watch them?

J

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what does future do after fn finish ?

[Niels van Klaveren]

+1 for Claypoole, it removed the needs of using agents or futures in 95% of 
the cases in my code.

On Thursday, February 1, 2018 at 9:54:36 PM UTC+1, Alan Thompson wrote:
>
> You may find that using the Claypoole library is the easiest way to handle 
> threadpools:   https://github.com/TheClimateCorporation/claypoole
> Alan
>
> On Thu, Feb 1, 2018 at 11:16 AM, Justin Smith  > wrote:
>
>> yes, that's the idea exactly
>>
>> also, you might want more fine grained control of how much parallelism 
>> occurs (eg. if every thread is writing to the same physical device, you can 
>> often get better throughput by not parallelizing at all, or keeping the 
>> parallelism quite limited - it's worth experimenting) - there are good ways 
>> to control that using ThreadPoolExecutor directly, or using 
>> clojure.lang.PersistentQueue/EMPTY as a control construct, or core.async 
>> channels, or ztellman's manifold library, or the claypoole threading library
>>
>> On Thu, Feb 1, 2018, 03:44 Jacek Grzebyta > > wrote:
>>
>>> Thanks folks. I see now! It should be a list of agents not list of 
>>> futures within agent.  Also any task sent to a agent is processed 
>>> within a thread anyway so I do not need to add future...
>>>
>>> On 1 February 2018 at 02:17, John Newman >> > wrote:
>>>
 Ah, he's using one agent, I see.

 On Jan 31, 2018 9:15 PM, "John Newman"  
 wrote:

> Multiple sen-doffs to one agent will serialize it's calls, but 
> spawning agents on each new task will spawn threads on a bounded thread 
> pool, I believe.
>
> On Jan 31, 2018 8:32 PM, "Justin Smith"  > wrote:
>
>> Doing all the actions via one agent means that the actions are 
>> serialized though - you end up with no performance improvement over 
>> doing 
>> them all in a doseq in one future - the right way to do this tends to be 
>> trickier than it looks at first glance, and depends on your 
>> requirements. 
>> agents, the claypoole library, and reducers are all potentially useful. 
>> If 
>> parallelization leads to complex coordination needs, core.async can help 
>> too.
>>
>> On Wed, Jan 31, 2018 at 5:18 PM John Newman > > wrote:
>>
>>> Agents manage a pool of threads for you. Try doing it without the 
>>> future call and see if that works (unless you're trying to do something 
>>> else).
>>>
>>> John
>>>
>>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta >> > wrote:
>>>
 Thanks a lot. I will check it tomorrow. 

 J

 On 1 Feb 2018 12:12 a.m., "Justin Smith"  wrote:

> this is exactly the kind of problem code I was describing - 
> there's no backpressure on existing future tasks to hold up the 
> launching 
> of more futures - the work done by the agent calling conj is 
> negligible. 
> You need to control the size of the pool of threads used, and you 
> need to 
> impose back-pressure.
>
> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <
> grzeby...@gmail.com > wrote:
>
>> On 31 January 2018 at 18:08, James Reeves > > wrote:
>>
>>> On 31 January 2018 at 17:59, Jacek Grzebyta >> > wrote:
>>>
 I have application with quite intense tripe store populating 
 ~30/40 k records per chunk (139 portions). The data are wrapped 
 within the 
 future:

 (conj agent (future (apply task args)))

  and that all together is send-off into (agent []). 

>>>
>>> What is "agent"? The first line of code indicates that it's a 
>>> local collection shadowing the code function, while the second code 
>>> snippet 
>>> indicates that you're using the core agent function.
>>>
>>> Also why are you sending off to an agent?
>>>
>>  
>> I have ~8sec computing task for each input dataset which 
>> generates those records. After that I write it into disk (in 
>> software-specific transaction). I just wanted to separate hard 
>> computing 
>> and io operations. I created a side-effect method which is injected 
>> together with the dataset into a future. The futures are async 
>> collected 
>> within a list wrapped in agent. After the computing the main thread 
>> is 
>> waiting until all io tasks will be finished. 
>>  
>>
>>>
>>> At the end of the main thread function I just use await-for and 
 after that: