[Newbe] how to 'unpack' clojure internal instance

[Jacek Grzebyta]

Hi All,

Sorry for trivial question but I am a newbe in Clojure.
I need to manage some object which is created by calling java method:


 (let [result #(.getStatements c nil nil nil (into-array Resource '[]))]
>   (log/debug (format "Result class: %s" (type result)))
>   (map #(log/debug "Element: " %1) result)
>   )
>

I know the result should be type of "CloseableIteration
http://rdf4j.org/doc/4/apidocs/org/openrdf/model/Statement.html>,
SailException
>" but
Clojure shows me the error that class
"triple.loader_test$test_repository$result__849" cannot be processed. As a
Java programmer I would say that clojure wraps/makes internal proxy from/
the origin object.
I have added all required class names into file head by namespace
declaration.

How can I "unpack" the pack? Maybe my approach is totally wrong? I am even
do not know what is the name of that issues so google is helpless in my
case.
Thanks a lot for help.

Best regards,
Jacek

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Newbe] how to 'unpack' clojure internal instance

[Jacek Grzebyta]

Thanks Alex. That was the problem. I have wrapped the results in anonymous 
function. I overuse anonymous functions due to my small experience with 
common lisp lambdas.

On Monday, 25 January 2016 15:59:57 UTC, Alex Miller wrote:
>
> I think your most immediate issue is the "#" after result - that makes 
> this into an anonymous function instance. Instead, you want to actually 
> evaluate that expression in the let:
>
>  (let [result (.getStatements c nil nil nil (into-array Resource '[]))]
>   (log/debug (format "Result class: %s" (type result)))
>   (map #(log/debug "Element: " %1) result))
>
> However, as far as I can tell, you will not be able to use the 
> CloseableIteration result as an input to map in the last line as 
> CloseableIteration is not something Clojure understands - that library 
> seems to have its own iterator hierarchy without any ties to the standard 
> Java iterators (seems like a questionable design choice to me). Due to this 
> you either need to create a wrapper to turn their iterator into a sequence, 
> something like this:
>
> (defn seq-iteration [^CloseableIteration iter]
>   (lazy-seq
> (if (.hasNext iter)
>   (cons (.next iter) (seq-iteration iter))
>   (do (.close iter)
> nil
>
> Here, I'm not handling iterator exceptions if they occur - they will just 
> bubble out during use. There's probably more that would need to be done to 
> ensure .close is called on the iter in the case of an exception during 
> .next.
>
> Or you need to traverse the iterator in a loop/recur:
>
> (let [result ^CloseableIteration (.getStatements c nil nil nil (into-array 
> Resource '[]))]
>   (try
> (loop []
>   (when (.hasNext result)
> (log/debug "Element: " (.next result))
> (recur)))
> (catch Exception e
>   (log/debug "Exception: " (.getMessage e)))
> (finally
>   (.close result)))
>
> Because the CloseableIteration can throw exceptions, I wrapped the whole 
> thing in a try/catch and a finally to .close the result. If you want to 
> accumulate some state during this recursion, you can add a loop binding to 
> collect info and return it at the end instead.
>
>
>
> On Monday, January 25, 2016 at 9:38:28 AM UTC-6, Jacek Grzebyta wrote:
>>
>> Hi All,
>>
>> Sorry for trivial question but I am a newbe in Clojure. 
>> I need to manage some object which is created by calling java method:
>>
>>
>>  (let [result #(.getStatements c nil nil nil (into-array Resource '[]))]
>>>   (log/debug (format "Result class: %s" (type result)))
>>>   (map #(log/debug "Element: " %1) result)
>>>   )
>>>
>>
>> I know the result should be type of "CloseableIteration 
>> <http://www.google.com/url?q=http%3A%2F%2Frdf4j.org%2Fdoc%2F4%2Fapidocs%2Finfo%2Faduna%2Fiteration%2FCloseableIteration.html&sa=D&sntz=1&usg=AFQjCNEKEPHoxbdwlQIaFE_1zCUlJ1e1Fw>>  
>> extends Statement 
>> <http://rdf4j.org/doc/4/apidocs/org/openrdf/model/Statement.html>,
>> SailException 
>> <http://rdf4j.org/doc/4/apidocs/org/openrdf/sail/SailException.html>>" 
>> but Clojure shows me the error that class 
>> "triple.loader_test$test_repository$result__849" cannot be processed. As 
>> a Java programmer I would say that clojure wraps/makes internal proxy from/ 
>> the origin object. 
>> I have added all required class names into file head by namespace 
>> declaration. 
>>
>> How can I "unpack" the pack? Maybe my approach is totally wrong? I am 
>> even do not know what is the name of that issues so google is helpless in 
>> my case. 
>> Thanks a lot for help.
>>
>> Best regards,
>> Jacek
>>
>>
>>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

What is replacement to maven-assembly-plugin in boot-clj environment?

[Jacek Grzebyta]

I need to build more complex package using boot-cl. I have cited my request
put on the StackOverflow:

Copy of What is replacement to maven-assembly-plugin in boot-clj
environment? 

I created a project which (semi-) finally gives jar file. The file is for
> command line usage. In next step I would like to add a few shell scripts to
> call relevant classes within jar. Also I would like to keep externally
> log4j.xml file.
>
> The obvious next step is to build a package to bring all components
> together. However I could not find a replacement of maven-assembly-plugin
> for boot-clj builder.
>
> Maybe there is a way doing that without that plugin: I mean that maybe
> boot has some build-in feature that solve the issue?
>

Thanks a lot,
Jacek

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

clojure alternative to AspectJ proxy in Java's code

[Jacek Grzebyta]

Hi,

I have a Clojure project which strongly uses some external java framework
(if you want to know it is Eclipse RDF4j). I need to override some method
within a java class. The issue is I am not able to inject my own
implementation of that class because it is in the middle of some complex
hierarchy.

In another Java (+ Spring Framework) project I had similar problem and I
had fixed that using AspectJ proxy which had had a breakpoint in calling a
particular java method. I read in many places that Clojure does not need it
and some explanation based on pure clojure examples. I would like to know
if Clojure is able to manage java code.

Quite often examples built a proxy with combination Java Class + protocol
and than injected the instance into somewhere. I thing that it does not
cover following example. The problems I found difficult are:

1. Distance between my code and Foo class is like 3 or 4 classes
2. Programmers quite strongly uses creation de novo rather than injection
already prepared instance

Thanks a lot for your help,
Jacek Grzebyta



++ java code+

class Baz {

// I need to proxy this method
protected List doAction(String arg 1, String arg2)

// weird work which I need to change

}


 Foo class

class Foo {

// cut


// elswhere

Baz jobber = new Baz(); // it creates de novo injection not supported


public String mySuperMethod(List args4) {

jobber.doAction(someList)

}


// cut
}


++ end ++

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

unit tests on gen-class outcome

[Jacek Grzebyta]

Hi,

I have a project which uses some java project. I made classes using
gen-class. Than in unit test just wanted to test it (clojure.test enviro).
If I run tests in emacs+cider they works. If I run test in the command line
I have class not found exception:

boot testing aot run-test

task testing adds test/ directory to source
run-test just executes test. It works if there are tests against clojure
code.

Adding paths to *.class files into CLASSPATH does not help.

Thanks a lot,
Jacek

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Releases timetable

[Jacek Grzebyta]

Hi,

Where can I find any timetable for next releases? Or does exist any planned
release date for 1.9 version?

Thanks,
Jacek

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Clojure design patterns

[Jacek Grzebyta]

That is my latest discovery. Very good blog with common design patterns
implemented in clojure:

http://mishadoff.com/blog/clojure-design-patterns/

All the best,
J

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Multi-threading with side-effects processing

[Jacek Grzebyta]

Hi,

I need to load a Java data-base presented as java Collection. Basically it
is org.eclipse.rdf4j.sail.model.SailModel   (
http://docs.rdf4j.org/javadoc/latest/org/eclipse/rdf4j/sail/model/SailModel.html).
Anyway the data are stored on the disk but the object is not thread safe.

In my program I creates threads which populate the model asynchronously.
each thread (pmap) follows pattern:

in the main defn

f-model .. ;; final target


(defn [args ]
(let [tmp-model1 ...
tmp-model2  .
.]
(locking f-model
(.addAll f-model tmp-model1)
...
)))

But I do not think it is a good solution - I found my program stop working
shortly after building up the final model.
In my previous version threads return single tmp-models but that caused out
of memory easy. What else can I try?

Regads,
Jacek

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Multi-threading with side-effects processing

[Jacek Grzebyta]

Finally I fixed that:
Threads populate simple vector within a transactions and I do finally
(after pmap) loading vector content into the final model.
That was easy because I do not need to do any hi level operations on the
data.
Also I noticed the final fingerprint is much smaller than in case of rdf4j
Models juggling.

Regards,
Jacek

On 1 December 2017 at 12:18, Jacek Grzebyta  wrote:

> Hi,
>
> I need to load a Java data-base presented as java Collection. Basically it
> is org.eclipse.rdf4j.sail.model.SailModel   (http://docs.rdf4j.org/
> javadoc/latest/org/eclipse/rdf4j/sail/model/SailModel.html). Anyway the
> data are stored on the disk but the object is not thread safe.
>
> In my program I creates threads which populate the model asynchronously.
> each thread (pmap) follows pattern:
>
> in the main defn
>
> f-model .. ;; final target
>
>
> (defn [args ]
> (let [tmp-model1 ...
> tmp-model2  .
> .]
> (locking f-model
> (.addAll f-model tmp-model1)
> ...
> )))
>
> But I do not think it is a good solution - I found my program stop working
> shortly after building up the final model.
> In my previous version threads return single tmp-models but that caused
> out of memory easy. What else can I try?
>
> Regads,
> Jacek
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

partition-all in clojure.core.asyncn channel does not work

[Jacek Grzebyta]

Hi,

I have to populate a triple store with a big number of data (~38k records x
12) and there is a deadly narrow bottleneck - IO operations speed. To fix
it I did:
1. To avoid threads overflow I put all compute results into channel.
2. Loading data in chunks is better than single transaction for single
record

I tried to do by creating channel with poputale-all traversal but it seems
doesn't work properly. In the following mock example it works when the
chunk size is  equal the data vector (i.e. 6): "value:  [of made is fruit
soup Berry]" - for now I do not care the order.

(let [q (a/chan 500 (partition-all 6))
  in ["Berry" "soup" "is" "made" "of" "fruit"]]
  (a/go-loop [j (a/! q itm


I cannot see any problem. How can I solve it? In the following example
chunk size should be max 6? I expected partition-all will work the same way
as itself:

(partition-all 5 ["Berry" "soup" "is" "made" "of" "fruit"]) ==>
(("Berry" "soup" "is" "made" "of") ("fruit"))

Thanks a lot,
Jacek

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: partition-all in clojure.core.asyncn channel does not work

[Jacek Grzebyta]

Thanks a lot.

I found you changed the finish slightly - you close q within go body. Where
should I put close! in case if I load a channel in many places within
several methods? I afraid the go body is done

E.g.:

(defn main-method [db-api]
  (let [main-queue (chan 2) ]
(sink-method main-queue db-api)  ;; population db within go-loop body
(loading-method-1 main-queue & rest)
(loading-method-n main-queue & rest)))

And methods pattern is like:

 (def loading-method-.. [ q ]
   (doseq
 (loading-sub-method q)
 (map
#(doseq

   (go
  >! q (processing i j k l m))
) .. ) ... ))


On 6 December 2017 at 11:50, Ray Miller  wrote:

> On 6 December 2017 at 11:23, Jacek Grzebyta 
> wrote:
>>
>> I have to populate a triple store with a big number of data (~38k records
>> x 12) and there is a deadly narrow bottleneck - IO operations speed. To fix
>> it I did:
>> 1. To avoid threads overflow I put all compute results into channel.
>> 2. Loading data in chunks is better than single transaction for
>> single record
>>
>> I tried to do by creating channel with poputale-all traversal but it
>> seems doesn't work properly. In the following mock example it works when
>> the chunk size is  equal the data vector (i.e. 6): "value:  [of made is
>> fruit soup Berry]" - for now I do not care the order.
>>
>> (let [q (a/chan 500 (partition-all 6))
>>   in ["Berry" "soup" "is" "made" "of" "fruit"]]
>>   (a/go-loop [j (a/> (when j
>>   (println "value: " j)
>>   (recur (a/>   (doseq [itm in]
>> (a/go (a/>! q itm
>>
>>
>> I cannot see any problem. How can I solve it? In the following example
>> chunk size should be max 6? I expected partition-all will work the same way
>> as itself:
>>
>> (partition-all 5 ["Berry" "soup" "is" "made" "of" "fruit"]) ==>
>> (("Berry" "soup" "is" "made" "of") ("fruit"))
>>
>>
> I think you just need to close the channel when you've finished populating
> it:
>
> (let [q (a/chan 500 (partition-all 5))
>   in ["Berry" "soup" "is" "made" "of" "fruit"]]
>   (a/go-loop [j (a/ (when j
>   (println "value: " j)
>   (recur (a/   (a/go
> (doseq [itm in]
>   (a/>! q itm))
> (a/close! q)))
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: partition-all in clojure.core.asyncn channel does not work

[Jacek Grzebyta]

Sorry. I put close at he and of the main method but I afraid the main
thread will reach close method before all the data will be put to the queue.
Is any way to check a state of the queue? If I handle channels returned by
all *-sub-methods with go inside I could check state if was finished?

Regards,
J



On 6 December 2017 at 12:21, Jacek Grzebyta  wrote:

> Thanks a lot.
>
> I found you changed the finish slightly - you close q within go body.
> Where should I put close! in case if I load a channel in many places within
> several methods? I afraid the go body is done
>
> E.g.:
>
> (defn main-method [db-api]
>   (let [main-queue (chan 2) ]
> (sink-method main-queue db-api)  ;; population db within go-loop body
> (loading-method-1 main-queue & rest)
> (loading-method-n main-queue & rest)))
>
> And methods pattern is like:
>
>  (def loading-method-.. [ q ]
>(doseq
>  (loading-sub-method q)
>  (map
> #(doseq
>
>(go
>   >! q (processing i j k l m))
> ) .. ) ... ))
>
>
> On 6 December 2017 at 11:50, Ray Miller  wrote:
>
>> On 6 December 2017 at 11:23, Jacek Grzebyta 
>> wrote:
>>>
>>> I have to populate a triple store with a big number of data (~38k
>>> records x 12) and there is a deadly narrow bottleneck - IO operations
>>> speed. To fix it I did:
>>> 1. To avoid threads overflow I put all compute results into channel.
>>> 2. Loading data in chunks is better than single transaction for
>>> single record
>>>
>>> I tried to do by creating channel with poputale-all traversal but it
>>> seems doesn't work properly. In the following mock example it works when
>>> the chunk size is  equal the data vector (i.e. 6): "value:  [of made is
>>> fruit soup Berry]" - for now I do not care the order.
>>>
>>> (let [q (a/chan 500 (partition-all 6))
>>>   in ["Berry" "soup" "is" "made" "of" "fruit"]]
>>>   (a/go-loop [j (a/>> (when j
>>>   (println "value: " j)
>>>   (recur (a/>>   (doseq [itm in]
>>> (a/go (a/>! q itm
>>>
>>>
>>> I cannot see any problem. How can I solve it? In the following example
>>> chunk size should be max 6? I expected partition-all will work the same way
>>> as itself:
>>>
>>> (partition-all 5 ["Berry" "soup" "is" "made" "of" "fruit"]) ==>
>>> (("Berry" "soup" "is" "made" "of") ("fruit"))
>>>
>>>
>> I think you just need to close the channel when you've finished
>> populating it:
>>
>> (let [q (a/chan 500 (partition-all 5))
>>   in ["Berry" "soup" "is" "made" "of" "fruit"]]
>>   (a/go-loop [j (a/> (when j
>>   (println "value: " j)
>>   (recur (a/>   (a/go
>> (doseq [itm in]
>>   (a/>! q itm))
>> (a/close! q)))
>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: partition-all in clojure.core.asyncn channel does not work

[Jacek Grzebyta]

So I found how to manage all sub-*-methods ( wrote:

> Sorry. I put close at he and of the main method but I afraid the main
> thread will reach close method before all the data will be put to the queue.
> Is any way to check a state of the queue? If I handle channels returned by
> all *-sub-methods with go inside I could check state if was finished?
>
> Regards,
> J
>
>
>
> On 6 December 2017 at 12:21, Jacek Grzebyta 
> wrote:
>
>> Thanks a lot.
>>
>> I found you changed the finish slightly - you close q within go body.
>> Where should I put close! in case if I load a channel in many places within
>> several methods? I afraid the go body is done
>>
>> E.g.:
>>
>> (defn main-method [db-api]
>>   (let [main-queue (chan 2) ]
>> (sink-method main-queue db-api)  ;; population db within go-loop body
>> (loading-method-1 main-queue & rest)
>> (loading-method-n main-queue & rest)))
>>
>> And methods pattern is like:
>>
>>  (def loading-method-.. [ q ]
>>(doseq
>>  (loading-sub-method q)
>>  (map
>> #(doseq
>>
>>(go
>>       >! q (processing i j k l m))
>> ) .. ) ... ))
>>
>>
>> On 6 December 2017 at 11:50, Ray Miller  wrote:
>>
>>> On 6 December 2017 at 11:23, Jacek Grzebyta 
>>> wrote:
>>>>
>>>> I have to populate a triple store with a big number of data (~38k
>>>> records x 12) and there is a deadly narrow bottleneck - IO operations
>>>> speed. To fix it I did:
>>>> 1. To avoid threads overflow I put all compute results into
>>>> channel.
>>>> 2. Loading data in chunks is better than single transaction for
>>>> single record
>>>>
>>>> I tried to do by creating channel with poputale-all traversal but it
>>>> seems doesn't work properly. In the following mock example it works when
>>>> the chunk size is  equal the data vector (i.e. 6): "value:  [of made is
>>>> fruit soup Berry]" - for now I do not care the order.
>>>>
>>>> (let [q (a/chan 500 (partition-all 6))
>>>>   in ["Berry" "soup" "is" "made" "of" "fruit"]]
>>>>   (a/go-loop [j (a/>>> (when j
>>>>   (println "value: " j)
>>>>   (recur (a/>>>   (doseq [itm in]
>>>> (a/go (a/>! q itm
>>>>
>>>>
>>>> I cannot see any problem. How can I solve it? In the following example
>>>> chunk size should be max 6? I expected partition-all will work the same way
>>>> as itself:
>>>>
>>>> (partition-all 5 ["Berry" "soup" "is" "made" "of" "fruit"]) ==>
>>>> (("Berry" "soup" "is" "made" "of") ("fruit"))
>>>>
>>>>
>>> I think you just need to close the channel when you've finished
>>> populating it:
>>>
>>> (let [q (a/chan 500 (partition-all 5))
>>>   in ["Berry" "soup" "is" "made" "of" "fruit"]]
>>>   (a/go-loop [j (a/>> (when j
>>>   (println "value: " j)
>>>   (recur (a/>>   (a/go
>>> (doseq [itm in]
>>>   (a/>! q itm))
>>> (a/close! q)))
>>>
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to clojure+unsubscr...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: partition-all in clojure.core.asyncn channel does not work

[Jacek Grzebyta]

Finally I will add atomically changed flag `isFinished`. And separate throw
to close if the flag is true.

On 6 December 2017 at 13:30, Jacek Grzebyta  wrote:

> So I found how to manage all sub-*-methods ( finish) but still I do not know where should I put close!. If I do at the
> end of the main method than the main channel is closed before having all
> data and db stays empty.
>
> J.
>
> On 6 December 2017 at 12:31, Jacek Grzebyta 
> wrote:
>
>> Sorry. I put close at he and of the main method but I afraid the main
>> thread will reach close method before all the data will be put to the queue.
>> Is any way to check a state of the queue? If I handle channels returned
>> by all *-sub-methods with go inside I could check state if was finished?
>>
>> Regards,
>> J
>>
>>
>>
>> On 6 December 2017 at 12:21, Jacek Grzebyta 
>> wrote:
>>
>>> Thanks a lot.
>>>
>>> I found you changed the finish slightly - you close q within go body.
>>> Where should I put close! in case if I load a channel in many places within
>>> several methods? I afraid the go body is done
>>>
>>> E.g.:
>>>
>>> (defn main-method [db-api]
>>>   (let [main-queue (chan 2) ]
>>> (sink-method main-queue db-api)  ;; population db within go-loop body
>>> (loading-method-1 main-queue & rest)
>>> (loading-method-n main-queue & rest)))
>>>
>>> And methods pattern is like:
>>>
>>>  (def loading-method-.. [ q ]
>>>(doseq
>>>  (loading-sub-method q)
>>>  (map
>>> #(doseq
>>>
>>>(go
>>>   >! q (processing i j k l m))
>>> ) .. ) ... ))
>>>
>>>
>>> On 6 December 2017 at 11:50, Ray Miller  wrote:
>>>
>>>> On 6 December 2017 at 11:23, Jacek Grzebyta 
>>>> wrote:
>>>>>
>>>>> I have to populate a triple store with a big number of data (~38k
>>>>> records x 12) and there is a deadly narrow bottleneck - IO operations
>>>>> speed. To fix it I did:
>>>>> 1. To avoid threads overflow I put all compute results into
>>>>> channel.
>>>>> 2. Loading data in chunks is better than single transaction for
>>>>> single record
>>>>>
>>>>> I tried to do by creating channel with poputale-all traversal but it
>>>>> seems doesn't work properly. In the following mock example it works when
>>>>> the chunk size is  equal the data vector (i.e. 6): "value:  [of made is
>>>>> fruit soup Berry]" - for now I do not care the order.
>>>>>
>>>>> (let [q (a/chan 500 (partition-all 6))
>>>>>   in ["Berry" "soup" "is" "made" "of" "fruit"]]
>>>>>   (a/go-loop [j (a/>>>> (when j
>>>>>   (println "value: " j)
>>>>>   (recur (a/>>>>   (doseq [itm in]
>>>>> (a/go (a/>! q itm
>>>>>
>>>>>
>>>>> I cannot see any problem. How can I solve it? In the following example
>>>>> chunk size should be max 6? I expected partition-all will work the same 
>>>>> way
>>>>> as itself:
>>>>>
>>>>> (partition-all 5 ["Berry" "soup" "is" "made" "of" "fruit"]) ==>
>>>>> (("Berry" "soup" "is" "made" "of") ("fruit"))
>>>>>
>>>>>
>>>> I think you just need to close the channel when you've finished
>>>> populating it:
>>>>
>>>> (let [q (a/chan 500 (partition-all 5))
>>>>   in ["Berry" "soup" "is" "made" "of" "fruit"]]
>>>>   (a/go-loop [j (a/>>> (when j
>>>>   (println "value: " j)
>>>>   (recur (a/>>>   (a/go
>>>> (doseq [itm in]
>>>>   (a/>! q itm))
>>>> (a/close! q)))
>>>>
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To post to this group, send email to clojure@googlegroups.com
>>>> Note that posts from new members are moderated - please be patient with
>>>> your first post.
>>>> To unsubscribe from this group, send email to
>>>> clojure+unsubscr...@googlegroups.com
>>>> For more options, visit this group at
>>>> http://groups.google.com/group/clojure?hl=en
>>>> ---
>>>> You received this message because you are subscribed to the Google
>>>> Groups "Clojure" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to clojure+unsubscr...@googlegroups.com.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>>
>>>
>>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Slow -main function termination by pmap

[Jacek Grzebyta]

Hi,

I have multi -mains project. Thus the execution looks like:

java -cp location/file.jar some.method ..


One -main method looks like:

(defn -main
  [& args]
  (let [validated (validate-args args)]
(if (:msg validated)
  (println (st/join \newline (:msg validated)))
  (run validated))
(log/debug "finish")))


There is nothing special in that except in the run time I have displayed
the last log "finish" and the program termination is done after ~1 min
later what is annoying. I found it is caused by deeply hidden pmap even if
it is wrapped by doall. I thought after pmap returns results all threads
are finished i.e. the main thread waits until all sub threads will be
finished. If I am right what else can cause the delay? The threads pool
manager?

Tanks a lot,
Jacek

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: Slow -main function termination by pmap

[Jacek Grzebyta]

That solved my problem! Thanks l lot!

Jacek

On 19 December 2017 at 20:08, Timothy Baldridge 
wrote:

> Sometimes there's some weird interop with stuff like pmap and program
> termination. What happens if you end your program with this function:
> https://clojuredocs.org/clojure.core/shutdown-agents
>
> On Tue, Dec 19, 2017 at 1:05 PM, Jacek Grzebyta 
> wrote:
>
>> Hi,
>>
>> I have multi -mains project. Thus the execution looks like:
>>
>> java -cp location/file.jar some.method ..
>>
>>
>> One -main method looks like:
>>
>> (defn -main
>>   [& args]
>>   (let [validated (validate-args args)]
>> (if (:msg validated)
>>   (println (st/join \newline (:msg validated)))
>>   (run validated))
>> (log/debug "finish")))
>>
>>
>> There is nothing special in that except in the run time I have displayed
>> the last log "finish" and the program termination is done after ~1 min
>> later what is annoying. I found it is caused by deeply hidden pmap even if
>> it is wrapped by doall. I thought after pmap returns results all threads
>> are finished i.e. the main thread waits until all sub threads will be
>> finished. If I am right what else can cause the delay? The threads pool
>> manager?
>>
>> Tanks a lot,
>> Jacek
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> “One of the main causes of the fall of the Roman Empire was that–lacking
> zero–they had no way to indicate successful termination of their C
> programs.”
> (Robert Firth)
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

what does future do after fn finish ?

[Jacek Grzebyta]

Hi,

I have application with quite intense tripe store populating ~30/40 k
records per chunk (139 portions). The data are wrapped within the future:

(conj agent (future (apply task args)))

 and that all together is send-off into (agent []).
At the end of the main thread function I just use await-for and after that:

(reduce + (map #(deref %) @data-loading-tasks))

For some reason I see the happy collecting (see attached screenshot of
jconsole). After seeing the source code of future I suppose that the memory
(data are kept as #{} set) is not released. The task returns only integer
so I do not think that might cause the problem.

Thanks a lot,
Jacek

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what does future do after fn finish ?

[Jacek Grzebyta]

On 31 January 2018 at 18:08, James Reeves  wrote:

> On 31 January 2018 at 17:59, Jacek Grzebyta 
> wrote:
>
>> I have application with quite intense tripe store populating ~30/40 k
>> records per chunk (139 portions). The data are wrapped within the future:
>>
>> (conj agent (future (apply task args)))
>>
>>  and that all together is send-off into (agent []).
>>
>
> What is "agent"? The first line of code indicates that it's a local
> collection shadowing the code function, while the second code snippet
> indicates that you're using the core agent function.
>
> Also why are you sending off to an agent?
>

I have ~8sec computing task for each input dataset which generates those
records. After that I write it into disk (in software-specific
transaction). I just wanted to separate hard computing and io operations. I
created a side-effect method which is injected together with the dataset
into a future. The futures are async collected within a list wrapped in
agent. After the computing the main thread is waiting until all io tasks
will be finished.


>
> At the end of the main thread function I just use await-for and after that:
>>
>> (reduce + (map #(deref %) @data-loading-tasks))
>>
>
As a control, tasks return number of written records.



>
>> For some reason I see the happy collecting (see attached screenshot of
>> jconsole).
>>
>
> "happy" = "heap"?
>

Both. As you can see on attached screenshot the heap usage grows easy until
aver. ~2 1/4 G than keep that  for a few minutes. In that moment I stopped.
After that starts grow till ~4G with tendency to do jumps a bit more that
4G.


>
>
>> After seeing the source code of future I suppose that the memory (data
>> are kept as #{} set) is not released. The task returns only integer so I do
>> not think that might cause the problem.
>>
>
> Can you provide more detail? You keep alluding to things that you don't
> provide code for, such as the sets of data.
>


The code is attached. However the important code is

L123 .
  (let [;; keeps all data loading futures.
;; waiting until all futures are finished
;; should be done outside the main loop
data-loading-tasks (agent [])]

L128
(doseq
 (let [r1 (long operation)]   L133
 (doseq
(let [r2 (v.v. long)]   L155

  L163   (send-off data-loading-task conj-task)

 )
 )
)
)


I guess first I will move data-loading-tasks list into one of inner lets.
Also I will create within an injecting function a separate abstract
function let inside. The task will populate tmp variable which will be
returned as a future result:


L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
result)


>
> --
> James Reeves
> booleanknot.com
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


experiment_result.clj
Description: Binary data

Re: what does future do after fn finish ?

[Jacek Grzebyta]

Thanks a lot. I will check it tomorrow.

J

On 1 Feb 2018 12:12 a.m., "Justin Smith"  wrote:

> this is exactly the kind of problem code I was describing - there's no
> backpressure on existing future tasks to hold up the launching of more
> futures - the work done by the agent calling conj is negligible. You need
> to control the size of the pool of threads used, and you need to impose
> back-pressure.
>
> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta 
> wrote:
>
>> On 31 January 2018 at 18:08, James Reeves  wrote:
>>
>>> On 31 January 2018 at 17:59, Jacek Grzebyta 
>>> wrote:
>>>
>>>> I have application with quite intense tripe store populating ~30/40 k
>>>> records per chunk (139 portions). The data are wrapped within the future:
>>>>
>>>> (conj agent (future (apply task args)))
>>>>
>>>>  and that all together is send-off into (agent []).
>>>>
>>>
>>> What is "agent"? The first line of code indicates that it's a local
>>> collection shadowing the code function, while the second code snippet
>>> indicates that you're using the core agent function.
>>>
>>> Also why are you sending off to an agent?
>>>
>>
>> I have ~8sec computing task for each input dataset which generates those
>> records. After that I write it into disk (in software-specific
>> transaction). I just wanted to separate hard computing and io operations. I
>> created a side-effect method which is injected together with the dataset
>> into a future. The futures are async collected within a list wrapped in
>> agent. After the computing the main thread is waiting until all io tasks
>> will be finished.
>>
>>
>>>
>>> At the end of the main thread function I just use await-for and after
>>>> that:
>>>>
>>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>>
>>>
>> As a control, tasks return number of written records.
>>
>>
>>
>>>
>>>> For some reason I see the happy collecting (see attached screenshot of
>>>> jconsole).
>>>>
>>>
>>> "happy" = "heap"?
>>>
>>
>> Both. As you can see on attached screenshot the heap usage grows easy
>> until aver. ~2 1/4 G than keep that  for a few minutes. In that moment I
>> stopped. After that starts grow till ~4G with tendency to do jumps a bit
>> more that 4G.
>>
>>
>>>
>>>
>>>> After seeing the source code of future I suppose that the memory (data
>>>> are kept as #{} set) is not released. The task returns only integer so I do
>>>> not think that might cause the problem.
>>>>
>>>
>>> Can you provide more detail? You keep alluding to things that you don't
>>> provide code for, such as the sets of data.
>>>
>>
>>
>> The code is attached. However the important code is
>>
>> L123 .
>>   (let [;; keeps all data loading futures.
>> ;; waiting until all futures are finished
>> ;; should be done outside the main loop
>> data-loading-tasks (agent [])]
>>
>> L128
>> (doseq
>>  (let [r1 (long operation)]   L133
>>  (doseq
>> (let [r2 (v.v. long)]   L155
>>
>>   L163   (send-off data-loading-task conj-task)
>>
>>  )
>>  )
>> )
>> )
>>
>>
>> I guess first I will move data-loading-tasks list into one of inner lets.
>> Also I will create within an injecting function a separate abstract
>> function let inside. The task will populate tmp variable which will be
>> returned as a future result:
>>
>>
>> L114 (conj agent (future (apply (fn [] (let [result (apply task args)]
>> result)
>>
>>
>>>
>>> --
>>> James Reeves
>>> booleanknot.com
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Clojure" group.
>>> To post to this group, send email to clojure@googlegroups.com
>>> Note that posts from new members are moderated - please be patient with
>>> your first post.
>>> To unsubscribe from this group, send email to
>>> clojure+unsubscr...@googlegroups.com
>>> For more options, visit this group at
>>> http://groups.google.com/group/clojure?hl=en
>>> ---
>>> You received this message because you are subsc

Re: what does future do after fn finish ?

[Jacek Grzebyta]

Thanks folks. I see now! It should be a list of agents not list of futures
within agent.  Also any task sent to a agent is processed within a
thread anyway so I do not need to add future...

On 1 February 2018 at 02:17, John Newman  wrote:

> Ah, he's using one agent, I see.
>
> On Jan 31, 2018 9:15 PM, "John Newman"  wrote:
>
>> Multiple sen-doffs to one agent will serialize it's calls, but spawning
>> agents on each new task will spawn threads on a bounded thread pool, I
>> believe.
>>
>> On Jan 31, 2018 8:32 PM, "Justin Smith"  wrote:
>>
>>> Doing all the actions via one agent means that the actions are
>>> serialized though - you end up with no performance improvement over doing
>>> them all in a doseq in one future - the right way to do this tends to be
>>> trickier than it looks at first glance, and depends on your requirements.
>>> agents, the claypoole library, and reducers are all potentially useful. If
>>> parallelization leads to complex coordination needs, core.async can help
>>> too.
>>>
>>> On Wed, Jan 31, 2018 at 5:18 PM John Newman  wrote:
>>>
>>>> Agents manage a pool of threads for you. Try doing it without the
>>>> future call and see if that works (unless you're trying to do something
>>>> else).
>>>>
>>>> John
>>>>
>>>> On Wed, Jan 31, 2018 at 7:31 PM, Jacek Grzebyta >>> > wrote:
>>>>
>>>>> Thanks a lot. I will check it tomorrow.
>>>>>
>>>>> J
>>>>>
>>>>> On 1 Feb 2018 12:12 a.m., "Justin Smith"  wrote:
>>>>>
>>>>>> this is exactly the kind of problem code I was describing - there's
>>>>>> no backpressure on existing future tasks to hold up the launching of more
>>>>>> futures - the work done by the agent calling conj is negligible. You need
>>>>>> to control the size of the pool of threads used, and you need to impose
>>>>>> back-pressure.
>>>>>>
>>>>>> On Wed, Jan 31, 2018 at 3:46 PM Jacek Grzebyta <
>>>>>> grzebyta@gmail.com> wrote:
>>>>>>
>>>>>>> On 31 January 2018 at 18:08, James Reeves 
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On 31 January 2018 at 17:59, Jacek Grzebyta >>>>>>> > wrote:
>>>>>>>>
>>>>>>>>> I have application with quite intense tripe store populating
>>>>>>>>> ~30/40 k records per chunk (139 portions). The data are wrapped 
>>>>>>>>> within the
>>>>>>>>> future:
>>>>>>>>>
>>>>>>>>> (conj agent (future (apply task args)))
>>>>>>>>>
>>>>>>>>>  and that all together is send-off into (agent []).
>>>>>>>>>
>>>>>>>>
>>>>>>>> What is "agent"? The first line of code indicates that it's a local
>>>>>>>> collection shadowing the code function, while the second code snippet
>>>>>>>> indicates that you're using the core agent function.
>>>>>>>>
>>>>>>>> Also why are you sending off to an agent?
>>>>>>>>
>>>>>>>
>>>>>>> I have ~8sec computing task for each input dataset which generates
>>>>>>> those records. After that I write it into disk (in software-specific
>>>>>>> transaction). I just wanted to separate hard computing and io 
>>>>>>> operations. I
>>>>>>> created a side-effect method which is injected together with the dataset
>>>>>>> into a future. The futures are async collected within a list wrapped in
>>>>>>> agent. After the computing the main thread is waiting until all io tasks
>>>>>>> will be finished.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> At the end of the main thread function I just use await-for and
>>>>>>>>> after that:
>>>>>>>>>
>>>>>>>>> (reduce + (map #(deref %) @data-loading-tasks))
>>>>>>>>>
>>>>

Re: what does future do after fn finish ?

[Jacek Grzebyta]

On 2 February 2018 at 08:34, Niels van Klaveren  wrote:

> +1 for Claypoole, it removed the needs of using agents or futures in 95%
> of the cases in my code.
>
>
Thanks a lot. I modify the code using claypoole. I imagine with-shutdown
will close the pool properly after finish all tasks so there is no need to
watch them?

J

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what does future do after fn finish ?

[Jacek Grzebyta]

OK I found what makes the memory leak.

In the project I work with I use a  java Model class which is java
Collection proxy/facade for a single transaction. Unfortunately it's not
thread safe. In a few places I passed single instance of model into several
threads Also I requested the instance with -> which makes new thread as
well. I was surprised that -> makes trouble but after thinking that might
be expected. Especially that internally the wrapper doesn't do simple
mapping - it uses some iterator, etc. Anyway the machinery is fragile. It
seems I need to rewrite code and replace all multithreading parts by
something simpler.



If you want see the stacktrace just look at:
https://github.com/jgrzebyta/triple-loader/issues/53

On 2 February 2018 at 11:16, Jacek Grzebyta  wrote:

>
> On 2 February 2018 at 08:34, Niels van Klaveren <
> niels.vanklave...@gmail.com> wrote:
>
>> +1 for Claypoole, it removed the needs of using agents or futures in 95%
>> of the cases in my code.
>>
>>
> Thanks a lot. I modify the code using claypoole. I imagine with-shutdown
> will close the pool properly after finish all tasks so there is no need to
> watch them?
>
> J
>
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: what does future do after fn finish ?

[Jacek Grzebyta]

HI,

So things works well. In terms of memory usage anyway. I use 50 main
threads (in process-data-by-condition-set) and 20 for replications
(process-growths). However I found that after good performance in the
beginning it slows down rapid and quick. VisualVM shows the threads usage
is poor - mainly they are waiting. Only 1 claypoole thread works + some
lower level threads managed probably by Clojure, etc. CPU usage is poor as
well. I guess the reason is on the lower - Java classes level with IO
operations bottleneck. IMO they are working by blocking. That why I wanted
to separate them. However IO speed operations go down as well.
I've tried async channels but that they were flooded and some data where
lost for some reason.

Regards,
Jacek


On 3 February 2018 at 00:14, Justin Smith  wrote:

> -> is just a list transform performed after reading your code into a list
> data structure containing symbols, and before compiling to byte code - it
> doesn't do anything directly.
>
> On Fri, Feb 2, 2018 at 3:55 PM Jacek Grzebyta 
> wrote:
>
>> OK I found what makes the memory leak.
>>
>> In the project I work with I use a  java Model class which is java
>> Collection proxy/facade for a single transaction. Unfortunately it's not
>> thread safe. In a few places I passed single instance of model into several
>> threads Also I requested the instance with -> which makes new thread as
>> well. I was surprised that -> makes trouble but after thinking that might
>> be expected. Especially that internally the wrapper doesn't do simple
>> mapping - it uses some iterator, etc. Anyway the machinery is fragile. It
>> seems I need to rewrite code and replace all multithreading parts by
>> something simpler.
>>
>>
>>
>> If you want see the stacktrace just look at:
>> https://github.com/jgrzebyta/triple-loader/issues/53
>>
>> On 2 February 2018 at 11:16, Jacek Grzebyta 
>> wrote:
>>
>>>
>>> On 2 February 2018 at 08:34, Niels van Klaveren <
>>> niels.vanklave...@gmail.com> wrote:
>>>
>>>> +1 for Claypoole, it removed the needs of using agents or futures in
>>>> 95% of the cases in my code.
>>>>
>>>>
>>> Thanks a lot. I modify the code using claypoole. I imagine with-shutdown
>>> will close the pool properly after finish all tasks so there is no need to
>>> watch them?
>>>
>>> J
>>>
>>>
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clojure@googlegroups.com
>> Note that posts from new members are moderated - please be patient with
>> your first post.
>> To unsubscribe from this group, send email to
>> clojure+unsubscr...@googlegroups.com
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+unsubscr...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your 
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
--- 
You received this message because you are subscribed to the Google Groups 
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.