On 14/10/12 00:17, Ben Mabey wrote:
I switched from pmap to
(r/fold n (r/monoid into vector) conj coll))
and the same thing happened again!
after approximately 50 minutes cpu utilisation dropped from 4/4 to
1/4...I don't understand!
Jim
Are you holding on to the head of the collection?
I switched from pmap to
(r/fold n (r/monoid into vector) conj coll))
and the same thing happened again!
after approximately 50 minutes cpu utilisation dropped from 4/4 to
1/4...I don't understand!
Jim
Are you holding on to the head of the collection? That is often the
source of memory
If it is a memory / gc problem, the easiest way to spot it is writing a gc
log.
Add the following parameters to your JVM parameters:
-Xloggc:D:/log/myGClog.log
-XX:+PrintGCDetails
Then, use a GC log viewer to keep a look on the logs (by refreshing or tail
refresh) with a GC log viewer. These
This feels like a memory leak problem. Try adding
-XX:-HeapDumpOnOutOfMemoryError. You might try lowering the max heap to
try and force an OOM earlier. Heap dumps can be analyzed using a variety
of tools. My favorite is eclipse.org/mat
On Saturday, October 13, 2012 8:16:54 AM UTC-7, Jim foo
On 13/10/12 12:55, Jim - FooBar(); wrote:
On 13/10/12 12:50, Tassilo Horn wrote:
pmap might not be as perfect for you use-case as you think. Because it
is lazy and chunked, it won't keep you cores busy. Some time ago, there
was a detailed thread about this on this list.
But it does keep my c
On 13/10/12 12:50, Tassilo Horn wrote:
pmap might not be as perfect for you use-case as you think. Because it
is lazy and chunked, it won't keep you cores busy. Some time ago, there
was a detailed thread about this on this list.
But it does keep my cores fully busy (99-100%) for the first 40
"Jim foo.bar" writes:
> So you can see, this is perfect for pmap and indeed it seems to be
> doing extremely well but only for the first 240 papers roughly! all
> the cpus are working hard but after approximately 30-40 min cpu
> utilisation and overall performance seems to degrade quite a bit...F
If you have the memory you could just increase the heap size to a higher
value (like -Xmx2048m or something). But even if you do that I would still
run jconsole to see what's happening.
~Adam~
On Fri, Oct 12, 2012 at 9:41 AM, Jim foo.bar wrote:
> No i haven't profiled memory , only cpu but w
No i haven't profiled memory , only cpu but what you're saying makes
perfect sense. In every single iteration (each file) I'm 'slurp'-ing the
2 files (the dictionary and the file to annotate) provided. Would you
suggest a different GC if that is the case or simply stop slurping?
Jim
On 12/1
Have you tried running jconsole to monitor the memory usage? It sounds
like maybe you're running out of heap space and you're mainly seeing the
garbage collector doing it's thing vs your actual program.
~Adam~
On Fri, Oct 12, 2012 at 9:23 AM, Jim foo.bar wrote:
> Hi all,
>
> I finally found a
Hi all,
I finally found an ideal use-case for pmap, however something very
strange seems to be happening after roughly 30 minutes of execution!
Ok so here is the scenario:
I've got 383 raw scienific papers (.txt) in directory that i'm grouping
using 'file-seq' and so I want to pmap a fn on e
11 matches
Mail list logo