Hi all,
I have written a Python script to analyze Minecraft levels and render a graph.
Then I did the same with Clojure. It takes Python 10 seconds to analyze a map,
while it takes Clojure over a minute.
After having tried different options without any significant improvement, I am
lost as to
On Thu, 4 Nov 2010 22:28:12 +0100
Pepijn de Vos wrote:
> Hi all,
>
> I have written a Python script to analyze Minecraft levels and render a
> graph. Then I did the same with Clojure. It takes Python 10 seconds to
> analyze a map, while it takes Clojure over a minute.
>
> After having tried d
On Thu, Nov 4, 2010 at 5:43 PM, Mike Meyer
wrote:
> On Thu, 4 Nov 2010 22:28:12 +0100
> Pepijn de Vos wrote:
>
>> Hi all,
>>
>> I have written a Python script to analyze Minecraft levels and render a
>> graph. Then I did the same with Clojure. It takes Python 10 seconds to
>> analyze a map, whi
I don't know how to check the GC activity on my project, but I did run
Mian on Jython. It performs much like my initial Clojure version. It
consumes absurd amounts of memory and never finishes.
So I think we can safely say that Java's GC or the way it stores data
is less efficient on this type of
Can you recommend any? I tied a few of the GC options, but that didn't
help much.
On Nov 4, 10:52 pm, Andrew Gwozdziewycz wrote:
> On Thu, Nov 4, 2010 at 5:43 PM, Mike Meyer
>
>
>
>
>
>
>
>
>
> wrote:
> > On Thu, 4 Nov 2010 22:28:12 +0100
> > Pepijn de Vos wrote:
>
> >> Hi all,
>
> >> I have wr
On Fri, Nov 5, 2010 at 10:41 AM, pepijn (aka fliebel) wrote:
> I don't know how to check the GC activity on my project, but I did run
> Mian on Jython. It performs much like my initial Clojure version. It
> consumes absurd amounts of memory and never finishes.
>
> So I think we can safely say tha
I will have a look around.
I listed the map I used in my first email, It's on my Dropbox:
http://dl.dropbox.com/u/10094764/World2.zip
Meanwhile I wrote a function that is already twice as fast as I had,
no memory problems, no threads. One tinny problem: it doesn't produce
the same result.
It's t
On Fri, Nov 5, 2010 at 17:38, pepijn (aka fliebel)
wrote:
> I will have a look around.
>
> I listed the map I used in my first email, It's on my Dropbox:
> http://dl.dropbox.com/u/10094764/World2.zip
>
> Meanwhile I wrote a function that is already twice as fast as I had,
> no memory problems, no
I'm very curios about this situation, please let us know if you manage to write
a version that's faster than the python one (as David claims is possible). I
would attempt it myself but I've only just recently had the time to dive back
into Clojure. :-\
- Greg
On Nov 5, 2010, at 9:38 AM, pepijn
On Fri, Nov 5, 2010 at 2:31 PM, Greg wrote:
> I'm very curios about this situation, please let us know if you manage to
> write a version that's faster than the python one (as David claims is
> possible). I would attempt it myself but I've only just recently had the
> time to dive back into Cloju
update! is of my own making, based on assoc! and update-in
On Nov 5, 7:30 pm, B Smith-Mannschott wrote:
> On Fri, Nov 5, 2010 at 17:38, pepijn (aka fliebel)
>
>
>
>
>
>
>
>
>
> wrote:
> > I will have a look around.
>
> > I listed the map I used in my first email, It's on my Dropbox:
> >http://dl
Could you refer me to some of those relevant to my problem? I tried
searching for them, and most stuff I found is about killing
reflection, using buffered IO and other basics I've already covered.
On Nov 5, 7:37 pm, David Nolen wrote:
> On Fri, Nov 5, 2010 at 2:31 PM, Greg wrote:
> > I'm very cu
You can use Visual VM (https://visualvm.dev.java.net/) to see how the
VM is using memory. I don't think it specifically show a log of GC
activity, but it is pretty clear from the graphs.
mch
On Nov 5, 8:41 am, "pepijn (aka fliebel)"
wrote:
> I don't know how to check the GC activity on my proje
I think you missed his point. (assoc! m k v) is *allowed* to modify m,
not *guaranteed*. It returns a pointer to a transient map, which may
be m, or may be a totally distinct map, or may be a new map that
shares some pointers with m. So your (do (update! blah foo
bar) ...more stuff) is potentially
Here's what I have so far. The code splits blocks into 128 smaller
sub-arrays, each representing a level, then calls a modified version
of frequencies (using areduce instead of reduce) on each level. On my
machine, with server mode on, it takes about 20 seconds to compute the
frequencies for an a
Oops, sorry, got my terminology wrong. The sub-arrays represent
*layers*, not levels. So the code should actually read as follows:
(def num-layers 128)
(defn get-layer [layer-num ^bytes blocks]
(let [size (/ (count blocks) num-layers)
output (byte-array size)]
(doseq [output-idx (
Awesome. You managed to reproduce my initial solution, but working
with arrays all the way. It is already quite a bit faster than what I
have, but still nowhere near Python.
I'll put it on the list for things to look at.
On Nov 6, 1:27 am, Benny Tsai wrote:
> Oops, sorry, got my terminology wron
> You can use Visual VM (https://visualvm.dev.java.net/) to see how the
> VM is using memory. I don't think it specifically show a log of GC
> activity, but it is pretty clear from the graphs.
Or just use -XX:+PrintGC and maybe -XX:+PrintGCDetails and
-XX:+PrintGCTimeStamps.
I haven't checked wh
[GC 524288K->103751K(2009792K), 0.1105872 secs]
[GC 628039K->105751K(2009792K), 0.0925628 secs]
[GC 630039K->109023K(2009792K), 0.0702017 secs]
[GC 633311K->115263K(2009792K), 0.0766341 secs]
[GC 639551K->117723K(2009792K), 0.0731049 secs]
[GC 642011K->120195K(1980096K), 0.0755788 secs]
[GC 614787K
On 2010-11-06, at 9:08 AM, pepijn (aka fliebel) wrote:
> I increased the heap space a lot, but I'm just bordering on the edge
> of my real memory, so it's not helping much.
Did you try pushing the minimum heap space up. I'm usually lazy and set them to
the same. I've had serious trouble caused
> For 30 second worth of calculations, this doesn't look to bad to me.
If that was for all of the 30 seconds then yeah, GC is not the issue.
--
/ Peter Schuller
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to cl
>> I increased the heap space a lot, but I'm just bordering on the edge
>> of my real memory, so it's not helping much.
>
> Did you try pushing the minimum heap space up. I'm usually lazy and set them
> to the same. I've had serious trouble caused by the way the JVM increases the
> heap space. Se
On 2010-11-06, at 10:22 AM, Peter Schuller wrote:
>>> I increased the heap space a lot, but I'm just bordering on the edge
>>> of my real memory, so it's not helping much.
>>
>> Did you try pushing the minimum heap space up. I'm usually lazy and set them
>> to the same. I've had serious trouble
While grocery shopping this morning, it occurred to me that it would
be even faster to do a single pass over the blocks array, and update
the count in one of 128 maps depending on the current index. Of
course, when I got home and took another look at your gist page, it
turns out that's you've alre
Implementing this in straight Java might help pinpoint whether this is
a JVM issue or a Clojure issue.
Also, FYI, there is clj-glob (https://github.com/jkk/clj-glob) for
finding files based on patterns like */*/*.dat
Justin
On Nov 4, 4:28 pm, Pepijn de Vos wrote:
> Hi all,
>
> I have written a
The problem is not the freqs function. The implementation in Python
and in Clojure are different algorithms.
Both produce a graph with the number of blocks per layer per type, for
6 particular types. These six types are the Clay, Coal ore, Diamond
ore, Gold ore, Iron ore, Obsidian, and Redstone or
Great catch, Gijs!
On Nov 7, 9:32 am, "Gijs S." wrote:
> The problem is not the freqs function. The implementation in Python
> and in Clojure are different algorithms.
>
> Both produce a graph with the number of blocks per layer per type, for
> 6 particular types. These six types are the Clay, Co
Gijs,
I would put the return value of the assoc! call as one of the values
of the loop.
Otherwise, as far as I understand, you may lose values associated with
the fl transient map.
(Transient maps should not be used any differently than regular maps.
It's just their internals that are different.)
Indeed I was not using the transients correctly. Transients are not
designed to be bashed in-place. (http://clojure.org/transients)
The gist is updated with a version that first worked without the
transients and then had the transient operations added. (https://
gist.github.com/666228)
The times
I played with this some more, and it seems like for accessing an
element in a vector, (v idx) is a bit faster than (nth v idx). If you
replace this line (line 52 in the gist)...
fl (nth freq-layer layer)]
with:
fl (freq-layer layer)]
...the total time needed to compute frequencies for all 92 t
I also get consistent better times with (freq-layer layer) vs. (nth
freq-layer layer).
Gist updated: https://gist.github.com/666228
-Gijs
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
N
Hi,
On 9 Nov., 00:31, "Gijs S." wrote:
> I also get consistent better times with (freq-layer layer) vs. (nth
> freq-layer layer).
This is to be expected, because (freq-layer layer) acts directly on
the datastructure, while (nth freq-layer layer) goes through another
function.
Sincerely
Meikel
I almost forgot about this. I talked to mfex on IRC, and he came up
with the winning solution.
http://clojure-log.n01se.net/date/2010-11-07.html#11:32
https://gist.github.com/666228
Basically it comes down to *not* doing work, rather than doing it
fast.
What the Python version does - and this ve
ouch didn't noticed the second page.
On Nov 10, 2:06 pm, "pepijn (aka fliebel)"
wrote:
> I almost forgot about this. I talked to mfex on IRC, and he came up
> with the winning
> solution.http://clojure-log.n01se.net/date/2010-11-07.html#11:32
>
> https://gist.github.com/666228
>
> Basically it c
I am reminded of an arcane implementation...
http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html
On Wed, Nov 10, 2010 at 8:06 AM, pepijn (aka fliebel)
wrote:
> I almost forgot about this. I talked to mfex on IRC, and he came up
> with the winning solution.
> http://clojure-
Me to: http://clojure-log.n01se.net/date/2010-11-07.html#11:57
On Nov 10, 5:59 pm, Leif Walsh wrote:
> I am reminded of an arcane
> implementation...http://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310...
>
> On Wed, Nov 10, 2010 at 8:06 AM, pepijn (aka fliebel)
>
>
>
>
>
>
>
>
36 matches
Mail list logo
