Re: finding retained head
or perhaps it's that macroexpand drops the ^:once On Thursday, September 12, 2013 10:55:28 AM UTC-7, Brian Craft wrote: I think my monkey-patch of jdbc/transaction didn't take. Doing user/sourcery on it shows your version, but doing a macroexpand-all shows the original. My code calls via korma as kdb: = (clojure.walk/macroexpand-all '(kdb/transaction nil)) (if (clojure.java.jdbc/find-connection) (clojure.java.jdbc/transaction* (fn* [] nil)) (clojure.java.jdbc/with-connection* (korma.db/get-connection (clojure.core/deref korma.db/_default)) (fn* ([] (clojure.java.jdbc/transaction* (fn* [] nil)) I thought I could get it to take by doing a (require 'clojure.java.jdbc) and applying the patch before doing a require on korma, but that doesn't seem to help. (require 'clojure.java.jdbc) ; XXX monkey-patch jdbc to avoid head retention bug (in-ns 'clojure.java.jdbc) (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) (ns blahblah (:require [korma.db as kdb])) .. and restart the repl, but macroexpand-all still gives me the result above. On Wednesday, September 11, 2013 1:45:41 PM UTC-7, Christophe Grand wrote: I don't get the same results: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61221 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) OutOfMemoryError GC overhead limit exceeded java.lang.Long.valueOf (Long.java:577) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) nil user= (t2 0 (range 100)) OutOfMemoryError GC overhead limit exceeded clojure.lang.ChunkBuffer.chunk (ChunkBuffer.java:29) BUT this is because the previous OOM left the JVM in a dirty state: try to reorder your expressions: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61245 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (t2 0 (range 100)) nil ^^this last one failed in the previous run. I'm not quite sure about why the doseq version works -- I would have to research a bit. My gut feeling is that doseq is based on loop and loops are lifted into ^:once fn* by the compiler in some cases. https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L5951 On Wed, Sep 11, 2013 at 8:20 PM, Brian Craft craft...@gmail.com wrote: Correct, I forgot to paste that part. ;) On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote: Just to confirm, (t2 0 (range 100)) -- using doseq instead of dorun -- does NOT run out of memory, correct? On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft craft...@gmail.com wrote: This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft...@gmail.com wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.com wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure
Re: finding retained head
After patching both transaction and with-connection (which is also used by korma.db/transaction, and also creates a closure), I can pass in the seq w/o leaking. Thanks, Christophe. On Thursday, September 12, 2013 11:20:11 AM UTC-7, Brian Craft wrote: or perhaps it's that macroexpand drops the ^:once On Thursday, September 12, 2013 10:55:28 AM UTC-7, Brian Craft wrote: I think my monkey-patch of jdbc/transaction didn't take. Doing user/sourcery on it shows your version, but doing a macroexpand-all shows the original. My code calls via korma as kdb: = (clojure.walk/macroexpand-all '(kdb/transaction nil)) (if (clojure.java.jdbc/find-connection) (clojure.java.jdbc/transaction* (fn* [] nil)) (clojure.java.jdbc/with-connection* (korma.db/get-connection (clojure.core/deref korma.db/_default)) (fn* ([] (clojure.java.jdbc/transaction* (fn* [] nil)) I thought I could get it to take by doing a (require 'clojure.java.jdbc) and applying the patch before doing a require on korma, but that doesn't seem to help. (require 'clojure.java.jdbc) ; XXX monkey-patch jdbc to avoid head retention bug (in-ns 'clojure.java.jdbc) (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) (ns blahblah (:require [korma.db as kdb])) .. and restart the repl, but macroexpand-all still gives me the result above. On Wednesday, September 11, 2013 1:45:41 PM UTC-7, Christophe Grand wrote: I don't get the same results: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61221 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) OutOfMemoryError GC overhead limit exceeded java.lang.Long.valueOf (Long.java:577) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) nil user= (t2 0 (range 100)) OutOfMemoryError GC overhead limit exceeded clojure.lang.ChunkBuffer.chunk (ChunkBuffer.java:29) BUT this is because the previous OOM left the JVM in a dirty state: try to reorder your expressions: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61245 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (t2 0 (range 100)) nil ^^this last one failed in the previous run. I'm not quite sure about why the doseq version works -- I would have to research a bit. My gut feeling is that doseq is based on loop and loops are lifted into ^:once fn* by the compiler in some cases. https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L5951 On Wed, Sep 11, 2013 at 8:20 PM, Brian Craft craft...@gmail.com wrote: Correct, I forgot to paste that part. ;) On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote: Just to confirm, (t2 0 (range 100)) -- using doseq instead of dorun -- does NOT run out of memory, correct? On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft craft...@gmail.com wrote: This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft...@gmail.com wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.com wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created
Re: finding retained head
The latest java.jdbc snapshot (of 0.3.0) includes this fix so you can either get it via Leiningen from the sonatype snapshots repo or git clone it and do mvn install (or lein install since java.jdbc has a project.clj file now). I'll try to cut an official alpha5 release shortly once I've reviewed the open tickets and updated the changes doc etc. Sean On Thu, Sep 12, 2013 at 11:37 AM, Brian Craft craft.br...@gmail.com wrote: After patching both transaction and with-connection (which is also used by korma.db/transaction, and also creates a closure), I can pass in the seq w/o leaking. Thanks, Christophe. On Thursday, September 12, 2013 11:20:11 AM UTC-7, Brian Craft wrote: or perhaps it's that macroexpand drops the ^:once On Thursday, September 12, 2013 10:55:28 AM UTC-7, Brian Craft wrote: I think my monkey-patch of jdbc/transaction didn't take. Doing user/sourcery on it shows your version, but doing a macroexpand-all shows the original. My code calls via korma as kdb: = (clojure.walk/macroexpand-all '(kdb/transaction nil)) (if (clojure.java.jdbc/find-connection) (clojure.java.jdbc/transaction* (fn* [] nil)) (clojure.java.jdbc/with-connection* (korma.db/get-connection (clojure.core/deref korma.db/_default)) (fn* ([] (clojure.java.jdbc/transaction* (fn* [] nil)) I thought I could get it to take by doing a (require 'clojure.java.jdbc) and applying the patch before doing a require on korma, but that doesn't seem to help. (require 'clojure.java.jdbc) ; XXX monkey-patch jdbc to avoid head retention bug (in-ns 'clojure.java.jdbc) (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) (ns blahblah (:require [korma.db as kdb])) .. and restart the repl, but macroexpand-all still gives me the result above. On Wednesday, September 11, 2013 1:45:41 PM UTC-7, Christophe Grand wrote: I don't get the same results: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61221 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) OutOfMemoryError GC overhead limit exceeded java.lang.Long.valueOf (Long.java:577) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) nil user= (t2 0 (range 100)) OutOfMemoryError GC overhead limit exceeded clojure.lang.ChunkBuffer.chunk (ChunkBuffer.java:29) BUT this is because the previous OOM left the JVM in a dirty state: try to reorder your expressions: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61245 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (t2 0 (range 100)) nil ^^this last one failed in the previous run. I'm not quite sure about why the doseq version works -- I would have to research a bit. My gut feeling is that doseq is based on loop and loops are lifted into ^:once fn* by the compiler in some cases. https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L5951 On Wed, Sep 11, 2013 at 8:20 PM, Brian Craft craft...@gmail.com wrote: Correct, I forgot to paste that part. ;) On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote: Just to confirm, (t2 0 (range 100)) -- using doseq instead of dorun -- does NOT run out of memory, correct? On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft craft...@gmail.com wrote: This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft
Re: finding retained head
I think my monkey-patch of jdbc/transaction didn't take. Doing user/sourcery on it shows your version, but doing a macroexpand-all shows the original. My code calls via korma as kdb: = (clojure.walk/macroexpand-all '(kdb/transaction nil)) (if (clojure.java.jdbc/find-connection) (clojure.java.jdbc/transaction* (fn* [] nil)) (clojure.java.jdbc/with-connection* (korma.db/get-connection (clojure.core/deref korma.db/_default)) (fn* ([] (clojure.java.jdbc/transaction* (fn* [] nil)) I thought I could get it to take by doing a (require 'clojure.java.jdbc) and applying the patch before doing a require on korma, but that doesn't seem to help. (require 'clojure.java.jdbc) ; XXX monkey-patch jdbc to avoid head retention bug (in-ns 'clojure.java.jdbc) (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) (ns blahblah (:require [korma.db as kdb])) .. and restart the repl, but macroexpand-all still gives me the result above. On Wednesday, September 11, 2013 1:45:41 PM UTC-7, Christophe Grand wrote: I don't get the same results: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61221 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) OutOfMemoryError GC overhead limit exceeded java.lang.Long.valueOf (Long.java:577) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) nil user= (t2 0 (range 100)) OutOfMemoryError GC overhead limit exceeded clojure.lang.ChunkBuffer.chunk (ChunkBuffer.java:29) BUT this is because the previous OOM left the JVM in a dirty state: try to reorder your expressions: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61245 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (t2 0 (range 100)) nil ^^this last one failed in the previous run. I'm not quite sure about why the doseq version works -- I would have to research a bit. My gut feeling is that doseq is based on loop and loops are lifted into ^:once fn* by the compiler in some cases. https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L5951 On Wed, Sep 11, 2013 at 8:20 PM, Brian Craft craft...@gmail.comjavascript: wrote: Correct, I forgot to paste that part. ;) On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote: Just to confirm, (t2 0 (range 100)) -- using doseq instead of dorun -- does NOT run out of memory, correct? On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft craft...@gmail.com wrote: This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft...@gmail.com wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.com wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ --
Re: finding retained head
jvisualvm has an innocuous button called Dump Memory or something. You'd expect it to write out a core dump or something, but actually it opens up a GUI which lets you interactively explore all of the objects on the heap. It is pretty amazing. Much better than jhat, which I've found to be really flakey. Good for finding Classloader leaks too, or just generally finding where all your memory has gone via the Compute Retained Sizes option. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft.br...@gmail.com wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.comjavascript: wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft.br...@gmail.com wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.com wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
(started lein with LEIN_JVM_OPTS=-Xmx10M lein repl, to make it easy to see a full heap) On Wednesday, September 11, 2013 8:59:50 AM UTC-7, Brian Craft wrote: This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft...@gmail.com wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.comwrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
I have nothing to add for the problem itself (sorry) but am very interested in the *process* of answering this question. Presuming there are things to document here, I would love to see someone create a wiki page (on http://dev.clojure.org/display/community/Home) or a clojure-doc note or some place to capture tips and techniques for debugging problems like this. Alex On Wednesday, September 11, 2013 10:59:50 AM UTC-5, Brian Craft wrote: This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft...@gmail.com wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.comwrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft...@gmail.comjavascript: wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.com wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
I did start with visualvm. I posted a screenshot in an earlier thread. However I'm unable to make sense of its output. jhat pointed straight to the closure with the reference. visualvm gave me a thousand cascading widgets to expand with names that were meaningless to me, none of which pointed back to the closure. I'm looking at it again now, trying to find another leak. The references for a char[] start at String and go back to cache (Java frame), with no symbol anywhere that I can relate to the program being run. I don't know what to do with this. The Strings are from another seq: lines from a file via line-seq. Unfortunately, jhat just hangs on this object, so I have no working instrumentation. On Wednesday, September 11, 2013 1:08:33 AM UTC-7, David Powell wrote: jvisualvm has an innocuous button called Dump Memory or something. You'd expect it to write out a core dump or something, but actually it opens up a GUI which lets you interactively explore all of the objects on the heap. It is pretty amazing. Much better than jhat, which I've found to be really flakey. Good for finding Classloader leaks too, or just generally finding where all your memory has gone via the Compute Retained Sizes option. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
Correct, I forgot to paste that part. ;) On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote: Just to confirm, (t2 0 (range 100)) -- using doseq instead of dorun -- does NOT run out of memory, correct? On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft craft...@gmail.comjavascript: wrote: This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft...@gmail.com wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.com wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clo...@googlegroups.comjavascript: Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@googlegroups.com javascript: For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com javascript:. For more options, visit https://groups.google.com/groups/opt_out. -- Sean A Corfield -- (904) 302-SEAN An Architect's View -- http://corfield.org/ World Singles, LLC. -- http://worldsingles.com/ Perfection is the enemy of the good. -- Gustave Flaubert, French realist novelist (1821-1880) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
Attaching a screenshot. Is this reference chain useful somehow? I can recognize the contents of the string from the input file. The input file is 4G, and apparently lines from line-seq are lingering, so I'm still blowing the heap. On Wednesday, September 11, 2013 11:20:18 AM UTC-7, Brian Craft wrote: I did start with visualvm. I posted a screenshot in an earlier thread. However I'm unable to make sense of its output. jhat pointed straight to the closure with the reference. visualvm gave me a thousand cascading widgets to expand with names that were meaningless to me, none of which pointed back to the closure. I'm looking at it again now, trying to find another leak. The references for a char[] start at String and go back to cache (Java frame), with no symbol anywhere that I can relate to the program being run. I don't know what to do with this. The Strings are from another seq: lines from a file via line-seq. Unfortunately, jhat just hangs on this object, so I have no working instrumentation. On Wednesday, September 11, 2013 1:08:33 AM UTC-7, David Powell wrote: jvisualvm has an innocuous button called Dump Memory or something. You'd expect it to write out a core dump or something, but actually it opens up a GUI which lets you interactively explore all of the objects on the heap. It is pretty amazing. Much better than jhat, which I've found to be really flakey. Good for finding Classloader leaks too, or just generally finding where all your memory has gone via the Compute Retained Sizes option. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. attachment: string references.png
Re: finding retained head
Just to confirm, (t2 0 (range 100)) -- using doseq instead of dorun -- does NOT run out of memory, correct? On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft craft.br...@gmail.com wrote: This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft...@gmail.com wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.com wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- Sean A Corfield -- (904) 302-SEAN An Architect's View -- http://corfield.org/ World Singles, LLC. -- http://worldsingles.com/ Perfection is the enemy of the good. -- Gustave Flaubert, French realist novelist (1821-1880) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
I don't get the same results: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61221 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) OutOfMemoryError GC overhead limit exceeded java.lang.Long.valueOf (Long.java:577) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) nil user= (t2 0 (range 100)) OutOfMemoryError GC overhead limit exceeded clojure.lang.ChunkBuffer.chunk (ChunkBuffer.java:29) BUT this is because the previous OOM left the JVM in a dirty state: try to reorder your expressions: $ LEIN_JVM_OPTS=-Xmx20M lein repl nREPL server started on port 61245 on host 127.0.0.1 REPL-y 0.2.1 Clojure 1.5.1 Docs: (doc function-name-here) (find-doc part-of-name-here) Source: (source function-name-here) Javadoc: (javadoc java-object-or-class-here) Exit: Control+D or (exit) or (quit) user= (defn f [g] (g)) #'user/f user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (t2 0 (range 100)) nil ^^this last one failed in the previous run. I'm not quite sure about why the doseq version works -- I would have to research a bit. My gut feeling is that doseq is based on loop and loops are lifted into ^:once fn* by the compiler in some cases. https://github.com/clojure/clojure/blob/master/src/jvm/clojure/lang/Compiler.java#L5951 On Wed, Sep 11, 2013 at 8:20 PM, Brian Craft craft.br...@gmail.com wrote: Correct, I forgot to paste that part. ;) On Wednesday, September 11, 2013 10:57:09 AM UTC-7, Sean Corfield wrote: Just to confirm, (t2 0 (range 100)) -- using doseq instead of dorun -- does NOT run out of memory, correct? On Wed, Sep 11, 2013 at 8:59 AM, Brian Craft craft...@gmail.com wrote: This appears to have no effect on the problem. I tested with jdbc/transaction, and with Sean's simple example: user= (defn f [g] (g)) #'user/f user= (defn t1 [n c] (f (fn [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) user= (defn t2 [n c] (f (fn [] (doseq [x c] (identity x) #'user/t2 user= (defn t1 [n c] (f (^:once fn* [] (dorun (map identity c) #'user/t1 user= (t1 0 (range 100)) java.lang.OutOfMemoryError: Java heap space (NO_SOURCE_FILE:0) On Wednesday, September 11, 2013 7:39:48 AM UTC-7, Christophe Grand wrote: ^:once on fn* (not fn, the fn macro doesn't propagate metadata) instructs the commielr to clear closed-overs ASAP. It follows that you can't call such a function twice because it forgets its closed-overs. On Wed, Sep 11, 2013 at 4:36 PM, Brian Craft craft...@gmail.com wrote: ugh. Can't find documentation for this. What does it do? On Wednesday, September 11, 2013 2:22:56 AM UTC-7, Christophe Grand wrote: On Wed, Sep 11, 2013 at 6:00 AM, Brian Craft craft...@gmail.com wrote: (defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. The fix for this is: (defmacro transaction [ body] `(transaction* (^:once fn* [] ~@body))) It should be the default for all one-shot fns. Christophe -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- On Clojure http://clj-me.cgrand.net/ Clojure Programming http://clojurebook.com Training, Consulting Contracting http://lambdanext.eu/ -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clo...@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+u...@**googlegroups.com For more options, visit this group at http://groups.google.com/**group/clojure?hl=enhttp://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@**googlegroups.com. For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out. -- Sean A Corfield -- (904) 302-SEAN An Architect's View -- http://corfield.org/ World Singles, LLC. -- http://worldsingles.com/ Perfection is the enemy of the good. -- Gustave Flaubert, French realist novelist
Re: finding retained head
Trying jhat now, reference chain here: -- cavm.h2$load_exp$fn__165@0x2aaab4b04660 (56 bytes) (field matrix:) -- clojure.lang.LazySeq@0x2aaab4b05388 (48 bytes) (field s:) -- clojure.lang.Cons@0x2aaab4b0fe08 (48 bytes) (field _more:) -- clojure.lang.LazySeq@0x2aaab4b10330 (48 bytes) (field s:) -- clojure.lang.Cons@0x2aaab6543ec0 (48 bytes) (field _more:) This looks to me like the head is being retained as a parameter to a function. load_exp signature is like so: defn load-exp [file timestamp filehash matrix] But surely passing a seq to a function doesn't retain the head? Here's the body of the function: (defn load-exp [file timestamp filehash matrix] (kdb/transaction (let [cid (merge-cohort file) exp (merge-exp file (format-timestamp timestamp) filehash cid) sids (load-samples cid (:samples (meta matrix)))] (load-exp-samples exp sids) (load-exp-matrix exp matrix load-exp-matrix calls (dorun (map)) on matrix. There's one other reference here, where we pull the metadata. Does any of this retain the head? On Tuesday, September 10, 2013 2:36:29 PM UTC-7, Brian Craft wrote: From jmap output it's pretty clear I'm retaining the head of a seq somehow, but I have no idea how, or where. Is there any way to find the reference, besides staring at code waiting for enlightenment? -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
It seems to be something about the jdbc/transaction macro. If I rewrite load_exp so it does nothing but walk the seq, it still consumes the heap: (defn load-exp [file timestamp filehash matrix] (jdbc/transaction (dorun (map identity matrix If I remove the jdbc/transaction call, it's fine. (defn load-exp [file timestamp filehash matrix] (dorun (map identity matrix))) On Tuesday, September 10, 2013 3:24:29 PM UTC-7, Brian Craft wrote: Trying jhat now, reference chain here: -- cavm.h2$load_exp$fn__165@0x2aaab4b04660 (56 bytes) (field matrix:) -- clojure.lang.LazySeq@0x2aaab4b05388 (48 bytes) (field s:) -- clojure.lang.Cons@0x2aaab4b0fe08 (48 bytes) (field _more:) -- clojure.lang.LazySeq@0x2aaab4b10330 (48 bytes) (field s:) -- clojure.lang.Cons@0x2aaab6543ec0 (48 bytes) (field _more:) This looks to me like the head is being retained as a parameter to a function. load_exp signature is like so: defn load-exp [file timestamp filehash matrix] But surely passing a seq to a function doesn't retain the head? Here's the body of the function: (defn load-exp [file timestamp filehash matrix] (kdb/transaction (let [cid (merge-cohort file) exp (merge-exp file (format-timestamp timestamp) filehash cid) sids (load-samples cid (:samples (meta matrix)))] (load-exp-samples exp sids) (load-exp-matrix exp matrix load-exp-matrix calls (dorun (map)) on matrix. There's one other reference here, where we pull the metadata. Does any of this retain the head? On Tuesday, September 10, 2013 2:36:29 PM UTC-7, Brian Craft wrote: From jmap output it's pretty clear I'm retaining the head of a seq somehow, but I have no idea how, or where. Is there any way to find the reference, besides staring at code waiting for enlightenment? -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
Copying the pattern of jdbc/transaction*, I tried this: (defn blah [func] (func)) (defn load-exp [file timestamp filehash matrix] (blah (fn [] (dorun (map identity matrix) which also consumes the heap. Can anyone explain to me what's happening here? Something about creating a anonymous function? On Tuesday, September 10, 2013 5:24:44 PM UTC-7, Brian Craft wrote: It seems to be something about the jdbc/transaction macro. If I rewrite load_exp so it does nothing but walk the seq, it still consumes the heap: (defn load-exp [file timestamp filehash matrix] (jdbc/transaction (dorun (map identity matrix If I remove the jdbc/transaction call, it's fine. (defn load-exp [file timestamp filehash matrix] (dorun (map identity matrix))) On Tuesday, September 10, 2013 3:24:29 PM UTC-7, Brian Craft wrote: Trying jhat now, reference chain here: -- cavm.h2$load_exp$fn__165@0x2aaab4b04660 (56 bytes) (field matrix:) -- clojure.lang.LazySeq@0x2aaab4b05388 (48 bytes) (field s:) -- clojure.lang.Cons@0x2aaab4b0fe08 (48 bytes) (field _more:) -- clojure.lang.LazySeq@0x2aaab4b10330 (48 bytes) (field s:) -- clojure.lang.Cons@0x2aaab6543ec0 (48 bytes) (field _more:) This looks to me like the head is being retained as a parameter to a function. load_exp signature is like so: defn load-exp [file timestamp filehash matrix] But surely passing a seq to a function doesn't retain the head? Here's the body of the function: (defn load-exp [file timestamp filehash matrix] (kdb/transaction (let [cid (merge-cohort file) exp (merge-exp file (format-timestamp timestamp) filehash cid) sids (load-samples cid (:samples (meta matrix)))] (load-exp-samples exp sids) (load-exp-matrix exp matrix load-exp-matrix calls (dorun (map)) on matrix. There's one other reference here, where we pull the metadata. Does any of this retain the head? On Tuesday, September 10, 2013 2:36:29 PM UTC-7, Brian Craft wrote: From jmap output it's pretty clear I'm retaining the head of a seq somehow, but I have no idea how, or where. Is there any way to find the reference, besides staring at code waiting for enlightenment? -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
Can anyone explain to me what's happening here? Something about creating a anonymous function? The problem is not creating the anonymous function but that it closes over the matrix argument. The closure passed on to blah will keep a reference to matrix until blah returns. This won't happen if the anonymous function takes the matrix as a parameter: (defn blah [func v] (func v)) (defn load-exp [matrix] (blah (fn [m] (dorun (map identity m))) matrix)) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
FWIW, Brian and I were looking at this off-list and we changed (dorun (map identity matrix)) to (doseq [x matrix] (identity x)) and that seemed to work just fine - even in Brian's more complicated case. Given that dorun specifically says it doesn't hold on to the head, I would have expected the two to behave identically. I also suspected the closure over the matrix argument as being the root cause but was puzzled when using doseq instead made the problem go away... Sean On Tue, Sep 10, 2013 at 7:34 PM, Armando Blancas abm221...@gmail.com wrote: Can anyone explain to me what's happening here? Something about creating a anonymous function? The problem is not creating the anonymous function but that it closes over the matrix argument. The closure passed on to blah will keep a reference to matrix until blah returns. This won't happen if the anonymous function takes the matrix as a parameter: (defn blah [func v] (func v)) (defn load-exp [matrix] (blah (fn [m] (dorun (map identity m))) matrix)) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out. -- Sean A Corfield -- (904) 302-SEAN An Architect's View -- http://corfield.org/ World Singles, LLC. -- http://worldsingles.com/ Perfection is the enemy of the good. -- Gustave Flaubert, French realist novelist (1821-1880) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
Ah. So the root of my problem is that the jdbc/transaction macro creates a closure in my function.. (defmacro transaction On Tuesday, September 10, 2013 7:34:32 PM UTC-7, Armando Blancas wrote: Can anyone explain to me what's happening here? Something about creating a anonymous function? The problem is not creating the anonymous function but that it closes over the matrix argument. The closure passed on to blah will keep a reference to matrix until blah returns. This won't happen if the anonymous function takes the matrix as a parameter: (defn blah [func v] (func v)) (defn load-exp [matrix] (blah (fn [m] (dorun (map identity m))) matrix)) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
(defmacro transaction [ body] `(transaction* (fn [] ~@body))) I'm not sure how to avoid that. The anonymous function created here doesn't take parameters. On Tuesday, September 10, 2013 8:55:42 PM UTC-7, Brian Craft wrote: Ah. So the root of my problem is that the jdbc/transaction macro creates a closure in my function.. (defmacro transaction On Tuesday, September 10, 2013 7:34:32 PM UTC-7, Armando Blancas wrote: Can anyone explain to me what's happening here? Something about creating a anonymous function? The problem is not creating the anonymous function but that it closes over the matrix argument. The closure passed on to blah will keep a reference to matrix until blah returns. This won't happen if the anonymous function takes the matrix as a parameter: (defn blah [func v] (func v)) (defn load-exp [matrix] (blah (fn [m] (dorun (map identity m))) matrix)) -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
Re: finding retained head
I also suspected the closure over the matrix argument as being the root cause but was puzzled when using doseq instead made the problem go away... Right, it doesn't seem to be a hold in the closure, unless the compiler could tell when to release it, which is the case when the code is macro expanded (doseq) or placed inline, but not when you call a function like dorun. That's really kind of puzzling. -- -- You received this message because you are subscribed to the Google Groups Clojure group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups Clojure group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
