Well, shoot. I went back and revisited this because it was bugging me … I
looked at the code generated with and without usage of ^:once fn* and that led
me down the right path. TL;DR – the problem appears to be locals clearing
always disabled in Cursive REPL. This is just my current hypothesis based on
testing and not 100% certainty. I guess I'll put this in a blog post or
something once I validate this hypothesis and get some feedback from Colin
(1.6.0-eap1-2017.2 if you're watching, Colin ;)
For mailing list posterity, here's the difference, using Lubomir's examples and
the decompiler in IntelliJ, between ^:once fn* and regular fn (or fn*)
Note the inner-bad function:
(defn bad []
(let [x (for [n (range)] (make-array Object 10000)), f (fn* inner-bad [] (nth
x 1e6))] (f)))
The inner-bad class generated is:
public final class core$bad$inner_bad__431 extends AFunction {
Object x;
public static final Object const__1 = 1000000.0D;
public core$bad$inner_bad__431(Object var1) {
this.x = var1;
}
public Object invoke() {
return RT.nth(this.x, RT.intCast((Number)const__1));
}
}
Note the inner-good function:
(defn good []
(let [x (for [n (range)] (make-array Object 10000)), f (^:once fn* inner-good
[] (nth x 1e6))] (f)))
The inner-good class generated is:
public final class core$good$inner_good__414 extends AFunction {
Object x;
public static final Object const__1 = 1000000.0D;
public core$good$inner_good__414(Object var1) {
this.x = var1;
}
public Object invoke() {
Object var10000 = this.x;
this.x = null;
return RT.nth(var10000, RT.intCast((Number)const__1));
}
}
You can see pretty clearly the difference is that each time the function is
invoked in the "good" case, the local reference to this.x is cleared, meaning
this function really is only good for one invocation. No wonder it's
undocumented!
=> (let [z 1000
f (^:once fn* one-shot [x] (println x))
p (f z)]
(p)
(p))
1000
CompilerException java.lang.NullPointerException
Anyway, I saw this and jumped to the thought: "locals!! I have fallen victim to
uncleared locals!!"
So I started some tests, going back to my previous code
<https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e> and trying
the JDBC result set that wasn't being lazy before. I put the test1 invocation
into -main and tried:
lein run – no OOM.
lein repl, then invoking (-main) – no OOM.
lein repl :headless, then connecting via lein repl :connect "localhost:54321"
and invoke (-main) – no OOM.
Cursive nREPL via Leiningen (locals clearing, debug REPL) – OOM!!
Cursive nREPL via Leiningen (no locals clearing, debug REPL) – OOM!!
Cursive nREPL via Leiningen (no locals clearing, regular REPL) – OOM!!
Cursive nREPL via Leiningen (no locals clearing, regular REPL) – OOM!!
lein run, with -Dclojure.compiler.disable-locals-clearing=true – OOM!!
I feel slightly better for knowing I can point the finger of blame at disabled
locals clearing. It would be great to have this fixed in Cursive, considering
that I run quite memory intensive applications and I'm sure locals clearing
being permanently disabled has caused me to engineer around some non-existent
problems. I have not tested CIDER or any other REPL, YMMV.
Luke.
> On Jun 20, 2017, at 11:26 AM, Lubomir Konstantinov <lubo...@gmail.com> wrote:
>
> ^{:once true}
>
> should deal with this. Try the following simplified test case with and
> without it:
>
> (defn query [result-fn]
> (let [x (for [n (range 1e6)] (make-array Object 100000))
> f (^:once fn* [rs] (result-fn rs))] (f x)))
>
> (defn testq []
> (let [myfn (fn [rs] (doseq [r rs] nil))]
> (query myfn)))
>
>
>
> On Tuesday, 20 June 2017 03:47:45 UTC+3, James Reeves wrote:
> This might be a bug in java.jdbc. The code that passes the result set seq to
> the function is:
>
> (^{:once true} fn* [rset]
> ((^{:once true} fn* [rs]
> (result-set-fn (if as-arrays?
> (cons (first rs)
> (map row-fn (rest rs)))
> (map row-fn rs))))
> (result-set-seq rset opts)))
>
> I'm wondering if this function holds onto the head of the seq, since it's
> bound to "rs".
>
> On 20 June 2017 at 00:20, Luke Burton <luke_...@me.com <>> wrote:
>
> Anyone have any insights here? Really the most important thing I'm trying to
> learn is 2) how to identify when a lazy seq head is being retained, other
> than waiting for it to become bad enough that your program OOMs.
>
>
>> On Jun 16, 2017, at 6:14 PM, Luke Burton <luke_...@me.com <>> wrote:
>>
>>
>> Riddle me this:
>>
>> https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e
>> <https://gist.github.com/hagmonk/a75621b143501966c22f53ed1e2bc36e>
>>
>> Wherein I synthesize a large table in Postgres, then attempt to lazily load
>> the table, discarding each row as I receive it. I tried *many* permutations
>> and experiments, but settled on these two tests to illustrate my point.
>> Which is that I simply can't get it to work with clojure.java.jdbc.
>>
>> test1, according to all my research and reading of the source code involved,
>> should consume the query results lazily. It does not, and I can't for the
>> life of me figure out why. Traffic starts to stream in, and the heap is
>> overwhelmed almost immediately. I've deliberately set the heap to 1 GB.
>>
>> test2 uses a technique I borrowed wholesale from Ghadi Shayban in JDBC-99
>> <https://dev.clojure.org/jira/browse/JDBC-99?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs>,
>> which is to have ResultSet implement IReduceInit. It consumes a nominal
>> amount of memory. I've verified it's actually doing something by putting
>> counters in, and using YourKit to watch about 20 MB/s of traffic streaming
>> into the JVM. It's brilliant, it doesn't even break 200 MB total heap usage.
>>
>> I used YourKit to track where the memory is being retained for test1.
>> Initially I made the mistake of not setting the fetchSize, so I saw an
>> ArrayList inside the driver holding the reference. The driver documentation
>> <https://jdbc.postgresql.org/documentation/head/query.html> confirms that
>> autoCommit must be disabled and the fetchSize set to some non-zero number.
>>
>> After making that change, YourKit confirmed that the GC root holding all the
>> memory was the stack local variable "rs". At least I think it did, as a
>> non-expert in this domain. I tried disassembling the functions using
>> no.disassemble and the IntelliJ decompiler but I'm not really at the point
>> where I understand what to look for.
>>
>> So my questions are:
>>
>> 1) what am I doing wrong with clojure.java.jdbc?
>>
>> Note some things I've already tried:
>>
>> * using row-fn instead of result-set-fn
>> * using prepared statements
>> * explicitly setting auto-commit false on the connection
>> * declaring my result-set-fn with (^{:once true} *fn […]) (I did not see a
>> change in the disassembly when using this)
>> * probably other things I am forgetting
>>
>> 2) in these situations where you suspect that the head of a lazy sequence is
>> being retained, how do you reason about it? I'm kind of lucky this one blew
>> the heap so quickly, who knows how much of my production code might burning
>> memory unnecessarily but not quite as fatally. Do you disassemble the
>> functions and observe some smoking gun? How do you peek under the covers to
>> see where the problem is?
>>
>> Luke.
>>
>> --
>> You received this message because you are subscribed to the Google
>> Groups "Clojure" group.
>> To post to this group, send email to clo...@googlegroups.com <>
>> Note that posts from new members are moderated - please be patient with your
>> first post.
>> To unsubscribe from this group, send email to
>> clojure+u...@googlegroups.com <>
>> For more options, visit this group at
>> http://groups.google.com/group/clojure?hl=en
>> <http://groups.google.com/group/clojure?hl=en>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "Clojure" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to clojure+u...@googlegroups.com <>.
>> For more options, visit https://groups.google.com/d/optout
>> <https://groups.google.com/d/optout>.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clo...@googlegroups.com <>
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+u...@googlegroups.com <>
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> <http://groups.google.com/group/clojure?hl=en>
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+u...@googlegroups.com <>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
>
>
>
> --
> James Reeves
> booleanknot.com <http://booleanknot.com/>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to clojure@googlegroups.com
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> <http://groups.google.com/group/clojure?hl=en>
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com
> <mailto:clojure+unsubscr...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout
> <https://groups.google.com/d/optout>.
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
