Re: Running out of memory when using loop/recur and destructuring

2009-11-10 Thread Christophe Grand
On Wed, Nov 4, 2009 at 6:46 AM, John Harrop wrote: > On Tue, Nov 3, 2009 at 1:53 AM, Alex Osborne wrote: >> >> The new loop uses the outer-let to get around this: >> (let [G__13697 s >>       [x & xs] G__13697 >>       y xs] >>   (loop* [G__13697 G__13697 >>           y y] >>          (let [[x &

Re: Running out of memory when using loop/recur and destructuring

2009-11-10 Thread John Harrop
On Tue, Nov 10, 2009 at 7:21 AM, Rich Hickey wrote: > Right - pervasive locals clearing will definitely do the trick here. > Interestingly, when I was at Microsoft and asked them about handling > this issue for the CLR they stated plainly it wasn't an issue at all - > their system can fully detec

Re: Running out of memory when using loop/recur and destructuring

2009-11-10 Thread Rich Hickey
On Wed, Nov 4, 2009 at 8:47 AM, Christophe Grand wrote: > > On Tue, Nov 3, 2009 at 7:27 PM, Paul  Mooser wrote: >> >> Ah -- I hadn't understood that when using destructuring, that >> subsequent bindings could refer to the destructured elements. I should >> have, since clojure "only" has let*, and

Re: Running out of memory when using loop/recur and destructuring

2009-11-09 Thread Paul Mooser
I imagine he's just busy. At this point, I plan to create a ticket on assembla, if that's possible - I think I just need to create a login and then file it. On Nov 9, 2:07 pm, John Harrop wrote: > On Mon, Nov 9, 2009 at 4:31 PM, Rock wrote: > > I've been following this thread, and I must say I'

Re: Running out of memory when using loop/recur and destructuring

2009-11-09 Thread John Harrop
On Mon, Nov 9, 2009 at 4:31 PM, Rock wrote: > I've been following this thread, and I must say I'm puzzled that Rich > hasn't said anything at all about this issue yet. It seems important > enough to hear his own opinion. My observation over the past few months is that Rich has long absences awa

Re: Running out of memory when using loop/recur and destructuring

2009-11-09 Thread Rock
I've been following this thread, and I must say I'm puzzled that Rich hasn't said anything at all about this issue yet. It seems important enough to hear his own opinion. On 6 Nov, 18:56, Paul Mooser wrote: > So, I've been hoping that Rich (or someone?) would weigh in on this, > and give the go

Re: Running out of memory when using loop/recur and destructuring

2009-11-05 Thread Paul Mooser
It does make me wonder, however, if having the lazy-seq cache things is sort of conflating laziness and consistency, since as you point out, not all ISeq implementations do any sort of caching. I wonder if it would be interesting to decompose it into 'lazy- seq' (uncached), and 'cached-seq'. I un

Re: Running out of memory when using loop/recur and destructuring

2009-11-04 Thread Paul Mooser
I completely understand the difference between the ISeq interface, and the particular implementation (lazy-seq) that results in these problems. It would be fairly straightforward, I think, to write some kind of uncached-lazy-seq which doesn't exhibit these problems, but I've felt that is sidestepp

Re: Running out of memory when using loop/recur and destructuring

2009-11-04 Thread Chouser
On Tue, Nov 3, 2009 at 11:51 PM, Mark Engelberg wrote: > > Clojure's built-in "range" function (last time I looked) essentially > produces an uncached sequence.  And that makes a lot of sense. 'range' has since changed and now produces a chunked lazy seq (master branch post-1.0). > Producing th

Re: Running out of memory when using loop/recur and destructuring

2009-11-04 Thread Paul Mooser
Well, I care (conceptually) more about the fix being made, rather than the exact timeframe. If we had to wait until clojure-in-clojure, I think I could live with that, since the issue can be readily avoided. We'll see if Rich has a chance to chime-in to acknowledge whether or not he considers this

Re: Running out of memory when using loop/recur and destructuring

2009-11-04 Thread Christophe Grand
On Tue, Nov 3, 2009 at 7:27 PM, Paul Mooser wrote: > > Ah -- I hadn't understood that when using destructuring, that > subsequent bindings could refer to the destructured elements. I should > have, since clojure "only" has let*, and this behavior seems > consistent with that, for binding. > > Ee

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread John Harrop
On Tue, Nov 3, 2009 at 1:53 AM, Alex Osborne wrote: > The new loop uses the outer-let to get around this: > > (let [G__13697 s > [x & xs] G__13697 > y xs] > (loop* [G__13697 G__13697 > y y] > (let [[x & xs] G__13697 >y y] >...))) > Now

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Mark Engelberg
I agree that seqs carry a large degree of risk. You have to work very hard to avoid giving your large sequences a name, lest you accidentally "hang on to the head". In Clojure's early days, I complained about this and described some of my own experiments with uncached sequences. Rich said he wa

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Paul Mooser
I understand the pragmatism of your approach, but it's really unfortunate. Seqs are a really convenient abstraction, and the ability to model arbitrarily large or infinite ones (with laziness) is really useful. In my opinion, only using seqs when all of the data can be fit into memory really under

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Paul Mooser
In the particular case given below, I'd assume that during the invocation of print-seq, the binding to "s" (the head of the sequence) would be retained, because my mental model for the execution environment of a function is that it is the environment in which they were declared, extended with the

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Brian Hurt
We encountered similar problems at work trying to wrap I/O up into lazy seq's. The problem is that it is very easy to accidentally hold on to the head of a seq while enumerating it's elements. In addition, we had problems with not closing file descriptors. A common pattern was to open a file, pr

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Brian Hurt
On Tue, Nov 3, 2009 at 5:19 PM, Paul Mooser wrote: > > I understand the pragmatism of your approach, but it's really > unfortunate. Seqs are a really convenient abstraction, and the ability > to model arbitrarily large or infinite ones (with laziness) is really > useful. In my opinion, only using

Re: Running out of memory when using loop/recur and destructuring

2009-11-03 Thread Paul Mooser
Ah -- I hadn't understood that when using destructuring, that subsequent bindings could refer to the destructured elements. I should have, since clojure "only" has let*, and this behavior seems consistent with that, for binding. Eeww. It seems like quite a thorny issue to solve, even if simple to

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Alex Osborne
Paul Mooser wrote: > Good job tracking down that diff -- upon looking at it, unfortunately, > I obviously don't understand the underlying issue being fixed (the > inter-binding dependencies) because the "old code" basically matches > what I would think would be the way to avoid introducing this in

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Paul Mooser
Good job tracking down that diff -- upon looking at it, unfortunately, I obviously don't understand the underlying issue being fixed (the inter-binding dependencies) because the "old code" basically matches what I would think would be the way to avoid introducing this in an outer let form -- clear

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Paul Mooser
This is great advice, of course. On the other hand, I feel it's important to be explicitly clear about which forms will hold on to (seemingly) transient data. Certain things are explicitly clear about this (such as the docstring for doseq), and this particular case is unfortunate because in the co

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread John Harrop
On Mon, Nov 2, 2009 at 2:39 PM, Christophe Grand wrote: > Right now I can't see how loop can be made to support both cases. > Hopefully someone else will. In the meantime, remember that it's always worth trying to implement seq-processing in terms of map, reduce, filter, for, and friends if poss

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Christophe Grand
Hi Paul, It's indeed surprising and at first glance, it looks like a bug but after researching the logs, this let form was introduced in the following commit http://github.com/richhickey/clojure/commit/288f34dbba4a9e643dd7a7f77642d0f0088f95ad with comment "fixed loop with destructuring and inter-

Re: Running out of memory when using loop/recur and destructuring

2009-11-02 Thread Paul Mooser
I'm a little surprised I haven't seen more response on this topic, since this class of bug (inadvertently holding onto the head of sequences) is pretty nasty to run into, and is sort of awful to debug. I'm wondering if there's a different way to write the loop macro so that it doesn't expand into

Re: Running out of memory when using loop/recur and destructuring

2009-10-31 Thread Paul Mooser
>From looking at the source code the loop macro, it looks like this might be particular to destructuring with loop, rather than being related to destructuring in general ? --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Group

Re: Running out of memory when using loop/recur and destructuring

2009-10-31 Thread Paul Mooser
A user on IRC named hiredman had the excellent idea (which should have occurred to me, but didn't) to macroexpand my code. A macro expansion of (loop [[head & tail] (repeat 1)] (recur tail)) results in: (let* [G__10 (repeat 1) vec__11 G__10 head (clojure.core/nth vec__11 0 ni

Re: Running out of memory when using loop/recur and destructuring

2009-10-31 Thread Paul Mooser
I actually restructured my code (not the toy example posted here) to avoid the destructuring, and was disappointed to find it also eventually blows up on 1.6 as well. I'm reasonably certain in that case that I'm not holding on to any of the sequence (since I don't refer to it outside the invocatio

Re: Running out of memory when using loop/recur and destructuring

2009-10-30 Thread John Harrop
On Fri, Oct 30, 2009 at 3:15 PM, Paul Mooser wrote: > Is this behavior due to some artifact of destructuring I'm not aware > of (or something else I'm missing), or is there a bug? If it sounds > like a bug, can anyone else reproduce? > > Thanks! I vaguely remember something like this coming up

Running out of memory when using loop/recur and destructuring

2009-10-30 Thread Paul Mooser
I was working with a large data set earlier today, and I had written a loop/recur where I was passing in a huge seq to the first iteration, and I was surprised when I ran out of heap space, because I was very careful not to hold on to the head of the seq, and I though that loop ended up rebinding