Re: Fine-grained locals clearing
On Sat, Dec 12, 2009 at 10:57 AM, Daniel Werner wrote: > On Dec 10, 3:10 pm, Rich Hickey wrote: >> I'm happy to announce I have implemented this fine-grained locals >> clearing in the compiler, in the 'new' branch. > > Is there a chance for this feature to find its way into master before > Clojure 1.1 is released? > Unlikely. It is a pretty extensive change, and hasn't yet seen the kind of exercise the rest of the code in 1.1 has. That said, the bulk of what might constitute a version 1.2 is already done, so the gap between 1.1 and 1.2 might be short. Rich -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Fine-grained locals clearing
On Dec 10, 3:10 pm, Rich Hickey wrote: > I'm happy to announce I have implemented this fine-grained locals > clearing in the compiler, in the 'new' branch. Is there a chance for this feature to find its way into master before Clojure 1.1 is released? -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Fine-grained locals clearing
> Also, it may be very useful to try all your code on 'new' *without* > taking > advantage of the new features, and reporting back on any > breakage. That's more what I was thinking. While I find the new features interesting, I'm less jazzed about spending the time to build on features that might go away. (Chicken and egg, of course.) It sounds like it's mostly the bleeding edge of `new` that's bleeding, so to speak, and the stable part is fairly reliable. If that's a correct impression, then it's probably worth developing against it. I'm not overly concerned about the occasional regression, because it's easy enough to switch to master to verify. It's systematic regression that I'd like to avoid. > You won't have to wait too long though, I suspect. Once 1.1 is > out the door I assume 'new' will be merged into 'master' fairly > quickly. All the more reason to bang on it now, I suppose! -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Fine-grained locals clearing
I can't express how thrilled I am that you did this work.Thanks so much - since I've run into a few of these classes of bugs, I'll see if I can switch over to new and try to run against some big data sets and give some feedback, if I can find the time. On Dec 10, 6:10 am, Rich Hickey wrote: > I'm happy to announce I have implemented this fine-grained locals > clearing in the compiler, in the 'new' branch. It should automatically > cover all cases in which the code doesn't explicitly reuse the head - > including non-tail usage, destructuring etc. In short, such cases > should 'just work' from now on. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Fine-grained locals clearing
On Thu, Dec 10, 2009 at 2:26 PM, Richard Newman wrote: >> +1 As cool as the new branch is, this is the first compelling reason >> I've seen to go to my boss and say we need to switch to it now. >> >> Thanks Rich! > > Speaking of which... I know the new branch is where Rich publishes his > current work. Does anyone (Rich included!) have any opinion on whether > it's reasonable to do 'mainstream' development against new (rather > than working against master and occasionally testing against new)? > > I currently work against master, but I keep my checkout up-to-date. I would not recommend doing large amounts of development against 'new'. Things change there in ways that break compatibility with earlier changes there (not usually with 'master' through), so if you start using features from 'new' and keep your checkout up-to-date, your code may break at any time without notice. For context: I don't think I've ever recommended against using master. I would however recommend playing around with 'new', trying out some of the new features on key parts of your code. Also, it may be very useful to try all your code on 'new' *without* taking advantage of the new features, and reporting back on any breakage. You won't have to wait too long though, I suspect. Once 1.1 is out the door I assume 'new' will be merged into 'master' fairly quickly. --Chouser -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Fine-grained locals clearing
> +1 As cool as the new branch is, this is the first compelling reason > I've seen to go to my boss and say we need to switch to it now. > > Thanks Rich! Speaking of which... I know the new branch is where Rich publishes his current work. Does anyone (Rich included!) have any opinion on whether it's reasonable to do 'mainstream' development against new (rather than working against master and occasionally testing against new)? I currently work against master, but I keep my checkout up-to-date. Imaginary counter-points would be "bugs that get fixed on master don't get merged into new", "it's really only for experimentation", "it's often broken for days at a time", "things behave differently to master", "functions keep changing their names", "building a jar doesn't work", etc. I ask because I maintain a bunch of libraries and do quite a lot of new development, and so I see a possible win/win: I get to enjoy new improvements, but I also act as a real-world-code tester for the community. That only applies, though, if new has a modicum of stability and reliability. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Fine-grained locals clearing
+1 As cool as the new branch is, this is the first compelling reason I've seen to go to my boss and say we need to switch to it now. Thanks Rich! On Dec 10, 6:40 am, "Stephen C. Gilardi" wrote: > On Dec 10, 2009, at 9:10 AM, Rich Hickey wrote: > > What a great change! We ran into the problem this solves just this week: > running out of memory on long lazy seqs with large elements that we intended > to be processing one item at a time. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Fine-grained locals clearing
On Dec 10, 2009, at 9:10 AM, Rich Hickey wrote: > I'm happy to announce I have implemented this fine-grained locals > clearing in the compiler, in the 'new' branch. It should automatically > cover all cases in which the code doesn't explicitly reuse the head - > including non-tail usage, destructuring etc. What a great change! We ran into the problem this solves just this week: running out of memory on long lazy seqs with large elements that we intended to be processing one item at a time. > In short, such cases should 'just work' from now on. ... adding yet another entry on the long list of Clojure facilities that adhere to the principle of least surprise. Nicely done! --Steve smime.p7s Description: S/MIME cryptographic signature
Re: Fine-grained locals clearing
Rockin'. On Thu, Dec 10, 2009 at 9:10 AM, Rich Hickey wrote: > One of the objectives of Clojure is to reduce incidental complexity. > And one of the biggest sources of incidental complexity in Clojure was > the retention of the head of a lazy sequence due to its being > referenced by some local (argument or local (let) binding). One might > expect that, if no subsequent code in the body of a function uses that > arg/local, it would be subject to GC. Unfortunately, on the JVM, that > is, in many cases, not true - the local is considered a live reference > and is thus not GCed. This yields the infamous 'holding onto the head' > problem, and subsequent Out Of Memory errors on large data sets. > > I had put in place a workaround, which was the 'clearing/nulling-out' > of locals on the tail call of the function. This helps in many, but > not all, cases. Not all logic flows are amenable to rearrangement to > leverage this cleanup. And there are many cases where the local is not > visible - e.g. when destructuring. > > The full solution is to track, during compilation, the lifetime of all > locals on all branches of the execution path and to emit code that > clears them at the point of last use in any particular branch. > > I'm happy to announce I have implemented this fine-grained locals > clearing in the compiler, in the 'new' branch. It should automatically > cover all cases in which the code doesn't explicitly reuse the head - > including non-tail usage, destructuring etc. In short, such cases > should 'just work' from now on. > > N.B. that this is strictly a lifetime management issue and does not > change the nature of lazy sequences - they are real, linked data > structures, the tail of which might not yet have been created. They > are most emphatically *not* ephemeral streams of values. However, with > fine-grained locals clearing, they are subject to GC 'as you go', > delivering the benefits of both. > > If you've got a pet case of incidental head-retention, please try out > the 'new' branch and let me know how it goes. > > Thanks, > > Rich > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
