Re: lazy-sequences and memory leaks
Hi Sunil, Might the problem lie within the function that consumes the seq-of-maps? There is a reduce in there that accumulates some data. I can't tell from the code whether that would amount to much, though. Regards, Caspar -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: lazy-sequences and memory leaks
Sorry, I don't have any relevant experience to share. As an experiment, I suggest that you try deferring the sh execution. Maybe logging the generated commands to a script file rather than calling sh during processing. Then execute one big script file at the end. That should make the program easier to profile and debug. On Feb 27, 2012, at 11:13 PM, Sunil S Nandihalli wrote: > Hi Everybody, > I am using lazy-seqs to join two very large csv files. I am very certain > that I am not holding on to any of the heads and If I did .. the jvm would be > out of memory far sooner than what I am seeing currently. The size of the > file is something like 73 G and the Ram allocated to the jvm is about 8G . It > seems like a very gradual leak. Has anybody else encountered similar > problems? In case some of you feel that my code might be the culprit, the > following gist has the source. > > https://gist.github.com/1929345 > > Thanks, > Sunil. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: lazy-sequences and memory leaks
It's not obvious to me from the code what the problem would be. Have you tried using a profiler to see what kinds of objects account for the memory? When I've run into seq issues and bugs in the past, that was pretty helpful in figuring out the underlying problem. On Feb 27, 8:13 pm, Sunil S Nandihalli wrote: > Hi Everybody, > I am using lazy-seqs to join two very large csv files. I am very certain > that I am not holding on to any of the heads and If I did .. the jvm would > be out of memory far sooner than what I am seeing currently. The size of > the file is something like 73 G and the Ram allocated to the jvm is about > 8G . It seems like a very gradual leak. Has anybody else encountered > similar problems? In case some of you feel that my code might be the > culprit, the following gist has the source. > > https://gist.github.com/1929345 > > Thanks, > Sunil. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: lazy-sequences and memory leaks
thank you .. some times it is hard to come up with good names .. suggestions welcome .. :) Sunil. On Tue, Feb 28, 2012 at 9:47 AM, Baishampayan Ghose wrote: > > I am using lazy-seqs to join two very large csv files. I am very certain > > that I am not holding on to any of the heads and If I did .. the jvm > would > > be out of memory far sooner than what I am seeing currently. The size of > the > > file is something like 73 G and the Ram allocated to the jvm is about 8G > . > > It seems like a very gradual leak. Has anybody else encountered similar > > problems? In case some of you feel that my code might be the culprit, the > > following gist has the source. > > OT, but I hereby nominate > > `lazy-join-sorted-map-seqs-with-only-second-map-seq-allowed-to-have-duplicate-fields` > as the most awesome fn name of the decade ;-) > > Regards, > BG > > -- > Baishampayan Ghose > b.ghose at gmail.com > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: lazy-sequences and memory leaks
I would like to hear what you may have to say regarding this. As always.. I pressed send sooner than I wanted to.. On Tue, Feb 28, 2012 at 9:43 AM, Sunil S Nandihalli < sunil.nandiha...@gmail.com> wrote: > Hi Everybody, > I am using lazy-seqs to join two very large csv files. I am very certain > that I am not holding on to any of the heads and If I did .. the jvm would > be out of memory far sooner than what I am seeing currently. The size of > the file is something like 73 G and the Ram allocated to the jvm is about > 8G . It seems like a very gradual leak. Has anybody else encountered > similar problems? In case some of you feel that my code might be the > culprit, the following gist has the source. > > https://gist.github.com/1929345 > > Thanks, > Sunil. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: lazy-sequences and memory leaks
> I am using lazy-seqs to join two very large csv files. I am very certain > that I am not holding on to any of the heads and If I did .. the jvm would > be out of memory far sooner than what I am seeing currently. The size of the > file is something like 73 G and the Ram allocated to the jvm is about 8G . > It seems like a very gradual leak. Has anybody else encountered similar > problems? In case some of you feel that my code might be the culprit, the > following gist has the source. OT, but I hereby nominate `lazy-join-sorted-map-seqs-with-only-second-map-seq-allowed-to-have-duplicate-fields` as the most awesome fn name of the decade ;-) Regards, BG -- Baishampayan Ghose b.ghose at gmail.com -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
lazy-sequences and memory leaks
Hi Everybody, I am using lazy-seqs to join two very large csv files. I am very certain that I am not holding on to any of the heads and If I did .. the jvm would be out of memory far sooner than what I am seeing currently. The size of the file is something like 73 G and the Ram allocated to the jvm is about 8G . It seems like a very gradual leak. Has anybody else encountered similar problems? In case some of you feel that my code might be the culprit, the following gist has the source. https://gist.github.com/1929345 Thanks, Sunil. -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
