Hello,
In line 9, you retain the head in a var.
Transform it into a function:
(defn xml []
(-> large-file
java.io/input-stream
clojure.data.xml/source-seq))
HTH,
Laurent
2014/1/9 Peter Ullah <[email protected]>
>
> Hi Everyone,
>
> I am trying to write some Clojure to process a large (310M) XML file, in
> doing so, the code seems to consume the heap. I wrote a smaller script to
> simplify the problem but I have now reduced it to a point where I am just
> confused with why the heap is being used up; I'm consuming a lazy sequence
> and not (that I am aware) maintaining a reference to the head. I have also
> removed Leiningen from the equation just in case the REPL was causing any
> references to be kept.
>
> I have managed to process the file as desired by increasing -Xmx and -Xms
> passed to the VM but it doesn't address the underlying problem and my
> confusion.
>
> Here is the code: https://gist.github.com/anonymous/8332220
>
> Here are some details:
>
> - XML file has 6,321,720 lines.
> - Using clojure.data.xml/source-seq to create a lazy sequence, which
> translates into ~15,000,000 elements in the sequence
> - Keeping a counter to determine how far through the processing I have
> got before failure... (i.e. GC out of memory errors)
> - Increased available and initial heap size to such a point that the
> processing completes.
> - Used VisualVM to profile memory characteristics - see attachment
>
>
> My problem is in understanding why the used heap is increasing and not
> constant.
>
> Any help would be great. thanks.
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to [email protected]
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> [email protected]
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
