Yeah, rewrite that to not hold the head of "lines". In particular, get rid
of "counts" entirely, get rid of "(nth lines ...)", and use something like
(loop [lines (seq lines)
res {}]
(if lines
(recur (next lines) (update-res res (first lines)))
res))
as your loop.
On Tue, Nov 12, 2013 at 3:19 AM, Jiaqi Liu <[email protected]> wrote:
> hi , guys
> I wrote the following codes to parse log files.
> it's alright to parse small one.
> But with big log files , i got the following error:
> "OutOfMemoryError GC overhead limit exceeded clojure.core/line-seq
> (core.clj:2679)"
>
> (defn parse-file
> ""
> [file]
> (with-open [rdr (io/reader file)]
> (println "Statistic result : " (parse-recur rdr))))
>
> (defn parse-recur
> ""
> [rdr]
> (let [lines (line-seq rdr)
> counts (count lines)]
> (loop [len counts
> res {}]
> (if (zero? len)
> res
> (recur (dec len)
> (update-res res (nth lines (- counts len))))))))
>
> (defn update-res
> ""
> [res line]
> (let [params (string/split line #",")
> id (if (> (count params) 1) (params 0) "0")]
> (if (res id)
> (update-in res [id] inc)
> (assoc res id 1))))
>
>
> How can fix this bug ?
> and how to optimize this produce to run faster ?
>
> Any suggestion will be appreciated~
>
> BR
>
> ------------------------------------
>
> 刘家齐 （Jacky Liu）
>
>
>
> 手机：15201091195 邮箱：[email protected]
>
> Skype：jacky_liu_1987 QQ：406229156
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to [email protected]
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> [email protected]
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
