I use line-seq, split, and destructuring to parse large CSVs.
Here's how I'd approach what I think you're trying to do:
(with-open [rdr (io/reader (io/resource csv :encoding "UTF-16"))]
(let [extract-url-hash (fn [line]
(let [[_ _ _ url & _] (str/split line
#"\t")]
[(m/md5 url) url]))]
(->> (drop 1 (line-seq rdr))
(map extract-url-hash)
(into {}))))
https://gist.github.com/danneu/8644022
On Tuesday, January 21, 2014 12:55:00 AM UTC-6, Jarrod Swart wrote:
>
> I'm processing a large csv with Clojure, honestly not even that big (~18k
> rows, 11mb). I have a list of exported data from a client and I am
> de-duplicating URLs within the list. My final output is a series of
> vectors: [url url-hash].
>
> The odd thing is how slow it seems to be going. I have tried implementing
> this as a reduce, and finally I thought to speed things up I might try a
> "with-open and a loop-recur". It doesn't seem to have done much in my
> case. I know I am doing something wrong I'm just not sure what yet. The
> best I can do is about 4 seconds, which may only seem slow because I
> implemented it in python first and it takes a half second to finish. Still
> this is one of the smaller files I will likely deal with so I'm worried
> that as the files grow it may get too slow.
>
> The code is here on ref-heap for easy viewing:
> https://www.refheap.com/26098
>
> Any advice is appreciated.
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
