If ordering isn't important, I'd just dump them all into a set instead of
manually checking whether or or not you already put the url into a set.
On Sunday, January 26, 2014 10:46:46 PM UTC-8, danneu wrote:
>
> I use line-seq, split, and destructuring to parse large CSVs.
>
> Here's how I'd approach what I think you're trying to do:
>
> (with-open [rdr (io/reader (io/resource csv :encoding "UTF-16"))]
> (let [extract-url-hash (fn [line]
> (let [[_ _ _ url & _] (str/split line
> #"\t")]
> [(m/md5 url) url]))]
> (->> (drop 1 (line-seq rdr))
> (map extract-url-hash)
> (into {}))))
>
> https://gist.github.com/danneu/8644022
>
> On Tuesday, January 21, 2014 12:55:00 AM UTC-6, Jarrod Swart wrote:
>>
>> I'm processing a large csv with Clojure, honestly not even that big (~18k
>> rows, 11mb). I have a list of exported data from a client and I am
>> de-duplicating URLs within the list. My final output is a series of
>> vectors: [url url-hash].
>>
>> The odd thing is how slow it seems to be going. I have tried
>> implementing this as a reduce, and finally I thought to speed things up I
>> might try a "with-open and a loop-recur". It doesn't seem to have done
>> much in my case. I know I am doing something wrong I'm just not sure what
>> yet. The best I can do is about 4 seconds, which may only seem slow
>> because I implemented it in python first and it takes a half second to
>> finish. Still this is one of the smaller files I will likely deal with so
>> I'm worried that as the files grow it may get too slow.
>>
>> The code is here on ref-heap for easy viewing:
>> https://www.refheap.com/26098
>>
>> Any advice is appreciated.
>>
>
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
