I'm loading data files of about 1-2G, which are composed of a bunch of numeric data blocks. I need to store the data blocks w/o storing duplicates. They arrive as vectors of floats, and are stored as primitive byte arrays.
I first tried memoizing the function that saves a block (returning an id), with the core memoize function. This failed because every block became a different key in the memoization, regardless of the content. It looks like clojure treats variables referencing primitive arrays as equal only if they refer to the same array. Note: cavm.core=> ({[1 2 3] "foo"} [1 2 3]) "foo" cavm.core=> ({(float-array [1 2 3]) "foo"} (float-array [1 2 3])) nil cavm.core=> (let [a (float-array [1 2 3])] ({a "foo"} a)) "foo" I next tried memoizing over the vector of floats, however performance became pathologically slow, and the process threw an OOM. I'm guessing this is due to the memory requirements of a clojure vector of floats vs. a primitive array of bytes holding the same data. Is there an easy way to compare the storage requirements? Any suggestions on how better to handle this? -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.