On Tuesday, September 3, 2013 9:14:30 PM UTC-7, Mikera wrote:
>
> On Wednesday, 4 September 2013 10:00:42 UTC+8, Brian Craft wrote:
>
>> I'm loading data files of about 1-2G, which are composed of a bunch of
>> numeric data blocks. I need to store the data blocks w/o storing
>> duplicates. They arrive as vectors of floats, and are stored as primitive
>> byte arrays.
>>
>> I first tried memoizing the function that saves a block (returning an
>> id), with the core memoize function. This failed because every block became
>> a different key in the memoization, regardless of the content. It looks
>> like clojure treats variables referencing primitive arrays as equal only if
>> they refer to the same array. Note:
>>
>> cavm.core=> ({[1 2 3] "foo"} [1 2 3])
>> "foo"
>> cavm.core=> ({(float-array [1 2 3]) "foo"} (float-array [1 2 3]))
>> nil
>> cavm.core=> (let [a (float-array [1 2 3])] ({a "foo"} a))
>> "foo"
>>
>>
>>
>> I next tried memoizing over the vector of floats, however performance
>> became pathologically slow, and the process threw an OOM. I'm guessing this
>> is due to the memory requirements of a clojure vector of floats vs. a
>> primitive array of bytes holding the same data. Is there an easy way to
>> compare the storage requirements?
>>
>> Any suggestions on how better to handle this?
>>
>
> You may want to use the :ndarray-float array implementation in the latest
> version of core.matrix.
>
> This is effectively a wrapper over a raw Java float array: so your storage
> requirement should be close to the size of the raw byte data (assuming the
> data blocks are large enough that the size of the wrapper is negligible)
>
Ah, interesting.
> *matrix-implementation*
:vectorz
> ({(matrix [1 2 3 4]) "foo"} (matrix [1 2 3 4]))
"foo"
I don't otherwise need core.matrix at this point in the loader, but this is
convenient. Why does that work?
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clojure@googlegroups.com
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
clojure+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to clojure+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
