Why not keep track of the hash of blocks already stored and discard
duplicates blocks from memory asap ?
I am talking here of keeping blocks as Clojure structures so the hash will be
consistent between two identical blocks.
Do you store them away as soon as you can ? Or are you trying to keep them
around somehow in memory ?
Luc
> I'm loading data files of about 1-2G, which are composed of a bunch of
> numeric data blocks. I need to store the data blocks w/o storing
> duplicates. They arrive as vectors of floats, and are stored as primitive
> byte arrays.
>
> I first tried memoizing the function that saves a block (returning an id),
> with the core memoize function. This failed because every block became a
> different key in the memoization, regardless of the content. It looks like
> clojure treats variables referencing primitive arrays as equal only if they
> refer to the same array. Note:
>
> cavm.core=> ({[1 2 3] "foo"} [1 2 3])
> "foo"
> cavm.core=> ({(float-array [1 2 3]) "foo"} (float-array [1 2 3]))
> nil
> cavm.core=> (let [a (float-array [1 2 3])] ({a "foo"} a))
> "foo"
>
>
>
> I next tried memoizing over the vector of floats, however performance
> became pathologically slow, and the process threw an OOM. I'm guessing this
> is due to the memory requirements of a clojure vector of floats vs. a
> primitive array of bytes holding the same data. Is there an easy way to
> compare the storage requirements?
>
> Any suggestions on how better to handle this?
>
> --
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to [email protected]
> Note that posts from new members are moderated - please be patient with your
> first post.
> To unsubscribe from this group, send email to
> [email protected]
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/groups/opt_out.
>
--
Softaddicts<[email protected]> sent by ibisMail from my ipad!
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
