On Wednesday, September 4, 2013 2:13:12 AM UTC-7, Brian Craft wrote:
>
>
>
> On Tuesday, September 3, 2013 10:03:00 PM UTC-7, Mikera wrote:
>>
>>
>>
>> On Wednesday, 4 September 2013 12:37:33 UTC+8, Brian Craft wrote:
>>>
>>>
>>>
>>> On Tuesday, September 3, 2013 9:14:30 PM UTC-7, Mikera wrote:
>>>>
>>>> On Wednesday, 4 September 2013 10:00:42 UTC+8, Brian Craft wrote:
>>>>
>>>>> I'm loading data files of about 1-2G, which are composed of a bunch of
>>>>> numeric data blocks. I need to store the data blocks w/o storing
>>>>> duplicates. They arrive as vectors of floats, and are stored as primitive
>>>>> byte arrays.
>>>>>
>>>>> I first tried memoizing the function that saves a block (returning an
>>>>> id), with the core memoize function. This failed because every block
>>>>> became
>>>>> a different key in the memoization, regardless of the content. It looks
>>>>> like clojure treats variables referencing primitive arrays as equal only
>>>>> if
>>>>> they refer to the same array. Note:
>>>>>
>>>>> cavm.core=> ({[1 2 3] "foo"} [1 2 3])
>>>>> "foo"
>>>>> cavm.core=> ({(float-array [1 2 3]) "foo"} (float-array [1 2 3]))
>>>>> nil
>>>>> cavm.core=> (let [a (float-array [1 2 3])] ({a "foo"} a))
>>>>> "foo"
>>>>>
>>>>>
>>>>>
>>>>> I next tried memoizing over the vector of floats, however performance
>>>>> became pathologically slow, and the process threw an OOM. I'm guessing
>>>>> this
>>>>> is due to the memory requirements of a clojure vector of floats vs. a
>>>>> primitive array of bytes holding the same data. Is there an easy way to
>>>>> compare the storage requirements?
>>>>>
>>>>> Any suggestions on how better to handle this?
>>>>>
>>>>
>>>> You may want to use the :ndarray-float array implementation in the
>>>> latest version of core.matrix.
>>>>
>>>> This is effectively a wrapper over a raw Java float array: so your
>>>> storage requirement should be close to the size of the raw byte data
>>>> (assuming the data blocks are large enough that the size of the wrapper is
>>>> negligible)
>>>>
>>>
>>> Ah, interesting.
>>>
>>> > *matrix-implementation*
>>> :vectorz
>>> > ({(matrix [1 2 3 4]) "foo"} (matrix [1 2 3 4]))
>>> "foo"
>>>
>>> I don't otherwise need core.matrix at this point in the loader, but this
>>> is convenient. Why does that work?
>>>
>>
>> That works because Vectorz (the underlying Java lib) has a sane
>> implementation of .equals and .hashCode. It's pretty fast as well, though
>> it is still O(n) since it doesn't do hashcode caching.
>>
>> Note that the :vectorz implementation uses 8-byte doubles rather than
>> 4-byte floats though - so if you really need single precision to keep the
>> overall memory usage down then it might not be the best choice. I
>> personally never use 4-byte floats because the numerical errors soon become
>> problematic, but YMMV.
>>
>>
> It looks like I can get this working for byte arrays as follows.
>
> (deftype BAHashable [ba]
> Object
> (equals [f g] (java.util.Arrays/equals ba (.ba g)))
> (hashCode [f] (java.util.Arrays/hashCode ba)))
>
> ({(BAHashable. (byte-array (map byte [1 2 3]))) "foo"} (BAHashable.
> (byte-array (map byte [1 2 3]))))
> "foo"
>
>
> I'm less certain of whether this is a good idea.
>
This gives me a number of reflection warnings, on field ba, on equals, and
on hashCode. I can eliminate the one on hashCode by type hinting the ba
parameter of BAHashable.
The one on (.ba g) and equals remains. Is there any way to hint these?
--
--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to [email protected]
Note that posts from new members are moderated - please be patient with your
first post.
To unsubscribe from this group, send email to
[email protected]
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups
"Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
For more options, visit https://groups.google.com/groups/opt_out.
