On Feb 26, 2009, at 13:00 , Manlio Perillo wrote:
Kenneth Hoste ha scritto:
Hello,
I'm having a go at the Netflix Prize using Haskell. Yes, I'm brave.
[...]
To see if I could efficiently represent the data set in this way, I
wrote a small
Haskell program (attached) which uses the following data type:
From what I see, to append a new integer to the Array, you convert
the array to a list, append the new element to the list, and then
convert to array again.
Isn't this a bit inefficient?
Yes, performance-wise this is terribly inefficient, I agree. But, it
was just an artefact of how the raw data is organized.
My main concern was the memory usage of the huge IntMap with UArray
elements.
Once I solved that, I would be able to get around the performance
issue by reorganizing the raw data.
However, as I posted yesterday, I've been able to circumvent the issue
by rethinking my data type, i.e. using
the ~18K movie IDs as key instead of the 480K user IDs, which
radically limits the overhead...
That way, I'm able to fit the data set in <700M of memory, without
having to reorganize the raw data.
The uvector package implements a vector of unboxed types, and has an
snocU operation, to append an element to the array.
I don't know how efficient it is, however.
By the way, about uvector: it has a Stream data type, and you can
build a vector from a stream.
Thanks for letting me know, I'll keep this in mind.
greetings,
Kenneth
--
Kenneth Hoste
Paris research group - ELIS - Ghent University, Belgium
email: kenneth.ho...@elis.ugent.be
website: http://www.elis.ugent.be/~kehoste
blog: http://boegel.kejo.be
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
