[
https://issues.apache.org/jira/browse/MAHOUT-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13758103#comment-13758103
]
Peng Cheng commented on MAHOUT-1286:
------------------------------------
The existing open addressing hash table is for 1d arrays, not 2d matrices. I
can get the concurrency done by next week, but there are simply too many
pending optimization. e.g. if you set loadfactor to 1.2 it is pretty slow. If
you can help improving on the TODO list in the code that will be awesome.
Not sure about the consequence as 2d matrix interface has an int (16bit) index,
but dataModel has a long (32bit) index. If you don't bother adding more things
to mahout-math, then it should be alright.
> Memory-efficient DataModel, supporting fast online updates and element-wise
> iteration
> -------------------------------------------------------------------------------------
>
> Key: MAHOUT-1286
> URL: https://issues.apache.org/jira/browse/MAHOUT-1286
> Project: Mahout
> Issue Type: Improvement
> Components: Collaborative Filtering
> Affects Versions: 0.9
> Reporter: Peng Cheng
> Labels: collaborative-filtering, datamodel, patch, recommender
> Fix For: 0.9
>
> Attachments: InMemoryDataModel.java, InMemoryDataModelTest.java,
> Semifinal-implementation-added.patch
>
> Original Estimate: 336h
> Remaining Estimate: 336h
>
> Most DataModel implementation in current CF component use hash map to enable
> fast 2d indexing and update. This is not memory-efficient for big data set.
> e.g. Netflix prize dataset takes 11G heap space as a FileDataModel.
> Improved implementation of DataModel should use more compact data structure
> (like arrays), this can trade a little of time complexity in 2d indexing for
> vast improvement in memory efficiency. In addition, any online recommender or
> online-to-batch converted recommender will not be affected by this in
> training process.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
