Hi, I've been trying to run Mahout (with Hadoop) on our data for quite sometime now. Everything is fine on relatively small data sets, but when I try to do K-Means clustering with the aid of Canopy on like 300000 documents, I can't even get past the canopy generation because of OOM. We're going to cluster similar news so T1, and T2 are set to 0.84, and 0.6 (those values lead to desired results on sample data).
I tried setting both "mapred.map.child.java.opts", and "mapred.reduce.child.java.opts" to "-Xmx4096M", I also exported HADOOP_HEAPSIZE to 4000, and still having issues. I'm running all of this in Hadoop's single node, pseudo-distributed mode on a machine with 16GB of RAM. Searching Internet for solutions I found this[1]. One of the bullet points states that: "In all of the algorithms, all clusters are retained in memory by the mappers and reducers" So my question is, does Mahout on Hadoop only help in distributing CPU bound operations? What one should do if they have a large dataset, and only a handful of low-RAM commodity nodes? I'm obviously a newbie, thanks for bearing with me. [1] http://mail-archives.apache.org/mod_mbox/mahout-user/201209.mbox/%3c506307eb.3090...@windwardsolutions.com%3E Cheers, Amir