Hi I am having difficulties linking my two machines into a hadoop cluster so I am running mahout jobs in a single machine and I am running into java.lang.OutOfMemoryError issues when the input files are big (see outputs below, one is "Java heap space" and the other is "GC overhead limit exceeded").
I don't have much experience with Java and I am wondering is there something I can tweak within Mahout, Hadoop or Java to increase the amount of memory accessible by these jobs? ./mahout-distribution-0.7/bin/mahout fpg -i ../data/input1.csv -o ../data/patterns -k 50 -method sequential -s 20 MAHOUT_LOCAL is not set; adding HADOOP_CONF_DIR to classpath. Running on hadoop, using /usr/bin/hadoop and HADOOP_CONF_DIR=/etc/hadoop MAHOUT-JOB: /root/src/mahout-distribution-0.7/mahout-examples-0.7-job.jar 13/02/15 20:46:25 INFO common.AbstractJob: Command line arguments: {--encoding=[UTF-8], --endPhase=[2147483647], --input=[../data/input1.csv], --maxHeapSize=[50], --method=[sequential], --minSupport=[20], --numGroups=[1000], --numTreeCacheEntries=[5], --output=[../data/patterns], --splitterPattern=[[ , ]*[,| ][ , ]*], --startPhase=[0], --tempDir=[temp]} 13/02/15 20:46:25 INFO pfpgrowth.FPGrowthDriver: Starting Sequential FPGrowth 13/02/15 20:46:26 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/02/15 20:46:38 INFO fpgrowth.FPGrowth: Number of unique items 3603 13/02/15 20:46:38 INFO fpgrowth.FPGrowth: Number of unique pruned items 3603 13/02/15 20:46:38 INFO fpgrowth.FPGrowth: FPTree Building: Read 10000 Transactions 13/02/15 20:46:38 INFO fpgrowth.FPGrowth: FPTree Building: Read 20000 Transactions 13/02/15 20:46:38 INFO fpgrowth.FPGrowth: FPTree Building: Read 30000 Transactions 13/02/15 20:46:39 INFO fpgrowth.FPGrowth: FPTree Building: Read 40000 Transactions 13/02/15 20:46:39 INFO fpgrowth.FPGrowth: FPTree Building: Read 50000 Transactions 13/02/15 20:46:39 INFO fpgrowth.FPGrowth: FPTree Building: Read 60000 Transactions 13/02/15 20:46:39 INFO fpgrowth.FPGrowth: FPTree Building: Read 70000 Transactions 13/02/15 20:46:40 INFO fpgrowth.FPGrowth: FPTree Building: Read 80000 Transactions 13/02/15 20:46:40 INFO fpgrowth.FPGrowth: FPTree Building: Read 90000 Transactions 13/02/15 20:46:40 INFO fpgrowth.FPGrowth: FPTree Building: Read 100000 Transactions 13/02/15 20:46:40 INFO fpgrowth.FPGrowth: FPTree Building: Read 110000 Transactions 13/02/15 20:46:41 INFO fpgrowth.FPGrowth: FPTree Building: Read 120000 Transactions 13/02/15 20:46:41 INFO fpgrowth.FPGrowth: FPTree Building: Read 130000 Transactions 13/02/15 20:46:41 INFO fpgrowth.FPGrowth: FPTree Building: Read 140000 Transactions 13/02/15 20:46:42 INFO fpgrowth.FPGrowth: FPTree Building: Read 150000 Transactions 13/02/15 20:46:42 INFO fpgrowth.FPGrowth: FPTree Building: Read 160000 Transactions 13/02/15 20:46:43 INFO fpgrowth.FPGrowth: FPTree Building: Read 170000 Transactions 13/02/15 20:46:44 INFO fpgrowth.FPGrowth: FPTree Building: Read 180000 Transactions 13/02/15 20:46:45 INFO fpgrowth.FPGrowth: FPTree Building: Read 190000 Transactions 13/02/15 20:46:45 INFO fpgrowth.FPGrowth: FPTree Building: Read 200000 Transactions 13/02/15 20:46:46 INFO fpgrowth.FPGrowth: FPTree Building: Read 210000 Transactions 13/02/15 20:46:48 INFO fpgrowth.FPGrowth: FPTree Building: Read 220000 Transactions 13/02/15 20:46:49 INFO fpgrowth.FPGrowth: FPTree Building: Read 230000 Transactions Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at org.apache.mahout.fpm.pfpgrowth.fpgrowth.FPTree.resize(FPTree.java:357) at org.apache.mahout.fpm.pfpgrowth.fpgrowth.FPTree.createNode(FPTree.java:189) at org.apache.mahout.fpm.pfpgrowth.fpgrowth.FPGrowth.treeAddCount(FPGrowth.java:699) at org.apache.mahout.fpm.pfpgrowth.fpgrowth.FPGrowth.generateTopKFrequentPatterns(FPGrowth.java:293) at org.apache.mahout.fpm.pfpgrowth.fpgrowth.FPGrowth.generateTopKFrequentPatterns(FPGrowth.java:174) at org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.runFPGrowth(FPGrowthDriver.java:183) at org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.run(FPGrowthDriver.java:132) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.main(FPGrowthDriver.java:56) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) ./mahout-distribution-0.7/bin/mahout fpg -i ../data/input2.csv -o ../data/patterns -k 50 -method sequential -s 20 > tmp_ouput 13/02/15 20:55:19 INFO common.AbstractJob: Command line arguments: {--encoding=[UTF-8], --endPhase=[2147483647], --input=[../data/input2.csv], --maxHeapSize=[50], --method=[sequential], --minSupport=[20], --numGroups=[1000], --numTreeCacheEntries=[5], --output=[../data/patterns], --splitterPattern=[[ , ]*[,| ][ , ]*], --startPhase=[0], --tempDir=[temp]} 13/02/15 20:55:19 INFO pfpgrowth.FPGrowthDriver: Starting Sequential FPGrowth 13/02/15 20:55:19 INFO util.NativeCodeLoader: Loaded the native-hadoop library Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded at org.apache.mahout.fpm.pfpgrowth.fpgrowth.FPGrowth.generateFList(FPGrowth.java:87) at org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.runFPGrowth(FPGrowthDriver.java:183) at org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.run(FPGrowthDriver.java:132) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.mahout.fpm.pfpgrowth.FPGrowthDriver.main(FPGrowthDriver.java:56) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68) at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) at org.apache.mahout.driver.MahoutDriver.main(MahoutDriver.java:195) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.apache.hadoop.util.RunJar.main(RunJar.java:156)