What's the value of "bsp.max.tasks.per.job"? Please try to run Pi example and then how many tasks are launched?
On Sun, Mar 9, 2014 at 8:24 PM, Ammar Sahib <[email protected]> wrote: > I was sure to restart HAMA. According to the web UI, I have the following in > my cluster: > > > master:40000 Hama BSP AdministrationState: RUNNING > Started: Sun Mar 09 12:17:01 CET 2014 > Version: 0.6.4 > Compiled By: edward > Compiled At Time: Mon Mar 3 19:14:32 KST 2014 > Identifier: 201403091217 > ________________________________ > > Groom Servers > BSP Task Capacity > Avg. Tasks/Node > Blacklisted Nodes > 4 120 30.00 0 > ________________________________ > > Running Jobs > No jobs found! > ________________________________ > > All Jobs History > No jobs found! > ________________________________ > Hama, 2014. > > > But When running my job I will something different: > > 14/03/09 12:19:21 INFO bsp.FileInputFormat: Total input paths to process : 1 > 14/03/09 12:19:22 INFO util.NativeCodeLoader: Loaded the native-hadoop library > 14/03/09 12:19:22 WARN snappy.LoadSnappy: Snappy native library not loaded > 14/03/09 12:19:22 INFO bsp.FileInputFormat: Total input paths to process : 1 > Exception in thread "main" java.io.IOException: Job failed! The number of > splits has exceeded the number of max tasks. The number of splits: 52, The > number of max tasks: 20 > at > org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:349) > at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296) > at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219) > at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226) > at org.apache.hama.bsp.BSPJobClient.partition(BSPJobClient.java:460) > at > org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:341) > at org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296) > at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219) > at org.apache.hama.graph.GraphJob.submit(GraphJob.java:208) > at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226) > at > de.rwthaachen.dbis.i5cloudmatch.controller.Matcher.main(Matcher.java:479) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) > > > > > > On Sunday, March 9, 2014 12:23 AM, Edward J. Yoon <[email protected]> > wrote: > > Please use the web UI to check the cluster capacity. > > I think your cluster is not working correctly now, or you didn't > restart your cluster. > > On Sun, Mar 9, 2014 at 2:00 AM, Ammar Sahib <[email protected]> wrote: >> Hi >> >> I tried to increase the bsp.tasks.maximum in hama-default.xml from 3 to 30 >> but I still get the same error. >> I am thinking of of reducing the number of blocks of the input file by >> controlling the parameter dfs.namenode.fs-limits.min-block-size in >> hdfs-default.xml. Do you think this might be a good approach that might >> solve the problem? >> >> >> >> On Friday, March 7, 2014 11:32 PM, Edward J. Yoon <[email protected]> >> wrote: >> If the number of blocks of input file is 52 (see [1]), you should >> increase number of task slots by adding new machine or increasing max >> tasks number per node "bsp.tasks.maximum". >> >> 1. >> http://stackoverflow.com/questions/11168427/viewing-the-number-of-blocks-for-a-file-in-hadoop >> >> On Sat, Mar 8, 2014 at 12:22 AM, Ammar Sahib <[email protected]> wrote: >>> Hi >>> >>> >>> I am using HAMA 0.6.4 and I am running my custom program using a cluster >>> of 4 machines. My input is a single file and I am setting the number of >>> BSP tasks to the number of Groom servers by using >>> (JOB.setNumBspTask(cluster.getGroomServers()). I am using the >>> HashPartitioner.class to partition the data. >>> >>> >>> I have a problem when I load my data. When I run my custom program I get >>> the following error messages: >>> >>> 14/03/07 16:02:34 INFO bsp.FileInputFormat: Total input paths to process : >>> 1 >>> 14/03/07 16:02:34 INFO util.NativeCodeLoader: Loaded the >>> native-hadoop library >>> 14/03/07 16:02:34 WARN snappy.LoadSnappy: Snappy native library not loaded >>> 14/03/07 16:02:34 INFO bsp.FileInputFormat: Total input paths to process : >>> 1 >>> Exception >>> in thread "main" java.io.IOException: Job failed! The number of splits >>> has exceeded the number of max tasks. The number of splits: 52, The >>> number of max tasks: 20 >>> at >>> org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:349) >>> at >>> org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296) >>> at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219) >>> at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226) >>> at >>> org.apache.hama.bsp.BSPJobClient.partition(BSPJobClient.java:460) >>> >>> at >>> org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:341) >>> at >>> org.apache.hama.bsp.BSPJobClient.submitJob(BSPJobClient.java:296) >>> at org.apache.hama.bsp.BSPJob.submit(BSPJob.java:219) >>> at org.apache.hama.graph.GraphJob.submit(GraphJob.java:208) >>> at org.apache.hama.bsp.BSPJob.waitForCompletion(BSPJob.java:226) >>> at >>> de.rwthaachen.dbis.i5cloudmatch.controller.Matcher.main(Matcher.java:479) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >>> >>> Any advice of how to solve this problem? >>> >>> RegardsAmmar >> >> >> >> -- >> Edward J. Yoon (@eddieyoon) >> Chief Executive Officer >> DataSayer, Inc. >> >> > > > > -- > Edward J. Yoon (@eddieyoon) > Chief Executive Officer > DataSayer, Inc. -- Edward J. Yoon (@eddieyoon) Chief Executive Officer DataSayer, Inc.
