Passing Custom Arguments for giraph.zkList
Hi, I am trying to pass a zookeeper quorum to my giraph job and it throws the following exception: 13/08/29 13:14:38 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 13/08/29 13:14:38 INFO utils.ConfigurationUtils: No output format specified. Ensure your OutputFormat does not require one. 13/08/29 13:14:38 INFO utils.ConfigurationUtils: Setting custom argument [giraph.zkList] to zk1 in GiraphConfiguration Exception in thread main java.lang.IllegalArgumentException: Unable to parse custom argument: zk2:port at org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:288) at org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:147) at com.paypal.risk.rd.giraph.AccountPropagation.run(AccountPropagation.java:46) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at com.paypal.risk.rd.giraph.AccountPropagation.main(AccountPropagation.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) I pass the zklist like this: Hadoop jar GRAPH.jar CLASSNAME -vip CLASS NAME -vif CLASS NAME -wc CLASS NAME -w worker number -ca giraph.zkList=zk1:port,zk2:port,zk3:port,zk4:port,zk5:port Please suggest what is wrong with this invocation. Thanks Arun Ramani
Re: Passing Custom Arguments for giraph.zkList
Hi Claudio, Yes zk1, zk2, zk3, zk4 and zk5 are all zookeeper hostnames. These 5 hosts make a zookeeper quorum. Please let me know how to pass this. Thanks Arun Ramani From: Claudio Martella claudio.marte...@gmail.commailto:claudio.marte...@gmail.com Reply-To: user@giraph.apache.orgmailto:user@giraph.apache.org user@giraph.apache.orgmailto:user@giraph.apache.org Date: Thursday, August 29, 2013 2:18 PM To: user@giraph.apache.orgmailto:user@giraph.apache.org user@giraph.apache.orgmailto:user@giraph.apache.org Subject: Re: Passing Custom Arguments for giraph.zkList zk1 is supposed to be a hostname. On Thu, Aug 29, 2013 at 11:05 PM, Ramani, Arun aram...@paypal.commailto:aram...@paypal.com wrote: Hi, I am trying to pass a zookeeper quorum to my giraph job and it throws the following exception: 13/08/29 13:14:38 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 13/08/29 13:14:38 INFO utils.ConfigurationUtils: No output format specified. Ensure your OutputFormat does not require one. 13/08/29 13:14:38 INFO utils.ConfigurationUtils: Setting custom argument [giraph.zkList] to zk1 in GiraphConfiguration Exception in thread main java.lang.IllegalArgumentException: Unable to parse custom argument: zk2:port at org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:288) at org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:147) at com.paypal.risk.rd.giraph.AccountPropagation.run(AccountPropagation.java:46) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) at com.paypal.risk.rd.giraph.AccountPropagation.main(AccountPropagation.java:98) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:197) I pass the zklist like this: Hadoop jar GRAPH.jar CLASSNAME -vip CLASS NAME -vif CLASS NAME -wc CLASS NAME -w worker number -ca giraph.zkList=zk1:port,zk2:port,zk3:port,zk4:port,zk5:port Please suggest what is wrong with this invocation. Thanks Arun Ramani -- Claudio Martella claudio.marte...@gmail.commailto:claudio.marte...@gmail.com
Re: Giraph and Fair Scheduler
Hi Claudio, Which timeout are you referring? Thanks Arun Ramani From: Claudio Martella claudio.marte...@gmail.commailto:claudio.marte...@gmail.com Reply-To: user@giraph.apache.orgmailto:user@giraph.apache.org user@giraph.apache.orgmailto:user@giraph.apache.org Date: Tuesday, May 7, 2013 5:23 AM To: user@giraph.apache.orgmailto:user@giraph.apache.org user@giraph.apache.orgmailto:user@giraph.apache.org Subject: Re: Giraph and Fair Scheduler it's probably an obvious question, but have you tried increasing the timeout? On Tue, May 7, 2013 at 3:37 AM, Ramani, Arun aram...@paypal.commailto:aram...@paypal.com wrote: Hi, I am running Fair scheduler with many applications in hadoop stack in my cluster (like pig, hive, hbase etc). I have dedicated a pool for Giraph and want to run giraph along with those other applications. I have configured pre-emption and and set the minsharepreemptiontimeout=5 (sec – for the jobs submitted to this pool to wait to get the min share). I am trying to run giraph in this mode. I see that jobs from other pools are getting pre-empted to give the giraph job's pool its configured min share but my job fails with Unable to create native thread error. This same job passes if the slots are available immediately without having to wait for the tasks from other queues to be pre-empted. I also tried to tweak the giraph.minPercentResponded=50.0f. My Giraph job still fails. Please help in this scenario. Basically, I wanted to know how to configure giraph to wait for a threshold for the slots to be available for it through pre-emption. Thanks Arun Ramani -- Claudio Martella claudio.marte...@gmail.commailto:claudio.marte...@gmail.com
