Re: Need YARN - Test job.

2014-01-03 Thread Gunnar Tapper
Hi Guarav,

What are you setting HADOOP_CONF_DIR to? IME, you get the hang if you don't
set it as:

 export HADOOP_CONF_DIR=/etc/hadoop/conf.cloudera.yarn1

Gunnar


On Fri, Jan 3, 2014 at 6:47 AM, Gaurav Shankhdhar <
shankhdhar.gau...@gmail.com> wrote:

> Folks,
>
> I am trying to run "teragen" program using YARN framework in CDH 4.5 but
> no luck.
> What i have figured out till now:
>
> 1. By default Demo VM run the MR in MRv1 uses JobTracker and TaskTracker.
> 2. You can either run MRv1 or MRv2 but not both.
> 3. I disabled MRv1 and configured YARN by increasing the alternatives
> priority for YARN, deployed the Client Configuration (Using Cloudera
> Manager).
> 4. Submitted the teragen program and can see the YARN framework is in
> action from Web UI's.
> 5. But it hangs at Map 0% and Reduce %.
>
> Tried multiple option to no avail.
>
> Any step by Step guide or doc to execute MR in MRv2 (YARN) mode in CDH4.5?
>
> Attached doc with screen shots.
>
> Reference:
> http://www.cloudera.com/content/cloudera-content/
> cloudera-docs/CM4Ent/latest/Cloudera-Manager-Managing-
> Clusters/cmmc_adding_YARN_MRv2.html
>
> Regards
> Gaurav
>
> On Friday, January 3, 2014 3:33:44 AM UTC+5:30, Dhanasekaran Anbalagan
> wrote:
>>
>> Hi Guys,
>>
>> we recently installed CDH5 in our test cluster. we need any test job for
>> YARN framework.
>>
>> we are able to run mapreduce job successfully on YARN framework without
>> code change.
>>
>> But we need to test yarn job functionality. can you please guide me.
>>
>> we tired https://github.com/hortonworks/simple-yarn-app it's not help us.
>> tech@dvcloudlab231:~$ *hadoop jar simple-yarn-app-1.0-SNAPSHOT.jar
>> com.hortonworks.simpleyarnapp.Client /bin/date 2
>> /apps/simple/simple-yarn-app-1.0-SNAPSHOT.jar*
>> 14/01/02 16:49:05 INFO client.RMProxy: Connecting to ResourceManager at
>> dvcloudlab231/192.168.70.231:8032
>> Submitting application application_1388687890867_0007
>> 14/01/02 16:49:05 INFO impl.YarnClientImpl: Submitted application
>> application_1388687890867_0007 to ResourceManager at dvcloudlab231/
>> 192.168.70.231:8032
>> Application application_1388687890867_0007 finished with state FAILED at
>> 1388699348344
>>
>> Note:
>> In Resource manager node. I don't see any container logs.
>>
>> -Dhanasekaran.
>>
>> Did I learn something today? If not, I wasted it.
>>
>  --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscr...@cloudera.org.
> For more options, visit
> https://groups.google.com/a/cloudera.org/groups/opt_out.
>



-- 
Thanks,

Gunnar
*If you think you can you can, if you think you can't you're right.*


Re: Need YARN - Test job.

2014-01-02 Thread Gunnar Tapper
Hi Dhahasekaran,

You can use the Hadoop workload to do this testing. The key is to set the
HADOOP_CONF_DIR envar to the right location:

*Yarn*

export HADOOP_CONF_DIR=/etc/hadoop/conf.cloudera.yarn1
hadoop fs -rm -r /terasort/TS_input2
yarn jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
teragen  -D dfs.replication=1 -Dmapred.map.tasks=1441
/terasort/TS_input2

*MapReduce*
export HADOOP_CONF_DIR=/etc/hadoop/conf.cloudera.mapreduce1
hadoop fs -rm -r /terasort/TS_input2
yarn jar
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.0-cdh4.4.0.jar
teragen  -D dfs.replication=1 -Dmapred.map.tasks=1441
/terasort/TS_input2
I hope this helps.
Thanks,

Gunnar


On Thu, Jan 2, 2014 at 3:03 PM, Dhanasekaran Anbalagan
wrote:

> Hi Guys,
>
> we recently installed CDH5 in our test cluster. we need any test job for
> YARN framework.
>
> we are able to run mapreduce job successfully on YARN framework without
> code change.
>
> But we need to test yarn job functionality. can you please guide me.
>
> we tired https://github.com/hortonworks/simple-yarn-app it's not help us.
> tech@dvcloudlab231:~$ *hadoop jar simple-yarn-app-1.0-SNAPSHOT.jar
> com.hortonworks.simpleyarnapp.Client /bin/date 2
> /apps/simple/simple-yarn-app-1.0-SNAPSHOT.jar*
> 14/01/02 16:49:05 INFO client.RMProxy: Connecting to ResourceManager at
> dvcloudlab231/192.168.70.231:8032
> Submitting application application_1388687890867_0007
> 14/01/02 16:49:05 INFO impl.YarnClientImpl: Submitted application
> application_1388687890867_0007 to ResourceManager at dvcloudlab231/
> 192.168.70.231:8032
> Application application_1388687890867_0007 finished with state FAILED at
> 1388699348344
>
> Note:
> In Resource manager node. I don't see any container logs.
>
> -Dhanasekaran.
>
> Did I learn something today? If not, I wasted it.
>
> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "CDH Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cdh-user+unsubscr...@cloudera.org.
> For more options, visit
> https://groups.google.com/a/cloudera.org/groups/opt_out.
>



-- 
Thanks,

Gunnar
*If you think you can you can, if you think you can't you're right.*