any thoughts from the logs and config I have shared? On Aug 21, 2016 8:32 AM, "rammohan ganapavarapu" <rammohanga...@gmail.com> wrote:
> so in job.properties what is the jobtracker property, is it RM ip: port or > scheduler port which is 8030, if I use 8030 I am getting unknown protocol > proto buffer error. > > On Aug 21, 2016 7:37 AM, "Sunil Govind" <sunil.gov...@gmail.com> wrote: > >> Hi. >> >> It seems its an oozie issue. From conf, RM scheduler is running at port >> 8030. >> But your job.properties is taking 8032. I suggest you could double >> confirm your oozie configuration and see the configurations are intact to >> contact RM. Sharing a link also >> https://discuss.zendesk.com/hc/en-us/articles/203355837-How- >> to-run-a-MapReduce-jar-using-Oozie-workflow >> >> Thanks >> Sunil >> >> >> On Sun, Aug 21, 2016 at 8:41 AM rammohan ganapavarapu < >> rammohanga...@gmail.com> wrote: >> >>> Please find the attached config that i got from yarn ui and AM,RM logs. >>> I only see that connecting to 0.0.0.0:8030 when i submit job using >>> oozie, but if i submit as yarn jar its working fine as i posted in my >>> previous posts. >>> >>> Here is my oozie job.properties file, i have a java class that just >>> prints >>> >>> nameNode=hdfs://master01:8020 >>> jobTracker=master01:8032 >>> workflowName=EchoJavaJob >>> oozie.use.system.libpath=true >>> >>> queueName=default >>> hdfsWorkflowHome=/user/uap/oozieWorkflows >>> >>> workflowPath=${nameNode}${hdfsWorkflowHome}/${workflowName} >>> oozie.wf.application.path=${workflowPath} >>> >>> Please let me know if you guys find any clue why its trying to connect >>> to 0.0.0.:8030. >>> >>> Thanks, >>> Ram >>> >>> >>> On Fri, Aug 19, 2016 at 11:54 PM, Sunil Govind <sunil.gov...@gmail.com> >>> wrote: >>> >>>> Hi Ram >>>> >>>> From the console log, as Rohith said, AM is looking for AM at 8030. So >>>> pls confirm the RM port once. >>>> Could you please share AM and RM logs. >>>> >>>> Thanks >>>> Sunil >>>> >>>> On Sat, Aug 20, 2016 at 10:36 AM rammohan ganapavarapu < >>>> rammohanga...@gmail.com> wrote: >>>> >>>>> yes, I did configured. >>>>> >>>>> On Aug 19, 2016 7:22 PM, "Rohith Sharma K S" <ksrohithsha...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hi >>>>>> >>>>>> From below discussion and AM logs, I see that AM container has >>>>>> launched but not able to connect to RM. >>>>>> >>>>>> This looks like your configuration issue. Would you check your >>>>>> job.xml jar that does *yarn.resourcemanager.scheduler.address *has >>>>>> been configured? >>>>>> >>>>>> Essentially, this address required by MRAppMaster for connecting to >>>>>> RM for heartbeats. If you don’t not configure, default value will be >>>>>> taken >>>>>> i.e 8030. >>>>>> >>>>>> >>>>>> Thanks & Regards >>>>>> Rohith Sharma K S >>>>>> >>>>>> On Aug 20, 2016, at 7:02 AM, rammohan ganapavarapu < >>>>>> rammohanga...@gmail.com> wrote: >>>>>> >>>>>> Even if the cluster dont have enough resources it should connect to " >>>>>> >>>>>> /0.0.0.0:8030" right? it should connect to my <RM_HOST:8030>, not sure >>>>>> why its trying to connect to 0.0.0.0:8030. >>>>>> >>>>>> I have verified the config and i removed traces of 0.0.0.0 still no luck. >>>>>> >>>>>> org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at >>>>>> /0.0.0.0:8030 >>>>>> >>>>>> If an one has any clue please share. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Ram >>>>>> >>>>>> >>>>>> >>>>>> On Fri, Aug 19, 2016 at 2:32 PM, rammohan ganapavarapu < >>>>>> rammohanga...@gmail.com> wrote: >>>>>> >>>>>>> When i submit a job using yarn its seems working only with oozie its >>>>>>> failing i guess, not sure what is missing. >>>>>>> >>>>>>> yarn jar /uap/hadoop/share/hadoop/mapre >>>>>>> duce/hadoop-mapreduce-examples-2.7.1.jar pi 20 1000 >>>>>>> Number of Maps = 20 >>>>>>> Samples per Map = 1000 >>>>>>> . >>>>>>> . >>>>>>> . >>>>>>> Job Finished in 19.622 seconds >>>>>>> Estimated value of Pi is 3.14280000000000000000 >>>>>>> >>>>>>> Ram >>>>>>> >>>>>>> On Fri, Aug 19, 2016 at 11:46 AM, rammohan ganapavarapu < >>>>>>> rammohanga...@gmail.com> wrote: >>>>>>> >>>>>>>> Ok, i have used yarn-utils.py to get the correct values for my >>>>>>>> cluster and update those properties and restarted RM and NM but still >>>>>>>> no >>>>>>>> luck not sure what i am missing, any other insights will help me. >>>>>>>> >>>>>>>> Below are my properties from yarn-site.xml and map-site.xml. >>>>>>>> >>>>>>>> python yarn-utils.py -c 24 -m 63 -d 3 -k False >>>>>>>> Using cores=24 memory=63GB disks=3 hbase=False >>>>>>>> Profile: cores=24 memory=63488MB reserved=1GB usableMem=62GB >>>>>>>> disks=3 >>>>>>>> Num Container=6 >>>>>>>> Container Ram=10240MB >>>>>>>> Used Ram=60GB >>>>>>>> Unused Ram=1GB >>>>>>>> yarn.scheduler.minimum-allocation-mb=10240 >>>>>>>> yarn.scheduler.maximum-allocation-mb=61440 >>>>>>>> yarn.nodemanager.resource.memory-mb=61440 >>>>>>>> mapreduce.map.memory.mb=5120 >>>>>>>> mapreduce.map.java.opts=-Xmx4096m >>>>>>>> mapreduce.reduce.memory.mb=10240 >>>>>>>> mapreduce.reduce.java.opts=-Xmx8192m >>>>>>>> yarn.app.mapreduce.am.resource.mb=5120 >>>>>>>> yarn.app.mapreduce.am.command-opts=-Xmx4096m >>>>>>>> mapreduce.task.io.sort.mb=1024 >>>>>>>> >>>>>>>> >>>>>>>> <property> >>>>>>>> <name>mapreduce.map.memory.mb</name> >>>>>>>> <value>5120</value> >>>>>>>> </property> >>>>>>>> <property> >>>>>>>> <name>mapreduce.map.java.opts</name> >>>>>>>> <value>-Xmx4096m</value> >>>>>>>> </property> >>>>>>>> <property> >>>>>>>> <name>mapreduce.reduce.memory.mb</name> >>>>>>>> <value>10240</value> >>>>>>>> </property> >>>>>>>> <property> >>>>>>>> <name>mapreduce.reduce.java.opts</name> >>>>>>>> <value>-Xmx8192m</value> >>>>>>>> </property> >>>>>>>> <property> >>>>>>>> <name>yarn.app.mapreduce.am.resource.mb</name> >>>>>>>> <value>5120</value> >>>>>>>> </property> >>>>>>>> <property> >>>>>>>> <name>yarn.app.mapreduce.am.command-opts</name> >>>>>>>> <value>-Xmx4096m</value> >>>>>>>> </property> >>>>>>>> <property> >>>>>>>> <name>mapreduce.task.io.sort.mb</name> >>>>>>>> <value>1024</value> >>>>>>>> </property> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> <property> >>>>>>>> <name>yarn.scheduler.minimum-allocation-mb</name> >>>>>>>> <value>10240</value> >>>>>>>> </property> >>>>>>>> >>>>>>>> <property> >>>>>>>> <name>yarn.scheduler.maximum-allocation-mb</name> >>>>>>>> <value>61440</value> >>>>>>>> </property> >>>>>>>> >>>>>>>> <property> >>>>>>>> <name>yarn.nodemanager.resource.memory-mb</name> >>>>>>>> <value>61440</value> >>>>>>>> </property> >>>>>>>> >>>>>>>> >>>>>>>> Ram >>>>>>>> >>>>>>>> On Thu, Aug 18, 2016 at 11:14 PM, tkg_cangkul < >>>>>>>> yuza.ras...@gmail.com> wrote: >>>>>>>> >>>>>>>>> maybe this link can be some reference to tune up the cluster: >>>>>>>>> >>>>>>>>> http://jason4zhu.blogspot.co.id/2014/10/memory-configuration >>>>>>>>> -in-hadoop.html >>>>>>>>> >>>>>>>>> >>>>>>>>> On 19/08/16 11:13, rammohan ganapavarapu wrote: >>>>>>>>> >>>>>>>>> Do you know what properties to tune? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ram >>>>>>>>> >>>>>>>>> On Thu, Aug 18, 2016 at 9:11 PM, tkg_cangkul < >>>>>>>>> yuza.ras...@gmail.com> wrote: >>>>>>>>> >>>>>>>>>> i think that's because you don't have enough resource. u can >>>>>>>>>> tune your cluster config to maximize your resource. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 19/08/16 11:03, rammohan ganapavarapu wrote: >>>>>>>>>> >>>>>>>>>> I dont see any thing odd except this not sure if i have to worry >>>>>>>>>> about it or not. >>>>>>>>>> >>>>>>>>>> 2016-08-19 03:29:26,621 INFO [main] >>>>>>>>>> org.apache.hadoop.yarn.client.RMProxy: >>>>>>>>>> Connecting to ResourceManager at /0.0.0.0:8030 >>>>>>>>>> 2016-08-19 03:29:27,646 INFO [main] org.apache.hadoop.ipc.Client: >>>>>>>>>> Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried >>>>>>>>>> 0 time(s); retry policy is RetryUpToMaximumCo >>>>>>>>>> untWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS) >>>>>>>>>> 2016-08-19 03:29:28,647 INFO [main] org.apache.hadoop.ipc.Client: >>>>>>>>>> Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried >>>>>>>>>> 1 time(s); retry policy is >>>>>>>>>> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, >>>>>>>>>> sleepTime=1000 MILLISECONDS) >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> its keep printing this log ..in app container logs. >>>>>>>>>> >>>>>>>>>> On Thu, Aug 18, 2016 at 8:20 PM, tkg_cangkul < >>>>>>>>>> yuza.ras...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> maybe u can check the logs from port 8088 on your browser. that >>>>>>>>>>> was RM UI. just choose your job id and then check the logs. >>>>>>>>>>> >>>>>>>>>>> On 19/08/16 10:14, rammohan ganapavarapu wrote: >>>>>>>>>>> >>>>>>>>>>> Sunil, >>>>>>>>>>> >>>>>>>>>>> Thanks you for your input, below are my server metrics for RM. >>>>>>>>>>> Also attached RM UI for capacity scheduler resources. How else i >>>>>>>>>>> can find? >>>>>>>>>>> >>>>>>>>>>> { >>>>>>>>>>> "name": "Hadoop:service=ResourceManage >>>>>>>>>>> r,name=QueueMetrics,q0=root", >>>>>>>>>>> "modelerType": "QueueMetrics,q0=root", >>>>>>>>>>> "tag.Queue": "root", >>>>>>>>>>> "tag.Context": "yarn", >>>>>>>>>>> "tag.Hostname": "hadoop001", >>>>>>>>>>> "running_0": 0, >>>>>>>>>>> "running_60": 0, >>>>>>>>>>> "running_300": 0, >>>>>>>>>>> "running_1440": 0, >>>>>>>>>>> "AppsSubmitted": 1, >>>>>>>>>>> "AppsRunning": 0, >>>>>>>>>>> "AppsPending": 0, >>>>>>>>>>> "AppsCompleted": 0, >>>>>>>>>>> "AppsKilled": 0, >>>>>>>>>>> "AppsFailed": 1, >>>>>>>>>>> "AllocatedMB": 0, >>>>>>>>>>> "AllocatedVCores": 0, >>>>>>>>>>> "AllocatedContainers": 0, >>>>>>>>>>> "AggregateContainersAllocated": 2, >>>>>>>>>>> "AggregateContainersReleased": 2, >>>>>>>>>>> "AvailableMB": 64512, >>>>>>>>>>> "AvailableVCores": 24, >>>>>>>>>>> "PendingMB": 0, >>>>>>>>>>> "PendingVCores": 0, >>>>>>>>>>> "PendingContainers": 0, >>>>>>>>>>> "ReservedMB": 0, >>>>>>>>>>> "ReservedVCores": 0, >>>>>>>>>>> "ReservedContainers": 0, >>>>>>>>>>> "ActiveUsers": 0, >>>>>>>>>>> "ActiveApplications": 0 >>>>>>>>>>> }, >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 18, 2016 at 6:49 PM, Sunil Govind < >>>>>>>>>>> sunil.gov...@gmail.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi >>>>>>>>>>>> >>>>>>>>>>>> It could be because of many of reasons. Also I am not sure >>>>>>>>>>>> about which scheduler your are using, pls share more details such >>>>>>>>>>>> as RM log >>>>>>>>>>>> etc. >>>>>>>>>>>> >>>>>>>>>>>> I could point out few reasons >>>>>>>>>>>> - Such as "Not enough resource is cluster" can cause this >>>>>>>>>>>> - If using Capacity Scheduler, if queue capacity is maxed out, >>>>>>>>>>>> such case can happen. >>>>>>>>>>>> - Similarly if max-am-resource-percent is crossed per queue >>>>>>>>>>>> level, then also AM container may not be launched. >>>>>>>>>>>> >>>>>>>>>>>> you could check RM log to get more information if AM container >>>>>>>>>>>> is laucnhed. >>>>>>>>>>>> >>>>>>>>>>>> Thanks >>>>>>>>>>>> Sunil >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 19, 2016 at 5:37 AM rammohan ganapavarapu < >>>>>>>>>>>> rammohanga...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, >>>>>>>>>>>>> >>>>>>>>>>>>> When i submit a MR job, i am getting this from AM UI but it >>>>>>>>>>>>> never get finished, what am i missing ? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> Ram >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> --------------------------------------------------------------------- >>>>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@hadoop.apache.org >>>>>>>>>>> For additional commands, e-mail: user-h...@hadoop.apache.org >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>
