Here is slider.log for Slider AM. Note the port range does not always fails
- it does work occassionally !
"appConf" :{
"schema" : "http://example.org/specification/v2.0.0";,
"metadata" : { },
"global" : {
"site.global.qaprod___gms.tenant" : "qaprod___gms",
"site.global.memory_val" : "200m",
"site.global.service_mgmt_log_dir" : "/data/processlauncher/log",
"site.global.xms_val" : "128m",
"site.global.slider.allowed.ports" : "32201–33100",
"java_home" : "/usr/java/latest",
"site.global.xmx_val" : "256m",
"zookeeper.quorum" : "host1:2181,host2:2181,host3:2181",
"site.global.qaprod___gms.principal" : "qaprod.gms",
"env.MALLOC_ARENA_MAX" : "4",
"site.global.qaprod___gms.http_port" :
"${qaprod___gms.ALLOCATED_PORT}{PER_CONTAINER}",
"site.global.bas_host" : "host4",
"site.global.app_version" : "1.0.0",
"site.global.bas_port" : "9009",
"application.def" : ".slider/package/spas/spas-1.0.0.zip",
"site.global.qaprod___gms.keytabfile" :
"/etc/hadoop/conf/qaprod.gms.keytab",
"zookeeper.hosts" : "host1,host2,host3",
"zookeeper.path" : "/services/slider/users/slideradmin/spas",
"site.global.additional_cp" : "/usr/lib/hadoop/lib/*",
"site.global.qaprod___gms.https_port" :
"${qaprod___gms.ALLOCATED_PORT}{PER_CONTAINER}",
"site.fs.defaultFS" : "hdfs://bdsqaprod2",
"site.dfs.namenode.kerberos.principal" : "hdfs/_HOST@BIGDATA",
"site.fs.default.name" : "hdfs://bdsqaprod2"
},
"credentials" : { },
"components" : {
"slider-appmaster" : {
"jvm.heapsize" : "256M",
"slider.keytab.principal.name" : "slideradmin",
"slider.am.keytab.local.path" : "/etc/hadoop/conf/slideradmin.keytab"
},
"qaprod___gms" : { }
}
}}:
2016-05-16 19:24:51,456 [main] INFO appmaster.SliderAppMaster - Conf dir
/hadoop/disk4/yarn/local/usercache/slideradmin/appcache/application_1462834537586_7708/container_e15_1462834537586_7708_09_000001/propagatedconf
does not exist.
2016-05-16 19:24:51,456 [main] INFO appmaster.SliderAppMaster - Parent dir
/hadoop/disk4/yarn/local/usercache/slideradmin/appcache/application_1462834537586_7708/container_e15_1462834537586_7708_09_000001:
tmp
launch_container.sh
container_tokens
lib
confdir
expandedarchive
2016-05-16 19:24:51,457 [main] INFO appmaster.SliderAppMaster - Cluster
provider type is agent
2016-05-16 19:24:51,500 [main] INFO appmaster.SliderAppMaster - RM is at
host9/10.52.88.218:23130
2016-05-16 19:24:51,553 [main] INFO appmaster.SliderAppMaster - AM for ID
7708
2016-05-16 19:24:51,596 [main] INFO impl.NMClientAsyncImpl - Upper bound
of the thread pool size is 500
2016-05-16 19:24:51,597 [main] INFO impl.ContainerManagementProtocolProxy
- yarn.client.max-cached-nodemanagers-proxies : 0
2016-05-16 19:24:51,654 [main] ERROR main.ServiceLauncher - No available
ports found in configured range {}
2016-05-16 19:24:51,656 [main] INFO util.ExitUtil - Exiting with status 77
2016-05-16 19:24:51,658 [Thread-1] INFO appmaster.SliderAppMaster -
Process has exited with exit code 0 mapped to 0 -ignoring
2016-05-16 19:24:51,659 [AMRM Callback Handler Thread] INFO
impl.AMRMClientAsyncImpl - Interrupted while waiting for queue
java.lang.InterruptedException
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:274)
On Mon, May 16, 2016 at 6:07 PM, Billie Rinaldi <[email protected]>
wrote:
> Manoj,
>
> Attachments don't come through on ASF mailing lists. What value are you
> using for site.global.slider.allowed.ports?
>
> Billie
>
> On Mon, May 16, 2016 at 4:20 PM, Manoj Samel <[email protected]>
> wrote:
>
> > Hi Tim,
> >
> > Thanks for your reply
> >
> > In my case, the slider AM itself fails to start. The app remains in
> > accepted state and the container_001 log (the slider AM) errors. I have
> > attached the container_001 slider.log below. It has all the configs,
> > including app config (host names and some other info is altered)
> >
> > The specified port range is not being used on cluster. Also, the error is
> > intermittent i.e. it works for a port range some times and does not works
> > for same for other times. It I removed the allocated_port, it works fine.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Mon, May 16, 2016 at 2:41 PM, Tim I <[email protected]> wrote:
> >
> >> I'm pretty sure the PER_CONTAINER statement works as follows for a each
> >> instantiation of a particular service within a slider app:
> >> 1) check the allowable port range
> >> 2) look for an available port on the machine within the allowed port
> range
> >> 3) launch with selected port
> >>
> >> In the case of Accumulo, if you're running two tservers on the same box
> >> and
> >> have a range of ports that they can use, they will not conflict.
> However,
> >> if you statically set the tserver's port and two launch on the same
> >> machine, they will both try to use it and conflict. One will eventually
> >> relaunch on a node with an available port provided resources exist and
> you
> >> don't exceed your failure threshold.
> >>
> >> If you are having an issue with your app master not launching, it might
> be
> >> because you specified a port or port range that is fully utilized by
> >> another application on the same box.
> >>
> >> Is your cluster idle or do you have other slider apps running? Do you
> >> have
> >> more complete output of the logs and possibly the appConfig that you can
> >> share? Are you sure it's the AM failing to start and not a service
> within
> >> your slider app?
> >>
> >> Tim
> >> On May 16, 2016 4:02 PM, "Manoj Samel" <[email protected]>
> wrote:
> >>
> >> Hello,
> >>
> >> When using ALLOCATED_PORT clause, there is a option "PER_CONTAINER".
> >>
> >> Can someone explain what does "PER_CONTAINER" option does ? It says keep
> >> port allocation private to container. What does that means ? If multiple
> >> containers are chosen to on same host machine, will this cause issue ?
> >>
> >> When using a specific port range using
> site.global.slider.allowed.ports, I
> >> am getting frequent errors in starting slider AM. The log says
> >> *2016-05-14 20:39:29,236 [main] ERROR main.ServiceLauncher - No
> available
> >> ports found in configured range {}*
> >> 2016-05-14 20:39:29,237 [main] INFO util.ExitUtil - Exiting with status
> >> 77
> >>
> >> The error is not hard, sometimes same port range works file. I am
> >> wondering
> >> if this has anything to do with PER_CONTAINER ...
> >>
> >> This is on slider 0.80 on Hadoop 2.6 secured cluster ...
> >>
> >> Than
> >>
> >
> >
>
