Hi Konstantin, Could you please post your first container's stderr log here which is always the AM log?As far as I know, ports except 8020,8080,8081,50070,50071 are all random socket ports determined by each job. So 33007 maybe just a temporary port for data transferation. The deeper reason for 33007 ConnectionRefused maybe something else.would you please check If it is always 33007 when you submit your job again ?
haiy On Thu, Jul 31, 2014 at 8:17 PM, Konstantin Kudryavtsev < kudryavtsev.konstan...@gmail.com> wrote: > Hi Haiyang, > > you are right, YARN takes over the resource management, bot I constantly > got Exception ConnectionRefused on mentioned port. So, I suppose some spark > internal communications are done via this port... but I don't know what > exactly and how can I change it... > > Thank you, > Konstantin Kudryavtsev > > > On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu <haiyangfu...@gmail.com> > wrote: > >> Hi Konstantin, >> Would you please post some more details? Error info or exception from the >> log on what situation？when you run spark job on yarn cluster mode ,yarn >> will take over all the resource management. >> >> >> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev < >> kudryavtsev.konstan...@gmail.com> wrote: >> >>> Hi Larry, >>> >>> I'm afraid this is standalone mode, I'm interesting in YARN >>> >>> Also, I don't see port-in-trouble 33007 which i believe related to >>> Akka >>> >>> Thank you, >>> Konstantin Kudryavtsev >>> >>> >>> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao <xia...@sjtu.edu.cn> wrote: >>> >>>> Hi Konstantin, >>>> >>>> I think you can find it at >>>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security >>>> and you can specify port for master or worker at conf/spark-env.sh >>>> >>>> Larry >>>> >>>> >>>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote: >>>> >>>> Hi there, >>>> >>>> I'm trying to run Spark on YARN cluster and face with issued that >>>> some ports are closed, particularly port 33007 (I suppose it's used by >>>> Akka) >>>> >>>> Could you please provide me with a list of all ports required for >>>> Spark? >>>> Also, is it possible to set up these ports? >>>> >>>> Thank you in advance, >>>> Konstantin Kudryavtsev >>>> >>>> >>>> >>> >> >
