Re: Ports required for running spark

[Andrew Ash]

Also Konstantin do you have a firewall between your Spark services?  If
that's what's causing these issues, then you may be interested in the
ability to configure every port a Spark service listens on --
https://issues.apache.org/jira/browse/SPARK-2157


On Thu, Jul 31, 2014 at 8:47 AM, Haiyang Fu  wrote:

> Hi Konstantin,
>
> Could you please post your first container's stderr log here which is
> always the AM log?As far as I know, ports except 8020,8080,8081,50070,50071
> are all random socket ports determined by each job. So 33007   maybe just
> a temporary port for data transferation. The deeper reason for 33007
> ConnectionRefused  maybe something else.would you please check If it is
> always 33007 when you submit your job again ?
>
>
> haiy
>
>
> On Thu, Jul 31, 2014 at 8:17 PM, Konstantin Kudryavtsev <
> kudryavtsev.konstan...@gmail.com> wrote:
>
>> Hi Haiyang,
>>
>> you are right, YARN takes over the resource management, bot I constantly
>> got Exception ConnectionRefused on mentioned port. So, I suppose some spark
>> internal communications are done via this port... but I don't know what
>> exactly and how can I change it...
>>
>> Thank you,
>> Konstantin Kudryavtsev
>>
>>
>> On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu 
>> wrote:
>>
>>> Hi Konstantin,
>>> Would you please post some more details? Error info or exception from
>>> the log on what situation？when you run spark job on yarn cluster mode ,yarn
>>> will take over all the resource management.
>>>
>>>
>>> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
>>> kudryavtsev.konstan...@gmail.com> wrote:
>>>
 Hi Larry,

 I'm afraid this is standalone mode, I'm interesting in YARN

 Also, I don't see port-in-trouble 33007   which i believe related to
 Akka

 Thank you,
 Konstantin Kudryavtsev


 On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao  wrote:

>  Hi Konstantin,
>
> I think you can find it at
> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
> and you can specify port for master or worker at conf/spark-env.sh
>
> Larry
>
>
> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>
> Hi there,
>
>  I'm trying to run Spark on YARN cluster and face with issued that
> some ports are closed, particularly port 33007 (I suppose it's used
> by Akka)
>
>  Could you please provide me with a list of all ports required for
> Spark?
> Also, is it possible to set up these ports?
>
> Thank you in advance,
> Konstantin Kudryavtsev
>
>
>

>>>
>>
>

Re: Ports required for running spark

[Haiyang Fu]

Hi Konstantin,

Could you please post your first container's stderr log here which is
always the AM log?As far as I know, ports except 8020,8080,8081,50070,50071
are all random socket ports determined by each job. So 33007   maybe just a
temporary port for data transferation. The deeper reason for 33007
ConnectionRefused  maybe something else.would you please check If it is
always 33007 when you submit your job again ?


haiy


On Thu, Jul 31, 2014 at 8:17 PM, Konstantin Kudryavtsev <
kudryavtsev.konstan...@gmail.com> wrote:

> Hi Haiyang,
>
> you are right, YARN takes over the resource management, bot I constantly
> got Exception ConnectionRefused on mentioned port. So, I suppose some spark
> internal communications are done via this port... but I don't know what
> exactly and how can I change it...
>
> Thank you,
> Konstantin Kudryavtsev
>
>
> On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu 
> wrote:
>
>> Hi Konstantin,
>> Would you please post some more details? Error info or exception from the
>> log on what situation？when you run spark job on yarn cluster mode ,yarn
>> will take over all the resource management.
>>
>>
>> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
>> kudryavtsev.konstan...@gmail.com> wrote:
>>
>>> Hi Larry,
>>>
>>> I'm afraid this is standalone mode, I'm interesting in YARN
>>>
>>> Also, I don't see port-in-trouble 33007   which i believe related to
>>> Akka
>>>
>>> Thank you,
>>> Konstantin Kudryavtsev
>>>
>>>
>>> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao  wrote:
>>>
  Hi Konstantin,

 I think you can find it at
 https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
 and you can specify port for master or worker at conf/spark-env.sh

 Larry


 On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:

 Hi there,

  I'm trying to run Spark on YARN cluster and face with issued that
 some ports are closed, particularly port 33007 (I suppose it's used by
 Akka)

  Could you please provide me with a list of all ports required for
 Spark?
 Also, is it possible to set up these ports?

 Thank you in advance,
 Konstantin Kudryavtsev



>>>
>>
>

Re: Ports required for running spark

[Konstantin Kudryavtsev]

Hi Haiyang,

you are right, YARN takes over the resource management, bot I constantly
got Exception ConnectionRefused on mentioned port. So, I suppose some spark
internal communications are done via this port... but I don't know what
exactly and how can I change it...

Thank you,
Konstantin Kudryavtsev


On Thu, Jul 31, 2014 at 2:53 PM, Haiyang Fu  wrote:

> Hi Konstantin,
> Would you please post some more details? Error info or exception from the
> log on what situation？when you run spark job on yarn cluster mode ,yarn
> will take over all the resource management.
>
>
> On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
> kudryavtsev.konstan...@gmail.com> wrote:
>
>> Hi Larry,
>>
>> I'm afraid this is standalone mode, I'm interesting in YARN
>>
>> Also, I don't see port-in-trouble 33007   which i believe related to Akka
>>
>> Thank you,
>> Konstantin Kudryavtsev
>>
>>
>> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao  wrote:
>>
>>>  Hi Konstantin,
>>>
>>> I think you can find it at
>>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
>>> and you can specify port for master or worker at conf/spark-env.sh
>>>
>>> Larry
>>>
>>>
>>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>>>
>>> Hi there,
>>>
>>>  I'm trying to run Spark on YARN cluster and face with issued that some
>>> ports are closed, particularly port 33007 (I suppose it's used by Akka)
>>>
>>>  Could you please provide me with a list of all ports required for
>>> Spark?
>>> Also, is it possible to set up these ports?
>>>
>>> Thank you in advance,
>>> Konstantin Kudryavtsev
>>>
>>>
>>>
>>
>

Re: Ports required for running spark

[Larry Xiao]

Sorry, I don't have experience with YARN.

I checked the YARN page 
http://spark.apache.org/docs/latest/running-on-yarn.html
And for configuration, it refers to 
http://spark.apache.org/docs/latest/configuration.html " Most of the 
configs are the same for Spark on YARN as for other deployment modes. 
See theconfiguration page 
for more 
information on those. These are configs that are specific to Spark on YARN."

And in Networking, there's spark.driver.port.

Don't know if it's helpful.

Larry

On 7/31/14, 6:17 PM, Konstantin Kudryavtsev wrote:

Hi Larry,

I'm afraid this is standalone mode, I'm interesting in YARN

Also, I don't see port-in-trouble 33007which i believe related to Akka

Thank you,
Konstantin Kudryavtsev


On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao > wrote:


Hi Konstantin,

I think you can find it at

https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
and you can specify port for master or worker at conf/spark-env.sh

Larry


On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:

Hi there,

I'm trying to run Spark on YARN cluster and face with issued that
some ports are closed, particularly port 33007 (I suppose it's
used by Akka)

Could you please provide me with a list of all ports required for
Spark?
Also, is it possible to set up these ports?

Thank you in advance,
Konstantin Kudryavtsev

Re: Ports required for running spark

[Haiyang Fu]

Hi Konstantin,
Would you please post some more details? Error info or exception from the
log on what situation？when you run spark job on yarn cluster mode ,yarn
will take over all the resource management.


On Thu, Jul 31, 2014 at 6:17 PM, Konstantin Kudryavtsev <
kudryavtsev.konstan...@gmail.com> wrote:

> Hi Larry,
>
> I'm afraid this is standalone mode, I'm interesting in YARN
>
> Also, I don't see port-in-trouble 33007   which i believe related to Akka
>
> Thank you,
> Konstantin Kudryavtsev
>
>
> On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao  wrote:
>
>>  Hi Konstantin,
>>
>> I think you can find it at
>> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
>> and you can specify port for master or worker at conf/spark-env.sh
>>
>> Larry
>>
>>
>> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>>
>> Hi there,
>>
>>  I'm trying to run Spark on YARN cluster and face with issued that some
>> ports are closed, particularly port 33007 (I suppose it's used by Akka)
>>
>>  Could you please provide me with a list of all ports required for
>> Spark?
>> Also, is it possible to set up these ports?
>>
>> Thank you in advance,
>> Konstantin Kudryavtsev
>>
>>
>>
>

Re: Ports required for running spark

[Konstantin Kudryavtsev]

Hi Larry,

I'm afraid this is standalone mode, I'm interesting in YARN

Also, I don't see port-in-trouble 33007   which i believe related to Akka

Thank you,
Konstantin Kudryavtsev


On Thu, Jul 31, 2014 at 1:11 PM, Larry Xiao  wrote:

>  Hi Konstantin,
>
> I think you can find it at
> https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security
> and you can specify port for master or worker at conf/spark-env.sh
>
> Larry
>
>
> On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:
>
> Hi there,
>
>  I'm trying to run Spark on YARN cluster and face with issued that some
> ports are closed, particularly port 33007 (I suppose it's used by Akka)
>
>  Could you please provide me with a list of all ports required for Spark?
> Also, is it possible to set up these ports?
>
> Thank you in advance,
> Konstantin Kudryavtsev
>
>
>

Re: Ports required for running spark

[Larry Xiao]

Hi Konstantin,

I think you can find it at 
https://spark.apache.org/docs/latest/spark-standalone.html#configuring-ports-for-network-security

and you can specify port for master or worker at conf/spark-env.sh

Larry

On 7/31/14, 6:04 PM, Konstantin Kudryavtsev wrote:

Hi there,

I'm trying to run Spark on YARN cluster and face with issued that some 
ports are closed, particularly port 33007 (I suppose it's used by Akka)


Could you please provide me with a list of all ports required for Spark?
Also, is it possible to set up these ports?

Thank you in advance,
Konstantin Kudryavtsev

Ports required for running spark

[Konstantin Kudryavtsev]

Hi there,

I'm trying to run Spark on YARN cluster and face with issued that some
ports are closed, particularly port 33007 (I suppose it's used by Akka)

Could you please provide me with a list of all ports required for Spark?
Also, is it possible to set up these ports?

Thank you in advance,
Konstantin Kudryavtsev