Re: spark-1.2.0--standalone-ha-zookeeper

[Paul Leclercq]

Hi Raghvendra and Spark users,

I also have trouble activating my stand by master when my first master is
shutdown (via a ./sbin/stop-master.sh or via a instance shut down) and just
want to share with you my thoughts.

To answer your question Raghvendra, in *spark-env.sh*, if 2 IPs are set
for SPARK_MASTER_IP(SPARK_MASTER_IP='W.X.Y.Z,A.B.C.D'), the standalone
cluster cannot be launched.

So I only use only one IP there, as the Spark context can know other
masters with a other way, as written in the Standalone Zookeeper HA

doc, "you might start your SparkContext pointing to
spark://host1:port1,host2:port2"

In my opinion, we should not have to set a SPARK_MASTER_IP as this is
stored in ZooKeeper :

you can launch multiple Masters in your cluster connected to the same
> ZooKeeper instance. One will be elected “leader” and the others will remain
> in standby mode.

When starting up, an application or Worker needs to be able to find and
> register with the current lead Master. Once it successfully registers,
> though, it is “in the system” (i.e., stored in ZooKeeper).

 -
http://spark.apache.org/docs/latest/spark-standalone.html#standby-masters-with-zookeeper

As I understand it, after a ./sbin/stop-master.sh on both master, a master
will be elected, and the other will be stand by.
To launch the workers, we can use ./sbin/start-slave.sh
spark://MASTER_ELECTED_IP:7077
I don't think if we can use the ./sbin/start-all.sh that use the salve file
to launch workers and masters as we cannot set 2 master IPs inside
spark-env.sh

My SPARK_DAEMON_JAVA_OPTS content :

SPARK_DAEMON_JAVA_OPTS='-Dspark.deploy.recoveryMode="ZOOKEEPER"
> -Dspark.deploy.zookeeper.url="ZOOKEEPER_IP:2181"
> -Dspark.deploy.zookeeper.dir="/spark"'


A good thing to check if everything went OK is the folder /spark on the
ZooKeeper server. I could not find it on my server.

Thanks for reading,

Paul


2016-01-19 22:12 GMT+01:00 Raghvendra Singh :

> Hi, there is one question. In spark-env.sh should i specify all masters
> for parameter SPARK_MASTER_IP. I've set SPARK_DAEMON_JAVA_OPTS already
> with zookeeper configuration as specified in spark documentation.
>
> Thanks & Regards
> Raghvendra
>
> On Wed, Jan 20, 2016 at 1:46 AM, Raghvendra Singh <
> raghvendra.ii...@gmail.com> wrote:
>
>> Here's the complete master log on reproducing the error
>> http://pastebin.com/2YJpyBiF
>>
>> Regards
>> Raghvendra
>>
>> On Wed, Jan 20, 2016 at 12:38 AM, Raghvendra Singh <
>> raghvendra.ii...@gmail.com> wrote:
>>
>>> Ok I Will try to reproduce the problem. Also I don't think this is an
>>> uncommon problem I am searching for this problem on Google for many days
>>> and found lots of questions but no answers.
>>>
>>> Do you know what kinds of settings spark and zookeeper allow for
>>> handling time outs during leader election etc. When one is down.
>>>
>>> Regards
>>> Raghvendra
>>> On 20-Jan-2016 12:28 am, "Ted Yu"  wrote:
>>>
 Perhaps I don't have enough information to make further progress.

 On Tue, Jan 19, 2016 at 10:55 AM, Raghvendra Singh <
 raghvendra.ii...@gmail.com> wrote:

> I currently do not have access to those logs but there were only about
> five lines before this error. They were the same which are present usually
> when everything works fine.
>
> Can you still help?
>
> Regards
> Raghvendra
> On 18-Jan-2016 8:50 pm, "Ted Yu"  wrote:
>
>> Can you pastebin master log before the error showed up ?
>>
>> The initial message was posted for Spark 1.2.0
>> Which release of Spark / zookeeper do you use ?
>>
>> Thanks
>>
>> On Mon, Jan 18, 2016 at 6:47 AM, doctorx 
>> wrote:
>>
>>> Hi,
>>> I am facing the same issue, with the given error
>>>
>>> ERROR Master:75 - Leadership has been revoked -- master shutting
>>> down.
>>>
>>> Can anybody help. Any clue will be useful. Should i change something
>>> in
>>> spark cluster or zookeeper. Is there any setting in spark which can
>>> help me?
>>>
>>> Thanks & Regards
>>> Raghvendra
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-0-standalone-ha-zookeeper-tp21308p25994.html
>>> Sent from the Apache Spark User List mailing list archive at
>>> Nabble.com.
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>>

>>
>

Re: spark-1.2.0--standalone-ha-zookeeper

[Raghvendra Singh]

Thanks Paul,
So your reply prevented me from looking in the wrong direction, but I an
back to my original problem with zookeeper.
"Leadership had been revoked master shutting down"

Can anyone provide some feedback or add to this.
Regards
Raghvendra
On 20-Jan-2016 2:31 pm, "Paul Leclercq"  wrote:

> Hi Raghvendra and Spark users,
>
> I also have trouble activating my stand by master when my first master is
> shutdown (via a ./sbin/stop-master.sh or via a instance shut down) and
> just want to share with you my thoughts.
>
> To answer your question Raghvendra, in *spark-env.sh*, if 2 IPs are set
> for SPARK_MASTER_IP(SPARK_MASTER_IP='W.X.Y.Z,A.B.C.D'), the standalone
> cluster cannot be launched.
>
> So I only use only one IP there, as the Spark context can know other
> masters with a other way, as written in the Standalone Zookeeper HA
> 
> doc, "you might start your SparkContext pointing to
> spark://host1:port1,host2:port2"
>
> In my opinion, we should not have to set a SPARK_MASTER_IP as this is
> stored in ZooKeeper :
>
> you can launch multiple Masters in your cluster connected to the same
>> ZooKeeper instance. One will be elected “leader” and the others will remain
>> in standby mode.
>
> When starting up, an application or Worker needs to be able to find and
>> register with the current lead Master. Once it successfully registers,
>> though, it is “in the system” (i.e., stored in ZooKeeper).
>
>  -
> http://spark.apache.org/docs/latest/spark-standalone.html#standby-masters-with-zookeeper
>
> As I understand it, after a ./sbin/stop-master.sh on both master, a
> master will be elected, and the other will be stand by.
> To launch the workers, we can use ./sbin/start-slave.sh
> spark://MASTER_ELECTED_IP:7077
> I don't think if we can use the ./sbin/start-all.sh that use the salve
> file to launch workers and masters as we cannot set 2 master IPs inside
> spark-env.sh
>
> My SPARK_DAEMON_JAVA_OPTS content :
>
> SPARK_DAEMON_JAVA_OPTS='-Dspark.deploy.recoveryMode="ZOOKEEPER"
>> -Dspark.deploy.zookeeper.url="ZOOKEEPER_IP:2181"
>> -Dspark.deploy.zookeeper.dir="/spark"'
>
>
> A good thing to check if everything went OK is the folder /spark on the
> ZooKeeper server. I could not find it on my server.
>
> Thanks for reading,
>
> Paul
>
>
> 2016-01-19 22:12 GMT+01:00 Raghvendra Singh :
>
>> Hi, there is one question. In spark-env.sh should i specify all masters
>> for parameter SPARK_MASTER_IP. I've set SPARK_DAEMON_JAVA_OPTS already
>> with zookeeper configuration as specified in spark documentation.
>>
>> Thanks & Regards
>> Raghvendra
>>
>> On Wed, Jan 20, 2016 at 1:46 AM, Raghvendra Singh <
>> raghvendra.ii...@gmail.com> wrote:
>>
>>> Here's the complete master log on reproducing the error
>>> http://pastebin.com/2YJpyBiF
>>>
>>> Regards
>>> Raghvendra
>>>
>>> On Wed, Jan 20, 2016 at 12:38 AM, Raghvendra Singh <
>>> raghvendra.ii...@gmail.com> wrote:
>>>
 Ok I Will try to reproduce the problem. Also I don't think this is an
 uncommon problem I am searching for this problem on Google for many days
 and found lots of questions but no answers.

 Do you know what kinds of settings spark and zookeeper allow for
 handling time outs during leader election etc. When one is down.

 Regards
 Raghvendra
 On 20-Jan-2016 12:28 am, "Ted Yu"  wrote:

> Perhaps I don't have enough information to make further progress.
>
> On Tue, Jan 19, 2016 at 10:55 AM, Raghvendra Singh <
> raghvendra.ii...@gmail.com> wrote:
>
>> I currently do not have access to those logs but there were only
>> about five lines before this error. They were the same which are present
>> usually when everything works fine.
>>
>> Can you still help?
>>
>> Regards
>> Raghvendra
>> On 18-Jan-2016 8:50 pm, "Ted Yu"  wrote:
>>
>>> Can you pastebin master log before the error showed up ?
>>>
>>> The initial message was posted for Spark 1.2.0
>>> Which release of Spark / zookeeper do you use ?
>>>
>>> Thanks
>>>
>>> On Mon, Jan 18, 2016 at 6:47 AM, doctorx >> > wrote:
>>>
 Hi,
 I am facing the same issue, with the given error

 ERROR Master:75 - Leadership has been revoked -- master shutting
 down.

 Can anybody help. Any clue will be useful. Should i change
 something in
 spark cluster or zookeeper. Is there any setting in spark which can
 help me?

 Thanks & Regards
 Raghvendra



 --
 View this message in context:
 http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-0-standalone-ha-zookeeper-tp21308p25994.html
 Sent from the Apache Spark 

Re: spark-1.2.0--standalone-ha-zookeeper

[Raghvendra Singh]

Hi, there is one question. In spark-env.sh should i specify all masters for
parameter SPARK_MASTER_IP. I've set SPARK_DAEMON_JAVA_OPTS already with
zookeeper configuration as specified in spark documentation.

Thanks & Regards
Raghvendra

On Wed, Jan 20, 2016 at 1:46 AM, Raghvendra Singh <
raghvendra.ii...@gmail.com> wrote:

> Here's the complete master log on reproducing the error
> http://pastebin.com/2YJpyBiF
>
> Regards
> Raghvendra
>
> On Wed, Jan 20, 2016 at 12:38 AM, Raghvendra Singh <
> raghvendra.ii...@gmail.com> wrote:
>
>> Ok I Will try to reproduce the problem. Also I don't think this is an
>> uncommon problem I am searching for this problem on Google for many days
>> and found lots of questions but no answers.
>>
>> Do you know what kinds of settings spark and zookeeper allow for handling
>> time outs during leader election etc. When one is down.
>>
>> Regards
>> Raghvendra
>> On 20-Jan-2016 12:28 am, "Ted Yu"  wrote:
>>
>>> Perhaps I don't have enough information to make further progress.
>>>
>>> On Tue, Jan 19, 2016 at 10:55 AM, Raghvendra Singh <
>>> raghvendra.ii...@gmail.com> wrote:
>>>
 I currently do not have access to those logs but there were only about
 five lines before this error. They were the same which are present usually
 when everything works fine.

 Can you still help?

 Regards
 Raghvendra
 On 18-Jan-2016 8:50 pm, "Ted Yu"  wrote:

> Can you pastebin master log before the error showed up ?
>
> The initial message was posted for Spark 1.2.0
> Which release of Spark / zookeeper do you use ?
>
> Thanks
>
> On Mon, Jan 18, 2016 at 6:47 AM, doctorx 
> wrote:
>
>> Hi,
>> I am facing the same issue, with the given error
>>
>> ERROR Master:75 - Leadership has been revoked -- master shutting down.
>>
>> Can anybody help. Any clue will be useful. Should i change something
>> in
>> spark cluster or zookeeper. Is there any setting in spark which can
>> help me?
>>
>> Thanks & Regards
>> Raghvendra
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-0-standalone-ha-zookeeper-tp21308p25994.html
>> Sent from the Apache Spark User List mailing list archive at
>> Nabble.com.
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>
>>>
>

Re: spark-1.2.0--standalone-ha-zookeeper

[Raghvendra Singh]

Here's the complete master log on reproducing the error
http://pastebin.com/2YJpyBiF

Regards
Raghvendra

On Wed, Jan 20, 2016 at 12:38 AM, Raghvendra Singh <
raghvendra.ii...@gmail.com> wrote:

> Ok I Will try to reproduce the problem. Also I don't think this is an
> uncommon problem I am searching for this problem on Google for many days
> and found lots of questions but no answers.
>
> Do you know what kinds of settings spark and zookeeper allow for handling
> time outs during leader election etc. When one is down.
>
> Regards
> Raghvendra
> On 20-Jan-2016 12:28 am, "Ted Yu"  wrote:
>
>> Perhaps I don't have enough information to make further progress.
>>
>> On Tue, Jan 19, 2016 at 10:55 AM, Raghvendra Singh <
>> raghvendra.ii...@gmail.com> wrote:
>>
>>> I currently do not have access to those logs but there were only about
>>> five lines before this error. They were the same which are present usually
>>> when everything works fine.
>>>
>>> Can you still help?
>>>
>>> Regards
>>> Raghvendra
>>> On 18-Jan-2016 8:50 pm, "Ted Yu"  wrote:
>>>
 Can you pastebin master log before the error showed up ?

 The initial message was posted for Spark 1.2.0
 Which release of Spark / zookeeper do you use ?

 Thanks

 On Mon, Jan 18, 2016 at 6:47 AM, doctorx 
 wrote:

> Hi,
> I am facing the same issue, with the given error
>
> ERROR Master:75 - Leadership has been revoked -- master shutting down.
>
> Can anybody help. Any clue will be useful. Should i change something in
> spark cluster or zookeeper. Is there any setting in spark which can
> help me?
>
> Thanks & Regards
> Raghvendra
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-0-standalone-ha-zookeeper-tp21308p25994.html
> Sent from the Apache Spark User List mailing list archive at
> Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

>>

Re: spark-1.2.0--standalone-ha-zookeeper

[doctorx]

Hi,
I am facing the same issue, with the given error

ERROR Master:75 - Leadership has been revoked -- master shutting down.

Can anybody help. Any clue will be useful. Should i change something in
spark cluster or zookeeper. Is there any setting in spark which can help me?

Thanks & Regards
Raghvendra



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-0-standalone-ha-zookeeper-tp21308p25994.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org

Re: spark-1.2.0--standalone-ha-zookeeper

[Ted Yu]

Can you pastebin master log before the error showed up ?

The initial message was posted for Spark 1.2.0
Which release of Spark / zookeeper do you use ?

Thanks

On Mon, Jan 18, 2016 at 6:47 AM, doctorx  wrote:

> Hi,
> I am facing the same issue, with the given error
>
> ERROR Master:75 - Leadership has been revoked -- master shutting down.
>
> Can anybody help. Any clue will be useful. Should i change something in
> spark cluster or zookeeper. Is there any setting in spark which can help
> me?
>
> Thanks & Regards
> Raghvendra
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-0-standalone-ha-zookeeper-tp21308p25994.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>

Re: spark-1.2.0--standalone-ha-zookeeper

[scar0909]

Hello, have you found a solution? I have the same issue



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/spark-1-2-0-standalone-ha-zookeeper-tp21308p23282.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org