Re: start-history-server.sh doesn't survive system reboot. Recommendation?

[James Yu]

The Ops guy would probably be fired if he doesn't make sure the container 
runtime is up 24/7.  

From: Mich Talebzadeh 
Sent: Wednesday, December 8, 2021 12:33 PM
Cc: user @spark 
Subject: Re: start-history-server.sh doesn't survive system reboot. 
Recommendation?

Well that is just relying on docker daemon to start after reboot. It may not

docker ps
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the 
docker daemon running?
systemctl start docker
docker ps
CONTAINER IDIMAGE   COMMAND CREATED 
STATUS  PORTS   NAMES

So your mileage varies. A proper way would be to have two instances of 
start-history-server.sh running on different hosts listening to the same Spark, 
thus avoiding a single point of failure.

HTH



 
[https://docs.google.com/uc?export=download=1-q7RFGRfLMObPuQPWSd9sl_H1UPNFaIZ=0B1BiUVX33unjMWtVUWpINWFCd0ZQTlhTRHpGckh4Wlg4RG80PQ]
   view my Linkedin 
profile<https://urldefense.com/v3/__https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/__;!!DvJyiiI1!bR1iEMzC4cyfUQYQRuFn4GzLtS2Qq2vn7gux-j3NU7HJmDM9jWzQgrP4D2HaDw$>



Disclaimer: Use it at your own risk. Any and all responsibility for any loss, 
damage or destruction of data or any other property which may arise from 
relying on this email's technical content is explicitly disclaimed. The author 
will in no case be liable for any monetary damages arising from such loss, 
damage or destruction.




On Wed, 8 Dec 2021 at 19:45, James Yu mailto:ja...@ispot.tv>> 
wrote:
Just thought about another possibility which is to containerize the history 
server and run the container with proper restart policy.  This may be the 
approach we will be taking because the deployment of such HS would be more 
flexible.

Thanks!

From: Sean Owen mailto:sro...@gmail.com>>
Sent: Tuesday, December 7, 2021 1:29 PM
To: James Yu mailto:ja...@ispot.tv>>
Cc: user @spark mailto:user@spark.apache.org>>
Subject: Re: start-history-server.sh doesn't survive system reboot. 
Recommendation?

The scripts just launch the processes. To make any process restart on system 
restart, you would need to set it up as a system service (i.e. controlled by 
'service'). That's easy enough but out of scope for Spark itself.

On Tue, Dec 7, 2021 at 3:23 PM James Yu mailto:ja...@ispot.tv>> 
wrote:
Hi Users,

We found that the history server launched by using the 
"start-history-server.sh" command does not survive system reboot.  Any 
recommendation of making it always up even after reboot?

Thanks,

James

Re: start-history-server.sh doesn't survive system reboot. Recommendation?

[Mich Talebzadeh]

Well that is just relying on docker daemon to start after reboot. It may not

*docker ps*
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the
docker daemon running?
*systemctl start docker*
*docker ps*
CONTAINER IDIMAGE   COMMAND CREATED
 STATUS  PORTS   NAMES

So your mileage varies. A proper way would be to have two instances of
start-history-server.sh running on different hosts listening to the same
Spark, thus avoiding a single point of failure.

HTH


   view my Linkedin profile
<https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/>



*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Wed, 8 Dec 2021 at 19:45, James Yu  wrote:

> Just thought about another possibility which is to containerize the
> history server and run the container with proper restart policy.  This may
> be the approach we will be taking because the deployment of such HS would
> be more flexible.
>
> Thanks!
> --
> *From:* Sean Owen 
> *Sent:* Tuesday, December 7, 2021 1:29 PM
> *To:* James Yu 
> *Cc:* user @spark 
> *Subject:* Re: start-history-server.sh doesn't survive system reboot.
> Recommendation?
>
> The scripts just launch the processes. To make any process restart on
> system restart, you would need to set it up as a system service (i.e.
> controlled by 'service'). That's easy enough but out of scope for Spark
> itself.
>
> On Tue, Dec 7, 2021 at 3:23 PM James Yu  wrote:
>
> Hi Users,
>
> We found that the history server launched by using the
> "start-history-server.sh" command does not survive system reboot.  Any
> recommendation of making it always up even after reboot?
>
> Thanks,
>
> James
>
>

Re: start-history-server.sh doesn't survive system reboot. Recommendation?

[James Yu]

Just thought about another possibility which is to containerize the history 
server and run the container with proper restart policy.  This may be the 
approach we will be taking because the deployment of such HS would be more 
flexible.

Thanks!

From: Sean Owen 
Sent: Tuesday, December 7, 2021 1:29 PM
To: James Yu 
Cc: user @spark 
Subject: Re: start-history-server.sh doesn't survive system reboot. 
Recommendation?

The scripts just launch the processes. To make any process restart on system 
restart, you would need to set it up as a system service (i.e. controlled by 
'service'). That's easy enough but out of scope for Spark itself.

On Tue, Dec 7, 2021 at 3:23 PM James Yu mailto:ja...@ispot.tv>> 
wrote:
Hi Users,

We found that the history server launched by using the 
"start-history-server.sh" command does not survive system reboot.  Any 
recommendation of making it always up even after reboot?

Thanks,

James

Re: start-history-server.sh doesn't survive system reboot. Recommendation?

[Mich Talebzadeh]

One way would be to cron the startup script


   1. enable cron on the host and Unix login that runs
$SPARK_HOME/sbin/start-history-server.sh
   where $SPARK_HOME is set in below cron to absolute value say /opt/spark.
   You can run it as non root, putting it in the crontab of the
   corresponding user
   2. crontab -e
   3. Add the following to the cronfile
   4. @reboot (/opt/spark/sbin/start-history-server.sh >
   /var/tmp/start-history-server_"`date +%Y%m%d_%H%M%S`.log" 2>&1)
   5. crontab -l to verify that it is added OK to the cronfile.

The other way would be to add it to startup in /etc/init.d and create
symlink to /etc/rc.d. Do a Google search for it


HTH


   view my Linkedin profile




*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.




On Tue, 7 Dec 2021 at 21:23, James Yu  wrote:

> Hi Users,
>
> We found that the history server launched by using the
> "start-history-server.sh" command does not survive system reboot.  Any
> recommendation of making it always up even after reboot?
>
> Thanks,
>
> James
>

Re: start-history-server.sh doesn't survive system reboot. Recommendation?

[Sean Owen]

The scripts just launch the processes. To make any process restart on
system restart, you would need to set it up as a system service (i.e.
controlled by 'service'). That's easy enough but out of scope for Spark
itself.

On Tue, Dec 7, 2021 at 3:23 PM James Yu  wrote:

> Hi Users,
>
> We found that the history server launched by using the
> "start-history-server.sh" command does not survive system reboot.  Any
> recommendation of making it always up even after reboot?
>
> Thanks,
>
> James
>

start-history-server.sh doesn't survive system reboot. Recommendation?

[James Yu]

Hi Users,

We found that the history server launched by using the 
"start-history-server.sh" command does not survive system reboot.  Any 
recommendation of making it always up even after reboot?

Thanks,

James