Re: [systemd-devel] systemctl status and service unit configuration with startup script that exits

2016-12-21 Thread Jonathan de Boyne Pollard 

Owens, Stephen:

The start script calls other scripts to start two application 
components and then the start script exits.


One of the application components also spawns a couple of child processes.

There ends up being two pid files, one for each component, describing 
the state of the application.




You have two services.


Owens, Stephen:


Environment=JAVA_HOME=/opt/xxx/jre

ExecStart=/bin/bash -l /opt/xxx/bin/startApp.sh

ExecStop=-/bin/bash -lc "/opt/xxx/bin/stopApp.sh -np"



Java, again.  Java is for some reason a magnet for this sort of stuff.

* http://jdebp.eu./FGA/systemd-house-of-horror/


Owens, Stephen:

Is there some way I can have systemd monitor the component pids to 
show ACTIVE status?


You have two services.  Make two service units.  Forget the nutty 
startApp and stopApp scripts.  When you look into them you'll almost 
certainly find that they are a process supervisor in shell script, 
written badly as such things always are.  The systemd House of Horror 
has examples of running Java programs directly from service units (It 
isn't hard.) without the intermediate badly written shell script process 
supervisors.  Stack Exchange has stuff on this, too, and more besides.


* http://unix.stackexchange.com/a/320319/5132

* http://unix.stackexchange.com/a/245038/5132

* http://unix.stackexchange.com/a/322778/5132

* http://unix.stackexchange.com/a/229525/5132

* http://unix.stackexchange.com/a/324286/5132

* http://unix.stackexchange.com/a/321716/5132

* http://unix.stackexchange.com/a/280399/5132


Owens, Stephen:


I'm stuck with 3rd party startup scripts.



No, you are not.  This is just a belief and a self-imposed restriction, 
and it is groundless when it comes down to it.  Look into it, as other 
people have, and you'll find the world is quietly, and finally, coming 
around to the daemontools idea, that has been around since 1996, of 
letting the service manager do this.  The MariaDB world, for example, 
has finally officially let go of its wrapper process supervisor shell 
scripts and documented how to just run the daemon directly under a 
service manager, 13 years after daemontools users asked for this.


* http://jdebp.eu./Softwares/nosh/mariadb-and-mysql.html#Prompt

The irony is that if you actually look at your scripts that start/stop 
two services you will probably find that they are wrappers for wrappers, 
like the startdb.sh and stopdb.sh that these people wrote:


* http://itdadao.com/articles/c15a369637p0.html

* 
https://mohamedazar.com/2015/08/27/oracle-applications-r12-auto-start-on-linux/


Even if you hang on to the incorrect belief that these wrapper scripts 
are set in stone, there's a glaringly obvious decomposition into two 
parts in such cases.  You have two services.


Whenever you see the patterns ...


ExecStart=/opt/example/bin/something.sh start
ExecStop=/opt/example/bin/something.sh stop


or


ExecStart=/opt/example/bin/startSomething.sh
ExecStop=/opt/example/bin/stopSomething.sh


... the Oracle camelCase and the "stop"/"start" arguments are strong 
indicators that in fact those scripts will turn out to be process 
supervisors written badly in shell script, grepping the process tree, 
using PID files, doing their own service restarts with while loops and 
wait, creating private log files that only ever expand indefinitely, and 
running the actual service processes as a grandchild or even a 
great-grandchild of the main process.  (Some of the most stupid cases 
are when one strips away one unnecessary shell script badly written 
process supervisor, only to find that the process that it is supervising 
is itself *another* shell script badly written process supervisor.)


Another giveaway is the "I had to add RemainAfterExit=yes because 
otherwise it didn't work." as exemplified in the ridiculous edifice 
(that clearly hasn't been tested otherwise the invalid attempts at 
redirection would have been spotted, like they were in three of the 
Stack Exchange questions mentioned earlier) recommended by Oracle 
consultant Tim Hall here:


* 
https://oracle-base.com/articles/linux/linux-services-systemd#creating-linux-services


M. Hall had to add RemainAfterExit=yes because xe is running shell 
scripts that in turn fork and execute programs such as "lsnrctl" and 
"sqlplus", some of which in turn fork and execute further processes and 
exit the parent processes.  The shell scripts aren't necessary at all.  
Xe actually has two services, just like you.  In contrast: this person 
has, at least (the "something start/stop" layer that forks still being 
there), decomposed having two services into having two service definitions:


* https://server-world.info/en/note?os=CentOS_7=oracle12c=6

If one has to add RemainAfterExit=yes in order to bodge things, then its 
quite likely that one has built such a ridiculous edifice, too. The 
shell scripts are forking subprocesses to run non-builtin commands, 
which are in turn probably *more* shell scripts that *also* fork,

Re: [systemd-devel] systemctl status and service unit configuration with startup script that exits

2016-12-21 Thread Jonathan de Boyne Pollard 

Owens, Stephen:


UMask=0764



This is almost certainly wrong, by the way.

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] systemctl status and service unit configuration with startup script that exits

2016-12-20 Thread Lennart Poettering 
On Tue, 20.12.16 17:41, Owens, Stephen (MASSIT) (stephen.ow...@mass.gov) wrote:

> I'm stuck with 3rd party startup scripts.
> 
> I was confused by the systemd-service documention for Type
> Type=
>   If set to forking, it is expected that the process configured with 
> ExecStart= will call fork() as part of its start-up
> 
> As far as I could see, the 3rd party scripts do not use fork.

Well, fork() is how on UNIX processes are created. systemd will fork()
once to invoke your startup script. As that startup script creates
additional processes as you say there must be fork() involved.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] systemctl status and service unit configuration with startup script that exits

2016-12-20 Thread Owens, Stephen (MASSIT) 
I'm stuck with 3rd party startup scripts.

I was confused by the systemd-service documention for Type
Type=
If set to forking, it is expected that the process configured with 
ExecStart= will call fork() as part of its start-up

As far as I could see, the 3rd party scripts do not use fork.

Thanks,

Stephen R. Owens
Senior Software Engineer
Massachusetts Office of Information Technology
Commonwealth of Massachusetts
One Ashburton Place, Room 1611
Boston, MA 02108
Tel:   617-626-4513  x24513
Email: stephen.ow...@state.ma.us


-Original Message-
From: Lennart Poettering [mailto:lenn...@poettering.net] 
Sent: Tuesday, December 20, 2016 12:28 PM
To: Owens, Stephen (MassIT)
Cc: systemd-devel@lists.freedesktop.org
Subject: Re: [systemd-devel] systemctl status and service unit configuration 
with startup script that exits

On Tue, 20.12.16 17:10, Owens, Stephen (MASSIT) (stephen.ow...@mass.gov) wrote:

> [Service]
> Type=oneshot
> RemainAfterExit=yes

Type=oneshot is for services that run at boot and exit at
boot. i.e. stuff that generally doesn't stay around, stuff like fsck
or so which runs and exits before the boot proceeds.

If you service spawns processes that stay around use Type=forking or so.

> KillMode=none

This is a hack. Don't use this. If you use the right Type= (as
discussed above), then you don't need to explicitly disable systemd's
service tracking.

> The Main PID: 15346 I assume was the pid for the startup script which exits.
> Is there some way I can have systemd monitor the component pids to show 
> ACTIVE status?
> Is there a better way to configure the service unit - given that we have to 
> use the 3rd party scripts?

Well, the systemd model is to expose each daemon as one seperate
.service. In your case however you appear to wrap two daemons in a
single service. This will always be awkward. In particular as systemd
only considers a single PID the "main PID" of a service (well, in fact
there may be zero or one main PIDs, but not more), which doesn't map
at all to your case, since you have two PIDs that could be considered
"main".

Any such mapping will hence be a bit off, and systemd won't notice
when your daemons stop (unless they both stop), and hence Restart= and
things like that won't work, and neither will runtime failure
detection.

My recommendation is: fix this properly, write two separate service
units, it's the better choice, in particular in the long run. If you
do, then systemd can properly track the service in all details.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

Re: [systemd-devel] systemctl status and service unit configuration with startup script that exits

2016-12-20 Thread Lennart Poettering 
On Tue, 20.12.16 17:10, Owens, Stephen (MASSIT) (stephen.ow...@mass.gov) wrote:

> [Service]
> Type=oneshot
> RemainAfterExit=yes

Type=oneshot is for services that run at boot and exit at
boot. i.e. stuff that generally doesn't stay around, stuff like fsck
or so which runs and exits before the boot proceeds.

If you service spawns processes that stay around use Type=forking or so.

> KillMode=none

This is a hack. Don't use this. If you use the right Type= (as
discussed above), then you don't need to explicitly disable systemd's
service tracking.

> The Main PID: 15346 I assume was the pid for the startup script which exits.
> Is there some way I can have systemd monitor the component pids to show 
> ACTIVE status?
> Is there a better way to configure the service unit - given that we have to 
> use the 3rd party scripts?

Well, the systemd model is to expose each daemon as one seperate
.service. In your case however you appear to wrap two daemons in a
single service. This will always be awkward. In particular as systemd
only considers a single PID the "main PID" of a service (well, in fact
there may be zero or one main PIDs, but not more), which doesn't map
at all to your case, since you have two PIDs that could be considered
"main".

Any such mapping will hence be a bit off, and systemd won't notice
when your daemons stop (unless they both stop), and hence Restart= and
things like that won't work, and neither will runtime failure
detection.

My recommendation is: fix this properly, write two separate service
units, it's the better choice, in particular in the long run. If you
do, then systemd can properly track the service in all details.

Lennart

-- 
Lennart Poettering, Red Hat
___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel

[systemd-devel] systemctl status and service unit configuration with startup script that exits

2016-12-20 Thread Owens, Stephen (MASSIT) 
Hi,

I'm not sure if this is the correct list to ask this question, it's about 
systemd unit configuration.

We have a 3rd party application that starts and stops via startup and shutdown 
scripts which we would like to have start up automatically at linux boot time.

The start script calls other scripts to start two application components and 
then the start script exits.
One of the application components also spawns a couple of child processes.
There ends up being two pid files, one for each component, describing the state 
of the application.

Below is an iteration of the unit service file for the application:

[Unit]
Description=XXX Application
RequiresMountsFor=/ /opt /tmp
After=network-online.target sshd.service
Requires=sshd.service

[Service]
Type=oneshot
User=xxx
UMask=0764
TimeoutSec=360s
LimitSTACK=infinity
LimitNOFILE=8192
Environment=HOME=/home/xxx
Environment=JAVA_HOME=/opt/xxx/jre
Environment=CLASSPATH=.:/opt/xxx/jre/lib
Environment=PATH=/opt/xxx/jre/bin:/opt/xxx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
WorkingDirectory=/opt/xxx/bin
RemainAfterExit=yes
ExecStart=/bin/bash -l /opt/xxx/bin/startApp.sh
ExecStop=-/bin/bash -lc "/opt/xxx/bin/stopApp.sh -np"
KillMode=none

[Install]
WantedBy=multi-user.target


The above unit file works for starting and stopping the application both at 
boot and via:
systemctl start xxx
systemctl stop xxx

When I check the systemctl service status
$ systemctl list-units -t service
  UNITLOAD   ACTIVE SUB DESCRIPTION
  xxx.service loaded active exited  XXX Application

I think the RemainAfterExit directive tells systemd to show the ACTIVE state 
even though the startup script exited.
Both the loaded and active status make sense.
The low-level unit activation state of exited makes sense since the start 
script for the application exits after starting the application components.

For some more info:
$ /bin/systemctl status xxx
● xxx.service - XXX Application
   Loaded: loaded (/etc/systemd/system/xxx.service; enabled; vendor preset: 
disabled)
   Active: active (exited) since Mon 2016-12-19 16:15:13 EST; 18h ago
Main PID: 15346 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/xxx.service
   ├─15402 /opt/xxx/jre/bin/java ...
   ├─16249 /opt/xxx/jre/bin/java ...
   ├─16893 /opt/xxx/...
   ├─16894 /opt/xxx/jre/bin/java ...
   ├─17146 /bin/sh /opt/xxx/...
   ├─17173 /opt/xxx/jre/bin/java ...
   ├─17201 /opt/xxx/...
   ├─17239 /opt/xxx/...
   └─17945 /opt/xxx/jre/bin/java ...

The Main PID: 15346 I assume was the pid for the startup script which exits.
Is there some way I can have systemd monitor the component pids to show ACTIVE 
status?
Is there a better way to configure the service unit - given that we have to use 
the 3rd party scripts?

Thanks,

Stephen R. Owens
Senior Software Engineer
Massachusetts Office of Information Technology
Commonwealth of Massachusetts
One Ashburton Place, Room 1611
Boston, MA 02108
Tel:   617-626-4513  x24513
Email: stephen.ow...@state.ma.us

___
systemd-devel mailing list
systemd-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/systemd-devel