Re: [systemd-devel] multiple starts for a socket-based service

2023-08-03 Thread Mantas Mikulėnas
On Thu, Aug 3, 2023, 21:09 Ross Boylan 
wrote:

> Hi, systemd-ers.  I'm trying to do something that seems at cross
> purposes with systemd's assumptions, and I'm hoping for some guidance.
>
> Goal: remote client sends a 1 line command to a server, which executes
> a script that does not create a long-running service.
> These events will be rare.  I believe the basic model for systemd
> sockets is that service is launched on first contact, and is then
> expected to hang around to handle later requests.  Because such
> requests are rare, I'd rather that the service exit and the process be
> repeated for later connections.
>
> Is there a way to achieve this with systemd?


> It looks as if Accept=yes in the [Socket] section might work, but I'm
> not sure about the details, and have several concerns:
> 1. systemd may attempt to restart my service script if it exits (as it
> seems to have in the logs below).

2. man systemd.socket recommends using Accept=no "for performance
> reasons".  But that seems to imply the service that is activated must
> hang around to handle future requests.
>

That's precisely the performance reason, at least historically. Spawning a
whole new process is heavier than having a running daemon that only needs
to fork, at most (and more commonly just starts a thread or even uses a
single-threaded event loop), so while it is perfectly fine for your case,
it's not recommended for "production" servers.

(Especially if the time needed to load up e.g. a python interpreter, import
all the modules, etc. might be more than the time needed for the script to
actually perform its task...)

3. Accept=yes seems to imply that several instances of the service
> could share the same socket, which seems dicey.  Is systemd
> automatically doing the handshake for TCP sockets,


It does, that's what "accept" means in socket programming.

However, accept() does not reuse the same socket – each accepted connection
spawns a *new* socket, and with Accept=yes it's that "per connection"
socket that's passed on to the service. The original "listening" socket
stays within systemd and is not used for communications, its only purpose
is to wait for new connections to arrive.

This is no different from how you'd handle multiple concurrent connections
in your own program – you create a base "listener" socket, bind it, then
each accept(listener) generates a new "client" socket.

With systemd, the default mode of Accept=no provides your service with the
"listener" but it is the service's job to loop over accept()-ing any number
of clients it wants.

For what it's worth, systemd's .socket units are based on the traditional
Unix "inetd" service that used to have the same two modes (Accept=yes was
called "nowait"). If systemd doesn't quite work for you, you can still
install and use "xinetd" on most Linux distros (and probably five or six
other similar superservers).

in which the server
> generates a new port, communicates it to the client, and it is this
> second port that gets handed to the service?  Or is the expectation
> that the client service will do that early on and close the original
> port?
>

That's not how the TCP handshake works at all. (It's how Arpanet protocols
worked fifty years ago, back when TCP/IP did not exist yet and RFCs were
numbered in the double digits.)

In TCP it is never necessary for a server to generate a new port for each
client, because the client *brings its own* port number – there's one for
each end, and the combination of  is what allows
distinguishing multiple concurrent connections. The sockets returned by
accept() are automatically associated with the correct endpoints.

If you look at the list of active sockets in `netstat -nt` or `ss -nt` (add
-l or -a to also show the listeners), all connections to your server stay
on the same port 14987, but the 4-value  is unique
for each.

Although calls to the service should be rare, it's easy to imagine
> that during development I inadvertently generate rapidly repeated
> calls so that several live processes could end up accessing the same
> socket.
>
> Finally, I'd like to ignore rapid bursts of requests.  Most systemd
> limits seem to put the unit in a permanently failed state if that
> happens, but I would prefer if that didn't happen, ie. ignore but
> continue, rather than ignore and fail.


> My first attempt follows.  It appears to have generated 5 quick
> invocations of the script and then permanent failure of the service
> with service-start-limit-hit.  So even when the burst window ended,
> the service remained down.  I think what happened was that when my
> script finished the service unit took that as a failure (? the logs do
> show the service succeeding, though RemainAfterExit=no)  and tried to
> restart it.  It did this 5 times and then hit a default limit.  Since
> the service and the socket were then considered failed, no more
> traffic on the socket triggered any action.
>

No; more likely what happened is that you forgot 

[systemd-devel] multiple starts for a socket-based service

2023-08-03 Thread Ross Boylan
Hi, systemd-ers.  I'm trying to do something that seems at cross
purposes with systemd's assumptions, and I'm hoping for some guidance.

Goal: remote client sends a 1 line command to a server, which executes
a script that does not create a long-running service.
These events will be rare.  I believe the basic model for systemd
sockets is that service is launched on first contact, and is then
expected to hang around to handle later requests.  Because such
requests are rare, I'd rather that the service exit and the process be
repeated for later connections.

Is there a way to achieve this with systemd?

It looks as if Accept=yes in the [Socket] section might work, but I'm
not sure about the details, and have several concerns:
1. systemd may attempt to restart my service script if it exits (as it
seems to have in the logs below).
2. man systemd.socket recommends using Accept=no "for performance
reasons".  But that seems to imply the service that is activated must
hang around to handle future requests.
3. Accept=yes seems to imply that several instances of the service
could share the same socket, which seems dicey.  Is systemd
automatically doing the handshake for TCP sockets, in which the server
generates a new port, communicates it to the client, and it is this
second port that gets handed to the service?  Or is the expectation
that the client service will do that early on and close the original
port?
Although calls to the service should be rare, it's easy to imagine
that during development I inadvertently generate rapidly repeated
calls so that several live processes could end up accessing the same
socket.

Finally, I'd like to ignore rapid bursts of requests.  Most systemd
limits seem to put the unit in a permanently failed state if that
happens, but I would prefer if that didn't happen, ie. ignore but
continue, rather than ignore and fail.

My first attempt follows.  It appears to have generated 5 quick
invocations of the script and then permanent failure of the service
with service-start-limit-hit.  So even when the burst window ended,
the service remained down.  I think what happened was that when my
script finished the service unit took that as a failure (? the logs do
show the service succeeding, though RemainAfterExit=no)  and tried to
restart it.  It did this 5 times and then hit a default limit.  Since
the service and the socket were then considered failed, no more
traffic on the socket triggered any action.

Here are the last few log entries:
Aug 01 01:42:55 barley systemd[1]: Finished Update kernel netboot info
for family system.
Aug 01 01:42:55 barley systemd[1]: Starting Update kernel netboot info
for family system...
Aug 01 01:42:55 barley systemd[1]: family.service: Succeeded.
Aug 01 01:42:55 barley systemd[1]: Finished Update kernel netboot info
for family system.
Aug 01 01:42:55 barley systemd[1]: family.service: Start request
repeated too quickly.
Aug 01 01:42:55 barley systemd[1]: family.service: Failed with result
'start-limit-hit'.
Aug 01 01:42:55 barley systemd[1]: Failed to start Update kernel
netboot info for family system.
Aug 01 01:42:55 barley systemd[1]: family.socket: Failed with result
'service-start-limit-hit'.

And the config files
# /etc/systemd/system/family.service
[Unit]
Description=Update kernel netboot info for family system

[Service]
Type=oneshot
# next is the default. will be considered failed, and since we want it
# to run multiple times that is good. [Well, maybe not]
RemainAfterExit=no
ExecStart=sh -c "date >> /root/washere"  #for testing

[Install]
WantedBy=network-online.target

# /etc/systemd/system/family.socket
[Unit]
Description=Socket to tickle to update family netboot config

[Install]
WantedBy=network-online.target

[Socket]
ListenStream=192.168.1.10:14987
BindToDevice=br0
# 2s is default
TriggerLimitIntervalSec=5s

Thanks.
Ross


[systemd-devel] systemd talk on Fedora Flock 2023

2023-08-03 Thread Luna Jernberg
https://youtu.be/GkYURkrIzx0

Zbigniew Jędrzejewski-Szmek talking about systemd for users at Fedoras
developer conference Flock 2023 in Cork, Ireland at the moment :)