Re: [systemd-devel] Using LoadCredential for passing API key to s3 bucket mount unit
Hi Lennart,
Thank you for your reply!
> systemd only resolves env vars in ExecXYZ= lines, nowhere else. And
> definitely not in Options=
Background: fuse mounts can use arbitrary executables that provide
mount-like calling interface. Those executables can be specified as type
fuse.$my_executable.
For example, the following script can be placed in arbitrary place in
$PATH as "my_wrapper":
#!/bin/sh
env
echo "My credentials directory: ${CREDENTIALS_DIRECTORY}"
echo "Its contents:"
ls -als "${CREDENTIALS_DIRECTORY}"
echo "My argv: $@"
# failing noop, not mounting anything
# this could be a call to s3fs
false
Then, this mount unit could call it:
[Unit]
Description=tmp bucket mount
After=network.target
[Mount]
What=temp-bucket
Where=/mnt/tmp
Type=fuse.my_wrapper
LoadCredential=password_file:/etc/s3fs/tmp_key
Options=passwd_file="${CREDENTIALS_DIRECTORY}"/password_file
This results in the following in journald:
systemd[1]: Mounting tmp bucket mount...
mount[539027]: CREDENTIALS_DIRECTORY=/run/credentials/mnt-tmp.mount
mount[539027]: PWD=/
mount[539027]: SYSTEMD_EXEC_PID=539025
mount[539027]: _=/usr/bin/env
mount[539027]: HOME=/root
mount[539027]: LANG=en_US.utf8
mount[539027]: INVOCATION_ID=f5f58395a5f04f349c97d19a400bfedf
mount[539027]: SHLVL=1
mount[539027]: JOURNAL_STREAM=8:14978691
mount[539026]: My credentials directory: /run/credentials/mnt-tmp.mount
mount[539026]: Its contents:
mount[539028]: ls: cannot access '/run/credentials/mnt-tmp.mount': No such file
or directory
mount[539026]: My argv: temp-bucket /mnt/tmp -o
rw,passwd_file="/run/credentials/mnt-tmp.mount"/password_file,dev,suid
systemd[1]: mnt-tmp.mount: Mount process exited, code=exited, status=1/FAILURE
systemd[1]: mnt-tmp.mount: Failed with result 'exit-code'.
systemd[1]: Failed to mount tmp bucket mount.
Judging by "My argv" line, while the systemd itself would not
resolve CREDENTIALS_DIRECTORY, the mounting script ultimately
does receive its contents if it was specified in Options=
> $CREDENIALS_DIRECTORY should already point to a dir with the unit name
> in it. i.e. what is the precise value?
Apologies, I misspoke in my previous mail. The actual directory is _not_
created, see output above.
$CREDENIALS_DIRECTORY variable does get the expected value
"/run/credentials/mnt-tmp.mount", but this path is not created, thus the
credentials are not passed, see above.
> Consider filing an issue on github, if the creds stuff doesn't
> work. But note that the env var replacement you need to do in mout
> mount.fuse.s3fs wrapper script really, PID 1 won't do that for you.
Can you please confirm that my example wrapper above looks like it should work?
If
so, I will file the issue.
--
With best regards,
--
Vladimir Timofeenko
Re: [systemd-devel] systemd | Requires statement with an instantiated service
Leon Fauster wrote on 02/09/2021 15:39: On 02.09.21 15:12, Andrei Borzenkov wrote: On 02.09.2021 15:10, Leon Fauster wrote: On 02.09.21 08:00, Andrei Borzenkov wrote: On 02.09.2021 01:19, Leon Fauster wrote: Example: a@.service b.service a@.service is started as a@host1.service and b.service must be started after a@host1.service but the unit will be differently parameterized (depended of the region). So I want to generalize the requires statement. If you need to manually instantiate a@.service, you can just as well manually add necessary Requires at the same time. E.g. a@.service: [Install] RequiredBy=b.service systemctl enable a@your-region.service Indeed that was also what I tried but it seems that my problem is that b.service needs a device from a.service, and that seems to need some time to come up. Systemd is here to "fast". Just for the sake It has nothing to do with being fast. Either a@.service should not complete activation until device became available or you are solving the wrong problem, because you actually want to order b.service against device, not some random service. of progress I implemented a workaround, b.service.d/dep.conf [Service] ExecStartPre=/bin/sleep 3 I lost you here. If device appears as result of ExecStart in a.service, no amount of delay *before* ExecStart is going to change anything. If device appears independently of a.service, you are again solving the wrong problem. Let me rephrase it: "a" starts and systemd goes on and starts "b". The "a" has Type=notify. My understanding (I'm not a systemd dev) is that "a" completes from the point of systemd, therefore "b" is then started. But the actual implementation of "b" needs a resource (device) of "a" that obviously is not there when "b" is started. So, "ExecStartPre=/bin/sleep 3" in b.service forces a pause and after this pause the resource of "a" is available and "b" starts successfully. I'm sure that this could be solved differently with all participating components but in real world the human resources are limited and the budget of the customer as well. ExecStartPre solves our problem and some testing shows that different scenarios are covered as well. When I find time I will take a closer look at the components especially the "Type=notify" entry ... Indeed. The correct way to solve this is to make "a" only notify that it has completed starting up when it's finished creating everything that any dependant service will need. This includes opening and listening on any sockets or creating any devices etc. Ultimately the problem here is "a" is notifying too early. Whether you solve this properly (i.e. fix "a") or with some duct tape and good will (i.e. a sleep 3 in "b") is up to you :-) Cheers Col -- Colin Guthrie gmane(at)colin.guthr.ie http://colin.guthr.ie/ Day Job: Tribalogic Limited http://www.tribalogic.net/ Open Source: Mageia Contributor http://www.mageia.org/ PulseAudio Hacker http://www.pulseaudio.org/ Trac Hacker http://trac.edgewall.org/
Re: [systemd-devel] systemd | Requires statement with an instantiated service
On 02.09.21 15:12, Andrei Borzenkov wrote: On 02.09.2021 15:10, Leon Fauster wrote: On 02.09.21 08:00, Andrei Borzenkov wrote: On 02.09.2021 01:19, Leon Fauster wrote: Example: a@.service b.service a@.service is started as a@host1.service and b.service must be started after a@host1.service but the unit will be differently parameterized (depended of the region). So I want to generalize the requires statement. If you need to manually instantiate a@.service, you can just as well manually add necessary Requires at the same time. E.g. a@.service: [Install] RequiredBy=b.service systemctl enable a@your-region.service Indeed that was also what I tried but it seems that my problem is that b.service needs a device from a.service, and that seems to need some time to come up. Systemd is here to "fast". Just for the sake It has nothing to do with being fast. Either a@.service should not complete activation until device became available or you are solving the wrong problem, because you actually want to order b.service against device, not some random service. of progress I implemented a workaround, b.service.d/dep.conf [Service] ExecStartPre=/bin/sleep 3 I lost you here. If device appears as result of ExecStart in a.service, no amount of delay *before* ExecStart is going to change anything. If device appears independently of a.service, you are again solving the wrong problem. Let me rephrase it: "a" starts and systemd goes on and starts "b". The "a" has Type=notify. My understanding (I'm not a systemd dev) is that "a" completes from the point of systemd, therefore "b" is then started. But the actual implementation of "b" needs a resource (device) of "a" that obviously is not there when "b" is started. So, "ExecStartPre=/bin/sleep 3" in b.service forces a pause and after this pause the resource of "a" is available and "b" starts successfully. I'm sure that this could be solved differently with all participating components but in real world the human resources are limited and the budget of the customer as well. ExecStartPre solves our problem and some testing shows that different scenarios are covered as well. When I find time I will take a closer look at the components especially the "Type=notify" entry ... a different workaround would be to let b.service fail and with the use of Restart=on-failure bring it later up (RestartSec=5) but honestly that seems to be more dirty then the above workaround. I had also a device.unit in mind as trigger but I can not say in advance what device will be used (dev0, dev1, dev2). ... I use also a Before=b.service statement for a@.service but that is not enough. Why? a@.service is started before b.service but in the same second, its to close for b.service to be successful. And how exactly Requires was going to help here? I am just following the documentation: - "its a common pattern to include a unit name in both" Just give me some patience :-) -- Leon
Re: [systemd-devel] systemd | Requires statement with an instantiated service
On 02.09.2021 15:10, Leon Fauster wrote: > On 02.09.21 08:00, Andrei Borzenkov wrote: >> On 02.09.2021 01:19, Leon Fauster wrote: >>> Example: >>> >>> a@.service >>> b.service >>> >>> a@.service is started as a@host1.service and b.service must be started >>> after a@host1.service but the unit will be differently parameterized >>> (depended of the region). So I want to generalize the requires >>> statement. >> >> If you need to manually instantiate a@.service, you can just as well >> manually add necessary Requires at the same time. E.g. >> >> a@.service: >> >> [Install] >> RequiredBy=b.service >> >> systemctl enable a@your-region.service >> > > > Indeed that was also what I tried but it seems that my problem is > that b.service needs a device from a.service, and that seems to need > some time to come up. Systemd is here to "fast". Just for the sake It has nothing to do with being fast. Either a@.service should not complete activation until device became available or you are solving the wrong problem, because you actually want to order b.service against device, not some random service. > of progress I implemented a workaround, > > b.service.d/dep.conf > [Service] > ExecStartPre=/bin/sleep 3 > I lost you here. If device appears as result of ExecStart in a.service, no amount of delay *before* ExecStart is going to change anything. If device appears independently of a.service, you are again solving the wrong problem. > a different workaround would be to let b.service fail and with the use > of Restart=on-failure bring it later up (RestartSec=5) but honestly that > seems to be more dirty then the above workaround. > > I had also a device.unit in mind as trigger but I can not say in advance > what device will be used (dev0, dev1, dev2). > > > ... >>> >>> I use also a Before=b.service statement for a@.service but that is not >>> enough. >>> >> >> Why? > > a@.service is started before b.service but in the same second, its to > close for b.service to be successful. > And how exactly Requires was going to help here?
Re: [systemd-devel] systemd | Requires statement with an instantiated service
On 02.09.21 08:00, Andrei Borzenkov wrote: On 02.09.2021 01:19, Leon Fauster wrote: Example: a@.service b.service a@.service is started as a@host1.service and b.service must be started after a@host1.service but the unit will be differently parameterized (depended of the region). So I want to generalize the requires statement. If you need to manually instantiate a@.service, you can just as well manually add necessary Requires at the same time. E.g. a@.service: [Install] RequiredBy=b.service systemctl enable a@your-region.service Indeed that was also what I tried but it seems that my problem is that b.service needs a device from a.service, and that seems to need some time to come up. Systemd is here to "fast". Just for the sake of progress I implemented a workaround, b.service.d/dep.conf [Service] ExecStartPre=/bin/sleep 3 a different workaround would be to let b.service fail and with the use of Restart=on-failure bring it later up (RestartSec=5) but honestly that seems to be more dirty then the above workaround. I had also a device.unit in mind as trigger but I can not say in advance what device will be used (dev0, dev1, dev2). My dropin file in ./b.service.d/dep.conf looks like [Unit] Requires="a@*.service" This just produces following error: 'Failed to add dependency on "a@*.service", ignoring: Invalid argument' I use also a Before=b.service statement for a@.service but that is not enough. Why? a@.service is started before b.service but in the same second, its to close for b.service to be successful. -- Leon
Re: [systemd-devel] Unable to boot Linux distribution ISO files that have systemd services
On Mi, 01.09.21 11:39, EpicLemon99 (epiclemo...@protonmail.com) wrote: > I am unable to boot up ISO files of Linux distributions that use systemd. My > computer is a HP Pavilion TG01-2856no, it is recent hardware. The boot gets > stuck when it tries to start systemd services, such as Network Time > Synchronization. > > For example, there are the messages I get when trying to boot up the Arch > Linux ISO: https://imgur.com/a/oKGjZk7 > > Booting it with the kernel argument init=/bin/bash however works. The second screenshot shows that you have some kernel issue, i.e. the upper part of the screen shows kernel debug output that happens on kernel oops. i.e. it's a driver issue, and systemd hangs simply because the kernel hangs/crashed. Please work with your distro, they might be able to help. Kernel/driver issues like this are out of scope for systemd though. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Using LoadCredential for passing API key to s3 bucket mount unit
On Mi, 01.09.21 13:31, Vladimir Timofeenko (vladi...@vtimofeenko.com) wrote:
> Hi,
>
> I am playing with the idea of using systemd mount to mount S3 bucket on
> the system using s3fs.
>
> To mount a bucket, an API key is required. s3fs can read the API key
> from a file specified as an option:
>
> s3fs $bucket_name $where -o passwd_file=${PATH_TO_PASSWORD_FILE} ...
>
> I tried to set up a .mount unit with LoadCredential directive:
>
> [Unit]
> Description=tmp bucket mount
> After=network.target
>
> [Mount]
> What=temp-bucket
> Where=/mnt/tmp
> Type=fuse.s3fs
> LoadCredential=password_file:/etc/s3fs/tmp_key
> Options=passwd_file="${CREDENTIALS_DIRECTORY}"/password_file,url=https://s3...
systemd only resolves env vars in ExecXYZ= lines, nowhere else. And
definitely not in Options=
> I have used a small wrapper that calls env before calling s3fs to
> investigate, and it appears that during the mount command execution
> ${CREDENTIALS_DIRECTORY} is created, but there is no subdirectory
> corresponding to the unit name.
$CREDENIALS_DIRECTORY should already point to a dir with the unit name
in it. i.e. what is the precise value?
I must admit I never tested credentials with mount units. We might be
missing something there, though I see no reason why it shouldn't work.
Consider filing an issue on github, if the creds stuff doesn't
work. But note that the env var replacement you need to do in mout
mount.fuse.s3fs wrapper script really, PID 1 won't do that for you.
Lennart
--
Lennart Poettering, Berlin
Re: [systemd-devel] systemd | Requires statement with an instantiated service
On 02.09.2021 01:19, Leon Fauster wrote: > Dear list, > > following requirement exists here (systemd-239 installed): > > Applying a "Requires" statement with an instantiated service. > > Example: > > a@.service > b.service > > a@.service is started as a@host1.service and b.service must be started > after a@host1.service but the unit will be differently parameterized > (depended of the region). So I want to generalize the requires statement. If you need to manually instantiate a@.service, you can just as well manually add necessary Requires at the same time. E.g. a@.service: [Install] RequiredBy=b.service systemctl enable a@your-region.service > > My dropin file in ./b.service.d/dep.conf looks like > > [Unit] > Requires="a@*.service" > > This just produces following error: > 'Failed to add dependency on "a@*.service", ignoring: Invalid argument' > > I use also a Before=b.service statement for a@.service but that is not > enough. > Why?
