bug#60657: Rethinking how service extensions work

2023-05-11 Thread Ludovic Courtès
Hi Bruno,

Bruno Victal  skribis:

> On 2023-02-25 17:46, Ludovic Courtès wrote:

[...]

>> As we once discussed on IRC, the conclusion to me is that some of the
>> code currently implemented as activation snippets should rather be
>> implemented either as part of the ‘start’ method of the corresponding
>> Shepherd service, or as a one-shot Shepherd service that the main
>> service would depend on.
>
> I think moving them into the ‘start’ method is the best course of action.
> I'm considering the following changes:
> * Adding (gnu build activation) to %default-imported-modules + 
> %default-modules in (gnu services shepherd).
>   I expect that mkdir-p/perms is going to be used frequently enough, using 
> the number of activation-service
>   extensions in use as a rough estimate.
> * Refactor the activation extensions into the ‘start’ method, where it makes 
> sense to do so.

OK.  Cosmetic considerations: how about adding a ‘pre-start’ field in
?  That would allow us to keep the “setup” bit
visually separate from the actual ‘start’ method, even if under the hood
they get “merged” together:

  (shepherd-service
;; …
(pre-start #~(mkdir-p "/whatever"))
(start #~(make-forkexec-constructor …)))

> There's one issue I'm somewhat concerned about, consider the following 
> snippet:
>
>
> (define log-directory "/var/log")
> (define username "notroot")
>
> (start
>  #~(lambda _
> (mkdir-p/perms #$log-directory (getpw #$username) #o750)
> ...))
>
> This is somewhat pitfall prone since you most likely don't want to chown 
> /var/log to a non-root user.
> I'm unsure what's the best course to take here, would a simple file-exist? 
> check before mkdir-p/perms be sufficient?

We ensure /var/log exists before anything else—see ‘directives’ in (gnu
build install).

If we want an extra safety, we can add a real activation snippet that
does (mkdir-p "/var/log"), with the understanding that it would notably
run at boot time before shepherd is started.

> In either case, with or without refactoring this issue is already present 
> (but in activation-service extensions)
> so it's no worse than the status quo.

Right.

>> Note that this should prolly be declared as a ‘file-system’ rather than
>> as a custom service.  That way, it would get a “standard” Shepherd
>> service.
>> 
>> There are cases where we add explicit dependencies on
>> ‘file-system-/media/foo’ or similar.   has a ‘dependencies’
>> field specifically for this purpose (info "(guix) File Systems").
>> 
>> Would that work for you?
>
> Unfortunately OverlayFS is filtered out from fstab by Guix (reported #60246) 
> and the dependencies field IMO is too restrictive,
> there should be a (sane) way to pass shepherd service symbols too. (for cases 
> where a file system depends on 'networking or
> depends on a particular interface e.g. NFS mount that uses a IPv6 link-local 
> address)

Sure, we could make these changes.  Let’s discuss it separately?

Thanks,
Ludo’.





bug#60657: Rethinking how service extensions work

2023-05-10 Thread Liliana Marie Prikler
Am Dienstag, dem 09.05.2023 um 20:12 +0100 schrieb Bruno Victal:
> Hi Ludo’,
> 
> On 2023-02-25 17:46, Ludovic Courtès wrote:
> > Bruno Victal  skribis:
> > > In [1], the issue arises from using activation-service-type to
> > > create files/directories for services
> > > when these should be either (1) shepherd one-shot services or
> > > moved into the 'start' procedure of the service.
> > > 'activation-service-type' should only be used for doing things
> > > "listed on its label", that is, performing
> > > actions at boot-time or after a system reconfigure.
> > 
> > Right.
> > 
> > As we once discussed on IRC, the conclusion to me is that some of
> > the
> > code currently implemented as activation snippets should rather be
> > implemented either as part of the ‘start’ method of the
> > corresponding
> > Shepherd service, or as a one-shot Shepherd service that the main
> > service would depend on.
> 
> I think moving them into the ‘start’ method is the best course of
> action.
> I'm considering the following changes:
> * Adding (gnu build activation) to %default-imported-modules +
> %default-modules in (gnu services shepherd).
>   I expect that mkdir-p/perms is going to be used frequently enough,
> using the number of activation-service
>   extensions in use as a rough estimate.
> * Refactor the activation extensions into the ‘start’ method, where
> it makes sense to do so.
> 
> 
> There's one issue I'm somewhat concerned about, consider the
> following snippet:
> 
> --8<---cut here---start->8---
> 
> (define log-directory "/var/log")
> (define username "notroot")
> 
> (start
>  #~(lambda _
>     (mkdir-p/perms #$log-directory (getpw #$username) #o750)
>     ...))
> 
> --8<---cut here---end--->8---
> 
> This is somewhat pitfall prone since you most likely don't want to
> chown /var/log to a non-root user.
> I'm unsure what's the best course to take here, would a simple file-
> exist? check before mkdir-p/perms be sufficient?
I think this question highlights perfectly why one-shot services (or
perhaps an as-of yet unknown type of services) are the way to go: With
clearly named services for the creation of directories, you don't need
to worry about creating some file with the wrong permissions as the
owner is already predetermined.  You also don't need mkdir-p; you
simply depend on the mkdir-#$(dirname my-directory) service.


Cheers





bug#60657: Rethinking how service extensions work

2023-05-09 Thread Bruno Victal
Hi Ludo’,

On 2023-02-25 17:46, Ludovic Courtès wrote:
> Bruno Victal  skribis:
>> In [1], the issue arises from using activation-service-type to create 
>> files/directories for services
>> when these should be either (1) shepherd one-shot services or moved into the 
>> 'start' procedure of the service.
>> 'activation-service-type' should only be used for doing things "listed on 
>> its label", that is, performing
>> actions at boot-time or after a system reconfigure.
> 
> Right.
> 
> As we once discussed on IRC, the conclusion to me is that some of the
> code currently implemented as activation snippets should rather be
> implemented either as part of the ‘start’ method of the corresponding
> Shepherd service, or as a one-shot Shepherd service that the main
> service would depend on.

I think moving them into the ‘start’ method is the best course of action.
I'm considering the following changes:
* Adding (gnu build activation) to %default-imported-modules + %default-modules 
in (gnu services shepherd).
  I expect that mkdir-p/perms is going to be used frequently enough, using the 
number of activation-service
  extensions in use as a rough estimate.
* Refactor the activation extensions into the ‘start’ method, where it makes 
sense to do so.


There's one issue I'm somewhat concerned about, consider the following snippet:

--8<---cut here---start->8---

(define log-directory "/var/log")
(define username "notroot")

(start
 #~(lambda _
(mkdir-p/perms #$log-directory (getpw #$username) #o750)
...))

--8<---cut here---end--->8---

This is somewhat pitfall prone since you most likely don't want to chown 
/var/log to a non-root user.
I'm unsure what's the best course to take here, would a simple file-exist? 
check before mkdir-p/perms be sufficient?

In either case, with or without refactoring this issue is already present (but 
in activation-service extensions)
so it's no worse than the status quo.

>> (simple-service 'mount-overlayfs shepherd-root-service-type
>> (list (shepherd-service (requirement '(foo-mount))
>> (provision '(overlayfs-foo))
>> (documentation "Mount OverlayFS.")
>> (one-shot? #t)
>> (start (let ((util-linux (@ (gnu 
>> packages linux) util-linux)))
>>  #~(lambda _
>>  (system* #$(file-append 
>> util-linux "/bin/mount")
>>   "-t" "overlay"
>>   "-o" 
>> (string-append "noatime,nodev,noexec,ro,"
>>  
>>  "lowerdir="
>>  
>>  (string-join '("/srv/foo/overlays/top-layer"
>>  
>> "/srv/foo/overlays/layer2"
>>  
>> "/srv/foo/overlays/layer1"
>>  
>> "/media/foo-base") ":"))
>>   "none" 
>> "/media/foo" )))
> 
> Note that this should prolly be declared as a ‘file-system’ rather than
> as a custom service.  That way, it would get a “standard” Shepherd
> service.
> 
> There are cases where we add explicit dependencies on
> ‘file-system-/media/foo’ or similar.   has a ‘dependencies’
> field specifically for this purpose (info "(guix) File Systems").
> 
> Would that work for you?

Unfortunately OverlayFS is filtered out from fstab by Guix (reported #60246) 
and the dependencies field IMO is too restrictive,
there should be a (sane) way to pass shepherd service symbols too. (for cases 
where a file system depends on 'networking or
depends on a particular interface e.g. NFS mount that uses a IPv6 link-local 
address)


Cheers,
Bruno





bug#60657: Rethinking how service extensions work

2023-02-25 Thread Ludovic Courtès
Hi Bruno,

Bruno Victal  skribis:

> The current situation with services in Guix is that service extensions do not 
> care about dependencies.

This is the result of “services” being unrelated to “Shepherd services”,
as noted in the manual (info "(guix) Services").

> This can result in cryptic errors as seen in [1].
>
> [1] https://issues.guix.gnu.org/57589#12
>
> In [1], the issue arises from using activation-service-type to create 
> files/directories for services
> when these should be either (1) shepherd one-shot services or moved into the 
> 'start' procedure of the service.
> 'activation-service-type' should only be used for doing things "listed on its 
> label", that is, performing
> actions at boot-time or after a system reconfigure.

Right.

As we once discussed on IRC, the conclusion to me is that some of the
code currently implemented as activation snippets should rather be
implemented either as part of the ‘start’ method of the corresponding
Shepherd service, or as a one-shot Shepherd service that the main
service would depend on.

> But both solutions (1) and (2) are still not enough as the directories 
> themselves might not yet
> be available and the services must be aware of this fact and wait for them to 
> be ready. One example
> would be a network dependent mount or a simple service that mounts a volume 
> such as:
>
> (simple-service 'mount-overlayfs shepherd-root-service-type
> (list (shepherd-service (requirement '(foo-mount))
> (provision '(overlayfs-foo))
> (documentation "Mount OverlayFS.")
> (one-shot? #t)
> (start (let ((util-linux (@ (gnu 
> packages linux) util-linux)))
>  #~(lambda _
>  (system* #$(file-append 
> util-linux "/bin/mount")
>   "-t" "overlay"
>   "-o" 
> (string-append "noatime,nodev,noexec,ro,"
>   
> "lowerdir="
>   
> (string-join '("/srv/foo/overlays/top-layer"
>   
>"/srv/foo/overlays/layer2"
>   
>"/srv/foo/overlays/layer1"
>   
>"/media/foo-base") ":"))
>   "none" 
> "/media/foo" )))

Note that this should prolly be declared as a ‘file-system’ rather than
as a custom service.  That way, it would get a “standard” Shepherd
service.

There are cases where we add explicit dependencies on
‘file-system-/media/foo’ or similar.   has a ‘dependencies’
field specifically for this purpose (info "(guix) File Systems").

Would that work for you?

HTH,
Ludo’.





bug#60657: Rethinking how service extensions work

2023-01-24 Thread Bruno Victal
On 2023-01-08 12:31, Bruno Victal wrote:
> (...) the issue arises from using activation-service-type to create 
> files/directories for services
> when these should be either (1) shepherd one-shot services or moved into the 
> 'start' procedure of the service.

Idea:
Instead of moving these procedures into the start procedure from 
shepherd-service and end up with a very
large start constructor, we could augment  with a 'pre-start' 
field that is responsible for
setting up the initial conditions for the service. That is, we move most of the 
code in the activation-service-type extensions
into this 'pre-start' field. We could also consider if it would make sense 
adding post-start, pre-stop and post-stop fields.


Cheers,
Bruno





bug#60657: Rethinking how service extensions work

2023-01-08 Thread Bruno Victal
Hi all,

The current situation with services in Guix is that service extensions do not 
care about dependencies.
This can result in cryptic errors as seen in [1].

In [1], the issue arises from using activation-service-type to create 
files/directories for services
when these should be either (1) shepherd one-shot services or moved into the 
'start' procedure of the service.
'activation-service-type' should only be used for doing things "listed on its 
label", that is, performing
actions at boot-time or after a system reconfigure.

But both solutions (1) and (2) are still not enough as the directories 
themselves might not yet
be available and the services must be aware of this fact and wait for them to 
be ready. One example
would be a network dependent mount or a simple service that mounts a volume 
such as:

--8<---cut here---start->8---
(simple-service 'mount-overlayfs shepherd-root-service-type
(list (shepherd-service (requirement '(foo-mount))
(provision '(overlayfs-foo))
(documentation "Mount OverlayFS.")
(one-shot? #t)
(start (let ((util-linux (@ (gnu 
packages linux) util-linux)))
 #~(lambda _
 (system* #$(file-append 
util-linux "/bin/mount")
  "-t" "overlay"
  "-o" 
(string-append "noatime,nodev,noexec,ro,"

  "lowerdir="

  (string-join '("/srv/foo/overlays/top-layer"

 "/srv/foo/overlays/layer2"

 "/srv/foo/overlays/layer1"

 "/media/foo-base") ":"))
  "none" 
"/media/foo" )))
--8<---cut here---end--->8---

This example also means that it's untenable to just look into the file-systems 
field entries and attempt
to intelligently discover which paths are required for the services and add 
them as dependencies (another hole to this idea
is that overlayfs and some fuse filesystems can mount over the same path).

I've proposed in [2] for the service procedure to accept optional arguments, 
these could be of interest in solving this problem.
Another place we should look at is how systemd manages its service 
dependencies, with the 'Wants', 'After', 'Before', 'RequiresMountsFor', etc. [3]
directives. These could potentially be implemented and used alongside [2].

Such changes might also imply that a UI change in herd is required to handle 
the structured information or to avoid cluttering it with too
much "noise".


[1]: https://issues.guix.gnu.org/57589#12
[2]: https://lists.gnu.org/archive/html/guix-devel/2022-12/msg00292.html
[3]: 
https://www.freedesktop.org/software/systemd/man/systemd.unit.html#%5BUnit%5D%20Section%20Options