Re: GSoC 2018 Syntax and semantics of systemd units in the Shepherd - 1st update

[Ioannis Panagiotis Koutsidis]

I am currently working on the implementation of .socket unit files, signalfd for 
signal handling, and fibers. It is mostly done, I just have to fix some issues 
that are left.


On 06/25/18 13:47, boskov...@gmail.com wrote:

Hello, could you please send us an update on your project?

Re: GSoC 2018 Syntax and semantics of systemd units in the Shepherd - 1st update

[Gábor Boskovits]

Hello, could you please send us an update on your project?

Re: GSoC 2018 Syntax and semantics of systemd units in the Shepherd - 1st update

[Ludovic Courtès]

Heya,

Ioannis Panagiotis Koutsidis  skribis:

> Thank you a lot for your comments! I will make sure to make the changes that 
> you
> suggested.

Awesome.

> As for match and things like car/cdr, I had issues with match and signal 
> handling
> in the service file, which was why I changed it with a cond. As for the unit 
> parser
> I also take the rest of the list via cdar because match in something like
> (x y rest ...) does not bind rest - I will probably have to use (x . (y . 
> rest)) in
> the replacement.

If you have a specific example, we can look at it.  Rest elements in
‘match’ patterns should definitely get bound.  Here’s an example at the
Guile REPL:

--8<---cut here---start->8---
scheme@(guile-user)> ,use(ice-9 match)
scheme@(guile-user)> (match '(hello brave gnu world!)
   ((x rest ...)
rest))
$2 = (brave gnu world!)
--8<---cut here---end--->8---

Here ‘rest’ is bound to the cdr of the list.

If you want let’s get in touch on IRC to discuss the issue that you had.

Cheers,
Ludo’.

Re: GSoC 2018 Syntax and semantics of systemd units in the Shepherd - 1st update

[Ioannis Panagiotis Koutsidis]

Thank you a lot for your comments! I will make sure to make the changes that you
suggested.

As for match and things like car/cdr, I had issues with match and signal 
handling
in the service file, which was why I changed it with a cond. As for the unit 
parser
I also take the rest of the list via cdar because match in something like
(x y rest ...) does not bind rest - I will probably have to use (x . (y . 
rest)) in
the replacement.

On 06/11/18 14:47, Ludovic Courtès wrote:

Hello Ioannis!

Thanks for the update!

Ioannis Panagiotis Koutsidis  skribis:


As the 1st phase is coming to an end I decided to post my progress. I have
implemented the unit file parsing as well as some of the basic entries supported
by it, such as ExecStart, User, Group, Restart, etc. In addition, support for
the systemd Restart values (on-success, on-failure, on-abnormal, and on-abort)
was added to the Shepherd via the restart-systemd field in the  class,
letting services written in guile to also use that feature.


Very nice!


During the next phases I will focus on other common .service entries, .socket
support, as well as thoroughly testing the code.


Cool.  Adding unit tests like those currently under tests/ is definitely
something you should do—you probably already run tests manually anyway,
so it’s mostly a matter of putting them in a file.

For things like the unit file parser, you may find it more convenient to
write the test in Scheme (currently all the tests are shell scripts.)
That can easily be done by using the .scm file name extension for your
test and then defining ‘SCM_LOG_COMPILER’ in Makefile.am.  If unsure,
you can look at how Guix itself does it, or just stop by on #guix or ask
on the list for details.

Some comments about the code:


 From a0a46ead5e43cd2672a08adb4c16919c377514c2 Mon Sep 17 00:00:00 2001
From: Ioannis Panagiotis Koutsidis 
Date: Sat, 9 Jun 2018 16:17:27 +0300
Subject: [PATCH] Initial systemd unit support


Could you try to split it in several patches, where each patch
represents a single “logical” change?

By that I mean that you could have a first patch that modifies ‘restart’
and all in (shepherd service), possibly with extended tests to exercise
the new functionality if appropriate.

A second patch would add the unit file parser in (shepherd systemd)
along with its unit test.

For commit logs, please try to follow the ChangeLog convention:
.  You
can look at ‘git log’ and basically try to mimic what’s been done
before.  Don’t lose your hair over commit logs though; it’s good to try
to follow the conventions, but if you’re unsure or if you make mistakes,
it’s not the end of the world.


@@ -165,6 +166,11 @@ respawned, shows that it has been respawned more than TIMES in 
SECONDS."
(respawn? #:init-keyword #:respawn?
#:init-value #f
#:getter respawn?)
+  ;; For the systemd restart values.  Can be 'no (when respawn? is #f),
+  ;; 'on-success, 'on-failure, 'on-abnormal, 'on-watchdog, 'on-abort, or 
'always
+  (respawn-systemd #:init-keyword #:respawn-systemd
+   #:init-value 'always
+   #:getter respawn-systemd)


As briefly discussed on IRC, I think we should keep a single field for
this.  So perhaps ‘respawn?’ must simply be renamed to ‘respawn’ (no
question mark), with a comment like above explaining what the possible
values are.


+  (let* ([e (status:exit-val status)]
+ [t (status:term-sig status)]
+ [r (respawn-systemd serv)]


Please avoid square brackets to remain consistent with the rest of the
code.  :-)


+ [clean (or (zero?  e)
+(equal? t SIGHUP)
+(equal? t SIGINT)
+(equal? t SIGTERM)
+(equal? t SIGPIPE))])


Use ‘=’ rather than ‘equal?’ when we know we’re dealing with numbers.


+(if (or (equal? r 'always)
+(equal? r 'on-watchdog) ;; not implemented yet
+(and (equal? r 'on-success) clean)
+(and (equal? r 'on-abnormal) (not clean) (equal? e #f))
+(and (equal? r 'on-failure)  (not clean))
+(and (equal? r 'on-abort)(equal? t SIGABRT)))


Likewise, use ‘eq?’ for symbols.


+++ b/modules/shepherd/systemd.scm
@@ -0,0 +1,143 @@
+;; systemd.scm -- Systemd support
+;; Copyright (C) 2018 Ioannis Panagiotis Koutsidis 
+;;
+;; This file is part of the GNU Shepherd.
+;;
+;; The GNU Shepherd is free software; you can redistribute it and/or modify it
+;; under the terms of the GNU General Public License as published by
+;; the Free Software Foundation; either version 3 of the License, or (at
+;; your option) any later version.
+;;
+;; The GNU Shepherd is distributed in the hope that it will be useful, but
+;; WITHOUT ANY WARRANTY; without even the implied warranty of
+;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+;; GNU General 

Re: GSoC 2018 Syntax and semantics of systemd units in the Shepherd - 1st update

[Ludovic Courtès]

Hello Ioannis!

Thanks for the update!

Ioannis Panagiotis Koutsidis  skribis:

> As the 1st phase is coming to an end I decided to post my progress. I have
> implemented the unit file parsing as well as some of the basic entries 
> supported
> by it, such as ExecStart, User, Group, Restart, etc. In addition, support for
> the systemd Restart values (on-success, on-failure, on-abnormal, and on-abort)
> was added to the Shepherd via the restart-systemd field in the  
> class,
> letting services written in guile to also use that feature.

Very nice!  

> During the next phases I will focus on other common .service entries, .socket
> support, as well as thoroughly testing the code.

Cool.  Adding unit tests like those currently under tests/ is definitely
something you should do—you probably already run tests manually anyway,
so it’s mostly a matter of putting them in a file.

For things like the unit file parser, you may find it more convenient to
write the test in Scheme (currently all the tests are shell scripts.)
That can easily be done by using the .scm file name extension for your
test and then defining ‘SCM_LOG_COMPILER’ in Makefile.am.  If unsure,
you can look at how Guix itself does it, or just stop by on #guix or ask
on the list for details.

Some comments about the code:

> From a0a46ead5e43cd2672a08adb4c16919c377514c2 Mon Sep 17 00:00:00 2001
> From: Ioannis Panagiotis Koutsidis 
> Date: Sat, 9 Jun 2018 16:17:27 +0300
> Subject: [PATCH] Initial systemd unit support

Could you try to split it in several patches, where each patch
represents a single “logical” change?

By that I mean that you could have a first patch that modifies ‘restart’
and all in (shepherd service), possibly with extended tests to exercise
the new functionality if appropriate.

A second patch would add the unit file parser in (shepherd systemd)
along with its unit test.

For commit logs, please try to follow the ChangeLog convention:
.  You
can look at ‘git log’ and basically try to mimic what’s been done
before.  Don’t lose your hair over commit logs though; it’s good to try
to follow the conventions, but if you’re unsure or if you make mistakes,
it’s not the end of the world.

> @@ -165,6 +166,11 @@ respawned, shows that it has been respawned more than 
> TIMES in SECONDS."
>(respawn? #:init-keyword #:respawn?
>   #:init-value #f
>   #:getter respawn?)
> +  ;; For the systemd restart values.  Can be 'no (when respawn? is #f),
> +  ;; 'on-success, 'on-failure, 'on-abnormal, 'on-watchdog, 'on-abort, or 
> 'always
> +  (respawn-systemd #:init-keyword #:respawn-systemd
> +   #:init-value 'always
> +   #:getter respawn-systemd)

As briefly discussed on IRC, I think we should keep a single field for
this.  So perhaps ‘respawn?’ must simply be renamed to ‘respawn’ (no
question mark), with a comment like above explaining what the possible
values are.

> +  (let* ([e (status:exit-val status)]
> + [t (status:term-sig status)]
> + [r (respawn-systemd serv)]

Please avoid square brackets to remain consistent with the rest of the
code.  :-)

> + [clean (or (zero?  e)
> +(equal? t SIGHUP)
> +(equal? t SIGINT)
> +(equal? t SIGTERM)
> +(equal? t SIGPIPE))])

Use ‘=’ rather than ‘equal?’ when we know we’re dealing with numbers.

> +(if (or (equal? r 'always)
> +(equal? r 'on-watchdog) ;; not implemented yet
> +(and (equal? r 'on-success) clean)
> +(and (equal? r 'on-abnormal) (not clean) (equal? e #f))
> +(and (equal? r 'on-failure)  (not clean))
> +(and (equal? r 'on-abort)(equal? t SIGABRT)))

Likewise, use ‘eq?’ for symbols.

> +++ b/modules/shepherd/systemd.scm
> @@ -0,0 +1,143 @@
> +;; systemd.scm -- Systemd support
> +;; Copyright (C) 2018 Ioannis Panagiotis Koutsidis 
> +;;
> +;; This file is part of the GNU Shepherd.
> +;;
> +;; The GNU Shepherd is free software; you can redistribute it and/or modify 
> it
> +;; under the terms of the GNU General Public License as published by
> +;; the Free Software Foundation; either version 3 of the License, or (at
> +;; your option) any later version.
> +;;
> +;; The GNU Shepherd is distributed in the hope that it will be useful, but
> +;; WITHOUT ANY WARRANTY; without even the implied warranty of
> +;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +;; GNU General Public License for more details.
> +;;
> +;; You should have received a copy of the GNU General Public License
> +;; along with the GNU Shepherd.  If not, see .
> +
> +(define-module (shepherd systemd)
> +  #:use-module (ice-9 match)
> +  #:use-module (ice-9 textual-ports)
> +  #:use-module (oop goops)
> +  #:use-module (shepherd service)
> +  #:export 

GSoC 2018 Syntax and semantics of systemd units in the Shepherd - 1st update

[Ioannis Panagiotis Koutsidis]

Hi Guix!

As the 1st phase is coming to an end I decided to post my progress. I have
implemented the unit file parsing as well as some of the basic entries supported
by it, such as ExecStart, User, Group, Restart, etc. In addition, support for
the systemd Restart values (on-success, on-failure, on-abnormal, and on-abort)
was added to the Shepherd via the restart-systemd field in the  class,
letting services written in guile to also use that feature.

During the next phases I will focus on other common .service entries, .socket
support, as well as thoroughly testing the code.
From a0a46ead5e43cd2672a08adb4c16919c377514c2 Mon Sep 17 00:00:00 2001
From: Ioannis Panagiotis Koutsidis 
Date: Sat, 9 Jun 2018 16:17:27 +0300
Subject: [PATCH] Initial systemd unit support

---
 modules/shepherd/service.scm |  78 ---
 modules/shepherd/systemd.scm | 143 +++
 2 files changed, 194 insertions(+), 27 deletions(-)
 create mode 100644 modules/shepherd/systemd.scm

diff --git a/modules/shepherd/service.scm b/modules/shepherd/service.scm
index 93d3779..5b0d72d 100644
--- a/modules/shepherd/service.scm
+++ b/modules/shepherd/service.scm
@@ -4,6 +4,7 @@
 ;; Copyright (C) 2014 Alex Sassmannshausen 
 ;; Copyright (C) 2016 Alex Kost 
 ;; Copyright (C) 2018 Carlo Zancanaro 
+;; Copyright (C) 2018 Ioannis Panagiotis Koutsidis 
 ;;
 ;; This file is part of the GNU Shepherd.
 ;;
@@ -165,6 +166,11 @@ respawned, shows that it has been respawned more than 
TIMES in SECONDS."
   (respawn? #:init-keyword #:respawn?
#:init-value #f
#:getter respawn?)
+  ;; For the systemd restart values.  Can be 'no (when respawn? is #f),
+  ;; 'on-success, 'on-failure, 'on-abnormal, 'on-watchdog, 'on-abort, or 
'always
+  (respawn-systemd #:init-keyword #:respawn-systemd
+   #:init-value 'always
+   #:getter respawn-systemd)
   ;; The action to perform to start the service.  This must be a
   ;; procedure and may take an arbitrary amount of arguments, but it
   ;; must be possible to call it without any argument.  If the
@@ -270,7 +276,7 @@ wire."
 (define-method (running? (obj ))
   (and (slot-ref obj 'running) #t))
 
-;; Return a list of all actions implemented by OBJ. 
+;; Return a list of all actions implemented by OBJ.
 (define-method (action-list (obj ))
   (map action-name (slot-ref obj 'actions)))
 
@@ -886,9 +892,12 @@ start."
 ;; Produce a destructor that sends SIGNAL to the process with the pid
 ;; given as argument, where SIGNAL defaults to `SIGTERM'.
 (define make-kill-destructor
-  (lambda* (#:optional (signal SIGTERM))
+  (lambda* (#:optional (signal SIGTERM)
+   (timeout #f))
 (lambda (pid . args)
   (kill pid signal)
+  ;; TODO: Make sure that the process has actually stopped by timeout.
+  ;; If it has not, send a SIGKILL
   #f)))
 
 ;; Produce a constructor that executes a command.
@@ -996,7 +1005,7 @@ otherwise by updating its state."
   ((0 . _)
;; Nothing left to wait for.
#t)
-  ((pid . _)
+  ((pid . status)
(let ((serv (find-service (lambda (serv)
(and (enabled? serv)
 (match (slot-ref serv 'running)
@@ -1007,13 +1016,13 @@ otherwise by updating its state."
  ;; SERV can be #f for instance when this code runs just after a
  ;; service's 'stop' method killed its process and completed.
  (when serv
-   (respawn-service serv))
+   (respawn-service serv status))
 
  ;; As noted in libc's manual (info "(libc) Process Completion"),
  ;; loop so we don't miss any terminated child process.
  (loop))
 
-(define (respawn-service serv)
+(define (respawn-service serv status)
   "Respawn a service that has stopped running unexpectedly. If we have
 attempted to respawn the service a number of times already and it keeps dying,
 then disable it."
@@ -1022,22 +1031,37 @@ then disable it."
(not (respawn-limit-hit? (slot-ref serv 'last-respawns)
 (car respawn-limit)
 (cdr respawn-limit
-  (if (not (slot-ref serv 'waiting-for-termination?))
-  (begin
-;; Everything is okay, start it.
-(local-output "Respawning ~a."
-  (canonical-name serv))
-(slot-set! serv 'last-respawns
-   (cons (current-time)
- (slot-ref serv 'last-respawns)))
-(start serv))
-  ;; We have just been waiting for the
-  ;; termination.  The `running' slot has already
-  ;; been set to `#f' by `stop'.
-  (begin
-(local-output "Service ~a terminated."
-  (canonical-name serv))
-(slot-set! serv 'waiting-for-termination? #f)))
+  (let* ([e (status:exit-val status)]
+  

Re: GSoC-2018 - Build daemon rewrite in Scheme Updates

[Ludovic Courtès]

Hi uniq10,

l...@gnu.org (Ludovic Courtès) skribis:

> Sandeep Subramanian  skribis:
>
>> `build-derivation` is now using `call-with-container`
>> and it seems to work well. `build-derivation` assumes
>> that all the inputs needed for the derivation is already
>> built and ready. As a next step I wanted to try and build
>> the `inputDrvs` if the inputs are missing.
>
> Nice, it sounds like a good start.
>
> Where is the code you’ve been working on?  Is it
> ?

It’s been more than a week and I haven’t seen any reply from you.
Could you please reply now?

As you know, GSoC is a full-time commitment so it’s a bad sign that
we’re not hearing from you.

Thanks in advance,
Ludo’.

Re: GSoC-2018 - Build daemon rewrite in Scheme Updates

[Ludovic Courtès]

Hello uniq10,

Thanks for the update!

Sandeep Subramanian  skribis:

> `build-derivation` is now using `call-with-container`
> and it seems to work well. `build-derivation` assumes
> that all the inputs needed for the derivation is already
> built and ready. As a next step I wanted to try and build
> the `inputDrvs` if the inputs are missing.

Nice, it sounds like a good start.

Where is the code you’ve been working on?  Is it
?

In general, I would encourage you to follow the project’s conventions as
closely as possible to ease future merges.  See
.
Please also consider showing up on #guix on IRC to share what you’re up
to and ask for advice whenever you feel like it.  Communication is key
is a distributed project like this.

As discussed off-list, I just started merging into ‘master’ proper work
from last year: .  This is an area where
we’ll obviously have to coordinate.  For example it’d be great if you
could rebase onto master once 31618 has been merged, and let me know if
anything is amiss.

> I looked through the C++ code to find out how exactly
> this is done and I have a good understanding of it now.
> I plan to emulate the behaviour of the C++ code for
> the entire derivation building pipeline.
>
> I spent a lot of time browsing the C++ code base and had
> to keep going back to it often, since I had trouble keeping
> the C++ code flow in memory. So, I thought it would be
> worthwhile to write some detailed notes on the C++ daemon
> code base that I can refer to while coding. The notes I made
> can be found here:
> https://github.com/uniq10/gsoc-guix-notes/blob/master/daemon-notes.md

Woow, very nice!  From a quick look it does seem to correctly reflect
what’s happening in that code.  It would have helped me a lot to have
such a document back then.  ;-)

BTW, note that you shouldn’t take the structure of the C++ code as a
reference to follow.  In Scheme we would more naturally use different
idioms than those of this code (object-oriented C++).  You’re welcome to
discuss on this list or on IRC how to structure things.

Thanks for the update and for the nice work!

Ludo’.

GSoC-2018 - Build daemon rewrite in Scheme Updates

[Sandeep Subramanian]

Hi all,

It has been two weeks since I started working on
the project and I thought I will give you all a quick
update regarding it.

`build-derivation` is now using `call-with-container`
and it seems to work well. `build-derivation` assumes
that all the inputs needed for the derivation is already
built and ready. As a next step I wanted to try and build
the `inputDrvs` if the inputs are missing.

I looked through the C++ code to find out how exactly
this is done and I have a good understanding of it now.
I plan to emulate the behaviour of the C++ code for
the entire derivation building pipeline.

I spent a lot of time browsing the C++ code base and had
to keep going back to it often, since I had trouble keeping
the C++ code flow in memory. So, I thought it would be
worthwhile to write some detailed notes on the C++ daemon
code base that I can refer to while coding. The notes I made
can be found here:
https://github.com/uniq10/gsoc-guix-notes/blob/master/daemon-notes.md

If anybody find any mistakes in those notes kindly let me know.

--
uniq10

Re: GSoC-2018

[Sandeep Subramanian]

Hi reepca,

Thank you!

I will keep in touch with you on #guix.

--
uniq10
On Sun, Apr 29, 2018 at 3:08 AM Caleb Ristvedt 
wrote:

> Hi Sandeep/uniq10, welcome!

> I'm the person that worked on the rewrite last year (I like to go by
> reepca in conversations and irc). I'm glad that you'll be working on it
> this summer. I still feel like I didn't do enough last summer, though,
> so if there's any way I can help, I'd be glad to (though the next couple
> of days are finals here, so I'll be rather busy just then).

> If you have any questions about the code or find yourself reading
> through a bunch of C++ you'd rather not be reading through and thinking
> to yourself "that other guy probably already had to do this...", feel
> free to ask. I'm in #guix pretty much all the time (though only on my
> desktop, so if I'm away the response may be delayed), though you'll
> probably have to ping me.

> - reepca

Re: GSoC-2018

[Ludovic Courtès]

Welcome Sandeep, and thank you reepca for offering to help!

Ludo’.

Re: GSoC-2018

[Pierre Neidhardt]

This is a great idea, I'd really love to see this come true!

Good luck and happy hacking!

-- 
Pierre Neidhardt

I'm glad we don't have to play in the shade.
-- Golfer Bobby Jones on being told that it was 105 degrees
   in the shade.


signature.asc
Description: PGP signature

Re: GSoC-2018

[Caleb Ristvedt]

Hi Sandeep/uniq10, welcome!

I'm the person that worked on the rewrite last year (I like to go by
reepca in conversations and irc). I'm glad that you'll be working on it
this summer. I still feel like I didn't do enough last summer, though,
so if there's any way I can help, I'd be glad to (though the next couple
of days are finals here, so I'll be rather busy just then).

If you have any questions about the code or find yourself reading
through a bunch of C++ you'd rather not be reading through and thinking
to yourself "that other guy probably already had to do this...", feel
free to ask. I'm in #guix pretty much all the time (though only on my
desktop, so if I'm away the response may be delayed), though you'll
probably have to ping me.

- reepca

Re: GSoC-2018

[Kei Kebreau]

Sandeep Subramanian <sandeepsubramania...@gmail.com> writes:

> Hi all,
>
> I am Sandeep Subramanian (uniq10) and I have been selected for the
> GSoC-2018 project "Continue rewrite build daemon in Guile Scheme".
> (Project description:
> https://libreplanet.org/wiki/Group:Guix/GSoC-2018#Continue_rewrite_build_daemon_in_Guile_Scheme
> )
>
> I am very excited to work on this project and am looking forward to
> interacting
> with the community here and on the #guix IRC channel (Nick: uniq10).
> I will mostly be active from UTC 08:00 to UTC 20:00.
>
> I hope to, with your help, successfully complete this project and make a
> meaningful contribution to the community.

Welcome!


signature.asc
Description: PGP signature

Re: GSoC-2018

[Catonano]

2018-04-27 19:43 GMT+02:00 Sandeep Subramanian <
sandeepsubramania...@gmail.com>:

> Hi all,
>
> I am Sandeep Subramanian (uniq10) and I have been selected for the
> GSoC-2018 project "Continue rewrite build daemon in Guile Scheme".
> (Project description:
> https://libreplanet.org/wiki/Group:Guix/GSoC-2018#Continue_
> rewrite_build_daemon_in_Guile_Scheme
> )
>
> I am very excited to work on this project and am looking forward to
> interacting
> with the community here and on the #guix IRC channel (Nick: uniq10).
> I will mostly be active from UTC 08:00 to UTC 20:00.
>
> I hope to, with your help, successfully complete this project and make a
> meaningful contribution to the community.
>
>
welcome and good luck !!

GSoC-2018

[Sandeep Subramanian]

Hi all,

I am Sandeep Subramanian (uniq10) and I have been selected for the
GSoC-2018 project "Continue rewrite build daemon in Guile Scheme".
(Project description:
https://libreplanet.org/wiki/Group:Guix/GSoC-2018#Continue_rewrite_build_daemon_in_Guile_Scheme
)

I am very excited to work on this project and am looking forward to
interacting
with the community here and on the #guix IRC channel (Nick: uniq10).
I will mostly be active from UTC 08:00 to UTC 20:00.

I hope to, with your help, successfully complete this project and make a
meaningful contribution to the community.