Re: [PATCH] services: lsh: Add graceful handling of daemonic option.
On 6 Dec 2014 15:28, Ludovic Courtès l...@gnu.org wrote: Deck Pickard deck.r.pick...@gmail.com skribis: From 1fef935d6292016c04b9234eedb5dcaf006dc152 Mon Sep 17 00:00:00 2001 From: nebuli nebu@kipple Date: Wed, 3 Dec 2014 22:51:48 +0100 Subject: [PATCH] services: lsh: Add graceful handling of daemonic option. * doc/guix.texi: Mention use case. * gnu/services/ssh.scm (lsh-service): New #:keys (daemonic?, pid-file?, pid-file). Build new lshd-command and expand service-requirement field. Nice! (define* (lsh-service #:key (lsh lsh) + (daemonic? #f) (host-key /etc/lsh/host-key) (interfaces '()) (port-number 22) (allow-empty-passwords? #f) (root-login? #f) (syslog-output? #t) + (pid-file? #f) + (pid-file /var/run/lshd.pid) (x11-forwarding? #t) (tcp/ip-forwarding? #t) (password-authentication? #t) I would be tempted to not expose #:daemonic?, #:pid-file? and #:syslog-output?, and instead always use --daemonic --pid-file=... In particular, when using --daemonic, having the PID file is required, otherwise dmd won’t know what the PID of this process is, and thus will be unable to control it. For that reason, #:pid-file? must not be exposed. WDYT? I implemented this because, from what I gather, lshd will write to syslog only in '--daemonic' mode, otherwise it spams the controlling terminal on which dmd is running. And I wanted lsh to use syslog! As it is now, dmd captures the right PID from the make-fork constructor alone, while having no idea of pid files; I went as far as to write dmd service (and 'deco sideloding' it), which printed out both PIDs, they were eqv... There might still remain a use case with daemonic? equal to false for someone out there, even for simple reason of lack of functioning syslog (as well as a use case of choosing not to log at all), shrug... Change default to (daemonic? #t) and adjust the docs? Your call... I did not mention pid file related keys in the docs, because it would be only useful to someone who had to bother to look at actual lsh-service signature, like someone who did need pid file for some strange purpose... + (define requires +(if (and daemonic? syslog-output?) +'(networking syslogd) +'(networking))) If we agree on the above, that would become '(networking syslogd) unconditionally. No, as I explained; one thing is having a chosen set of defaults, another removing flexibility... lsh and/or dmd behaviour could change or someone could like to rewrite lsh service definition. (return (service (documentation GNU lsh SSH server) (provision '(ssh-daemon)) - (requirement '(networking)) + (requirement #~(#$@requires)) This is strictly equivalent to: (requirement `(,@requires)) or simply: (requirement requires) :-) G-expressions are only needed when capturing references to /gnu/store items, packages, etc. Thanks, Ludo’. Roger, still groking my way around, at least it doesn't matter apart from couple useless macro expansions. Drp, -- (or ((,\ (x) `(,x x)) '(,\ (x) `(,x x))) (smth (that 'like)))
[PATCH] services: lsh: Add graceful handling of daemonic option.
#~(#$@ looks freaky, but if this is what it takes... Tried couple of other figures, this one appears to generate right dmd.conf, though I haven't had yet a chance to reboot. Drp, -- (or ((,\ (x) `(,x x)) '(,\ (x) `(,x x))) (smth (that 'like))) 0001-services-lsh-Add-graceful-handling-of-daemonic-optio.patch Description: Binary data
[PATCH] guix: scripts: Fix GUIX_BUILD_OPTIONS handling.
Perhaps, now it WILL twerk. Drp, -- (or ((,\ (x) `(,x x)) '(,\ (x) `(,x x))) (smth (that 'like))) 0001-guix-scripts-Fix-GUIX_BUILD_OPTIONS-handling.patch Description: Binary data
Re: [PATCH] Core sanity and taking build options from environment.
On 28 Nov 2014 22:52, Ludovic Courtès l...@gnu.org wrote:
Deck Pickard deck.r.pick...@gmail.com skribis:
From 3693753aefc27b5a68a2b762feeebc41320e79ef Mon Sep 17 00:00:00 2001
From: nebuli nebu@kipple
Date: Wed, 26 Nov 2014 19:51:37 +0100
Subject: [PATCH 1/2] guix: Default to daemon's default --cores setting
of 1.
I think most of the time one would prefer to use all the available cores
when building. WDYT?
Perhaps for one-off 15 minutes worth of compiles it's a OK, but not for
hours of work. And you are creating more confusion by having different
defaults from guix-daemon. Lastly, some unsuspecting newb may still
stumble into N^2 trap with '-M my number of shiny cores'... ouch, 36
threads competing for silicon on a 6 core box...
From fa8738ff2cf48886c9ef8fbacfa806f547f3c3c8 Mon Sep 17 00:00:00 2001
From: nebuli nebu@kipple
Date: Thu, 27 Nov 2014 23:36:29 +0100
Subject: [PATCH 2/2] guix: scripts: Add handling of options from
GUIX_BUILD
environmental variable.
[...]
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -401,7 +401,13 @@ arguments with packages that use the specified
source.
(define (guix-build . args)
(define (parse-options)
;; Return the alist of option values.
-(args-fold* args %options
+(args-fold* (append args
+(args-from-env GUIX_BUILD
+ (lambda (var opts)
+ (format (current-error-port)
+ (_ guix build: ~a:
~a~%)
+ var opts
+%options
This sounds like a good idea.
Some suggestions that come to mind:
• What about $GUIX_BUILD_OPTIONS? (grep uses $GREP_OPTIONS, for
reference.)
• What about factorizing the above ‘args-from-env’ call like this:
(define (environment-build-options)
Return additional build options passed as environment variables.
(args-from-env GUIX_BUILD_OPTIONS))
This procedure would go in (guix ui).
• I would leave out the second argument to ‘args-from-env’. I don’t
think it would be convenient to have that extra line printed every
time.
+(define (args-from-env var . rest)
+ Retrieve value of environment variable denoted by string VAR in the
form
+of a list of strings ('char-set:graphic' tokens) suitable for
consumption by
+the fold-arg family of functions. If VAR is defined, call car of a
non-null
s/fold-arg family of functions/'args-fold'/
+REST on the VAR and result, otherwise return an empty list.
+ (let ((env-opts (getenv variable)))
+(if env-opts
+(let ((opts (string-tokenize env-opts char-set:graphic)))
+ (and (not (null? rest))
+ (apply (car rest)
+ (list variable opts)))
+ opts)
+'(
Please write it like this:
(define (arguments-from-environment-variable variable)
(let ((env (getenv variable)))
...))
Could you also update guix.texi, near the end of “Invoking guix build”?
WDYT? Could you send an updated patch?
DONE,
Drp
--
(or ((,\ (x) `(,x x)) '(,\ (x) `(,x x))) (smth (that 'like)))
0001-guix-scripts-Add-GUIX_BUILD_OPTIONS-environment-hand.patch
Description: Binary data
[PATCH] Core sanity and taking build options from environment.
Perhaps GUIX_BUILD (suffix with _OPTS or some such?) is not the best name and I'm not certain if it shouldn't be stdout (why are we using error port for normal communication? Emacs?), but I think it might prove useful if user keeps being reminded his environment is polluted or not as his intentions might have been, or it could go only into guix|guix --help... To doc or not to doc, Drp -- .sig place holder 0001-guix-Default-to-daemon-s-default-cores-setting-of-1.patch Description: Binary data 0002-guix-scripts-Add-handling-of-options-from-GUIX_BUILD.patch Description: Binary data
Re: [PATCH] Hotfix (repeat)
On 23 Nov 2014 21:49, Ludovic Courtès l...@gnu.org wrote: Deck Pickard deck.r.pick...@gmail.com skribis: From 8e297904d80b39cd510ba0cced37acdb9b1aeb89 Mon Sep 17 00:00:00 2001 From: nebuli nebu@kipple Date: Sat, 22 Nov 2014 19:58:24 +0100 Subject: [PATCH 2/4] guix build: Add --max-jobs option (without handling code). * doc/guix.texi: Mention in the docs. * guix/scripts/build.scm: Extend (show-build-options-help) and (%standard-build-options) functions. Actually I had overlooked that this patch does nothing. :-) Could you send an updated version that passes the right option to ‘set-build-options’? No. Using '-c 0 -M 0' fails with cryptic message. On a four core system innocent (and logically consistent, I mean, from the description of those options, one expects the daemon to do some load balancing) '-c 4 -M 4' ends up with the same annoying N^2 behaviour. As it is now, not using one of the options leads to sub-optimal saturation of the through-output by default. Not very impressive, if you want to attract hackers who are willing to spend their time and resources to actually build locally from source. After all it's the only way to test and find possible bugs on a wide set of possible configurations. If you want it right, either fix it yourself (and please think hard and carefully what to put in '-from-commandline' function if you want to expose both options to the user) or stop with the antics and apply the patch. I can live with constant branch rebasing, but will end users appreciate their machines locking up? I mean every proper Linux user is expected to tail their logs... Unimpressed, drp -- .sig place holder
[PATCH 2/4] guix build: Add --max-jobs option (without handling code).
* doc/guix.texi: Mention in the docs.
* guix/scripts/build.scm: Extend (show-build-options-help) and
(%standard-build-options) functions.
---
doc/guix.texi | 5 +
guix/scripts/build.scm | 14 --
2 files changed, 17 insertions(+), 2 deletions(-)
diff --git a/doc/guix.texi b/doc/guix.texi
index 2a33cb5..02edee0 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -2676,6 +2676,11 @@ may be helpful when debugging setup issues with the
build daemon.
Allow the use of up to @var{n} CPU cores for the build. The special
value @code{0} means to use as many CPU cores as available.
+@item --max-jobs=@var{n}
+@itemx -M @var{n}
+Allow at most @var{n} build jobs. The special value @code{0} means to
+create as many jobs as the number of available CPU cores.
+
@end table
Behind the scenes, @command{guix build} is essentially an interface to
diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm
index 7b7f419..d10b95b 100644
--- a/guix/scripts/build.scm
+++ b/guix/scripts/build.scm
@@ -119,7 +119,9 @@ options handled by
'set-build-options-from-command-line', and listed in
(display (_
--verbosity=LEVEL use the given verbosity LEVEL))
(display (_
- -c, --cores=N allow the use of up to N CPU cores for the build)))
+ -c, --cores=N allow the use of up to N CPU cores for the build))
+ (display (_
+ -M, --max-jobs=N allow at most N build jobs)))
(define (set-build-options-from-command-line store opts)
Given OPTS, an alist as returned by 'args-fold' given
@@ -192,7 +194,15 @@ options handled by
'set-build-options-from-command-line', and listed in
(let ((c (false-if-exception (string-number arg
(if c
(apply values (alist-cons 'cores c result) rest)
- (leave (_ ~a: not a number~%) arg)))
+ (leave (_ not a number: '~a' option argument: ~a~%)
+ name arg)
+ (option '(#\M max-jobs) #t #f
+ (lambda (opt name arg result . rest)
+ (let ((c (false-if-exception (string-number arg
+ (if c
+ (apply values (alist-cons 'max-jobs c result) rest)
+ (leave (_ not a number: '~a' option argument: ~a~%)
+ name arg)))
;;;
--
2.1.2
[PATCH 1/4] store: default to serial scheduler
* guix/store.scm (set-build-options): exchange default argument values --- guix/store.scm | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/guix/store.scm b/guix/store.scm index bc4c641..571cc06 100644 --- a/guix/store.scm +++ b/guix/store.scm @@ -435,14 +435,14 @@ encoding conversion errors. (define* (set-build-options server #:key keep-failed? keep-going? fallback? (verbosity 0) - (max-build-jobs (current-processor-count)) + (max-build-jobs 1) timeout (max-silent-time 3600) (use-build-hook? #t) (build-verbosity 0) (log-type 0) (print-build-trace #t) - (build-cores 1) + (build-cores (current-processor-count)) (use-substitutes? #t) (binary-caches '())) ; client untrusted cache URLs ;; Must be called after `open-connection'. -- 2.1.2
[PATCH 4/4] guix build: Try to handle --cores and --max-jobs in a sane way.
* guix/scripts/build.scm (set-build-options-from-command-line): use make-schedule-sane to parse and set scheduling options. --- guix/scripts/build.scm | 24 ++-- 1 file changed, 14 insertions(+), 10 deletions(-) diff --git a/guix/scripts/build.scm b/guix/scripts/build.scm index d10b95b..31f17d2 100644 --- a/guix/scripts/build.scm +++ b/guix/scripts/build.scm @@ -20,6 +20,7 @@ (define-module (guix scripts build) #:use-module (guix ui) #:use-module (guix store) + #:use-module (guix schedule) #:use-module (guix derivations) #:use-module (guix packages) #:use-module (guix utils) @@ -127,16 +128,19 @@ options handled by 'set-build-options-from-command-line', and listed in Given OPTS, an alist as returned by 'args-fold' given '%standard-build-options', set the corresponding build options on STORE. ;; TODO: Add more options. - (set-build-options store - #:keep-failed? (assoc-ref opts 'keep-failed?) - #:build-cores (or (assoc-ref opts 'cores) 0) - #:fallback? (assoc-ref opts 'fallback?) - #:use-substitutes? (assoc-ref opts 'substitutes?) - #:use-build-hook? (assoc-ref opts 'build-hook?) - #:max-silent-time (assoc-ref opts 'max-silent-time) - #:timeout (assoc-ref opts 'timeout) - #:print-build-trace (assoc-ref opts 'print-build-trace?) - #:verbosity (assoc-ref opts 'verbosity))) + (let ((sched (make-schedule-sane #:max-cores (assoc-ref opts 'cores) + #:max-jobs (assoc-ref opts 'max-jobs + (set-build-options store + #:keep-failed? (assoc-ref opts 'keep-failed?) + #:build-cores (schedule-max-cores sched) + #:max-build-jobs (schedule-max-jobs sched) + #:fallback? (assoc-ref opts 'fallback?) + #:use-substitutes? (assoc-ref opts 'substitutes?) + #:use-build-hook? (assoc-ref opts 'build-hook?) + #:max-silent-time (assoc-ref opts 'max-silent-time) + #:timeout (assoc-ref opts 'timeout) + #:print-build-trace (assoc-ref opts 'print-build-trace?) + #:verbosity (assoc-ref opts 'verbosity (define %standard-build-options ;; List of standard command-line options for tools that build something. -- 2.1.2
[PATCH 3/4] guix: Add schedule module.
* guix/schedule.scm: New file. To handle --cores and --max-jobs options in 'guix build'. * Makefile.am (MODULES): Add *this. --- Makefile.am | 1 + guix/schedule.scm | 102 ++ 2 files changed, 103 insertions(+) create mode 100644 guix/schedule.scm diff --git a/Makefile.am b/Makefile.am index 5c4ce90..1806a05 100644 --- a/Makefile.am +++ b/Makefile.am @@ -56,6 +56,7 @@ MODULES = \ guix/ftp-client.scm \ guix/http-client.scm \ guix/gnupg.scm \ + guix/schedule.scm \ guix/store.scm \ guix/svn-download.scm \ guix/ui.scm \ diff --git a/guix/schedule.scm b/guix/schedule.scm new file mode 100644 index 000..26c7b6b --- /dev/null +++ b/guix/schedule.scm @@ -0,0 +1,102 @@ +;;; GNU Guix --- Functional package management for GNU +;;; Copyright © 2014 Nebulieu nebu@kipple +;;; +;;; This file is part of GNU Guix. +;;; +;;; GNU Guix is free software; you can redistribute it and/or modify it +;;; under the terms of the GNU General Public License as published by +;;; the Free Software Foundation; either version 3 of the License, or (at +;;; your option) any later version. +;;; +;;; GNU Guix is distributed in the hope that it will be useful, but +;;; WITHOUT ANY WARRANTY; without even the implied warranty of +;;; MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +;;; GNU General Public License for more details. +;;; +;;; You should have received a copy of the GNU General Public License +;;; along with GNU Guix. If not, see http://www.gnu.org/licenses/. + +(define-module (guix schedule) + #:use-module (guix records) + #:export (schedule? + schedule + schedule-name + schedule-max-cores + schedule-max-jobs + schedule-override-cores? + schedule-override-jobs? + + make-schedule-sane)) + +(define-record-type* schedule + schedule make-schedule + schedule? + (name schedule-name ; symbol + (default 'serial)) + (max-cores schedule-max-cores ; non-negative integer + (default 0)) ; use all available horse-power + (max-jobs schedule-max-jobs ; non-negative integer + (default 1)) ; there can be only one + ;; unused, for now, rethink unified override + (override-cores? schedule-override-cores? ; boolean + (default #t)) + (override-jobs? schedule-override-jobs? ; boolean + (default #t))) + +; will rather need one `make-schedule-with-name` and switch on 'symbol +(define (make-schedule-serial) + (schedule)) +; redundant for now... + +; macro? +(define (real-schedule symname cores jobs + override-c override-j) + (schedule + (name symname) + (max-cores cores) + (max-jobs jobs) + (override-cores? override-c) + (override-jobs? override-j))) + +; better name? +(define (1 num) + (if ( num 1) + 1 + num)) + +;; TODO: increase number of jobs with spare cores??? perhaps in real-schedule +(define* (make-schedule-sane #:key max-cores max-jobs) + (let ((sym-name 'serial-sane) + ;; should overriding one override both (think: yes, e.g. + ;; setting cores to max with guix-daemon default [max-jobs = 0] + ;; will again lead to the N^2 phenomenon... + (override-cores (and max-cores #t)) + (override-jobs (and max-jobs #t)) + ;; scheduling needs be centralized (think: override always) + (max-threads (min (current-processor-count) + (total-processor-count + (let ((default-max-cores (1 (- max-threads 1))) + (default-max-jobs 1) + (validate (lambda (arg default) + (if (or (not arg) + (not (integer? arg))) + default + (or (and (= arg 0) max-threads) + (and ( arg 0) default) + arg) + ;; perhaps we shouldn't be so symmetrical? + (let ((cores (validate max-cores default-max-cores)) + (jobs (validate max-jobs default-max-jobs))) + ;; cut-off + (let loop ((c (min cores max-threads)) + (j (min jobs max-threads))) + (let ((threads (* c j))) + (if (= threads max-threads) + (real-schedule sym-name c j + override-cores override-jobs) + ;; maximize cores at the cost of jobs... + (let ((j- (1 (- j 1 + (if (= (* c j-) max-threads) + (real-schedule sym-name c j- + override-cores override-jobs) + (loop (1 (- c 1)) j-)) -- 2.1.2
[PATCH 0/4] Fix 'guix build' local overheating
Aka No, Laptot, no fry... nebuli (4): store: default to serial scheduler guix build: Add --max-jobs option (without handling code). guix: Add schedule module. guix build: Try to handle --cores and --max-jobs in a sane way. Makefile.am | 1 + doc/guix.texi | 5 +++ guix/schedule.scm | 102 + guix/scripts/build.scm | 38 -- guix/store.scm | 4 +- 5 files changed, 136 insertions(+), 14 deletions(-) create mode 100644 guix/schedule.scm -- 2.1.2
