Security announcement for old PC-BSD/TrueOS machines

[Jonathan de Boyne Pollard]

A small security announcement for people who have old PC-BSD/TrueOS 
machines running, or that you might boot up and run:


* https://tty0.social/@JdeBP/110204963467594963

Details of the problem, of the service fix, and of local fixes that you 
can apply for both my service management system and the original one in 
PC-BSD/TrueOS, as well as pointers to other affected systems (for which 
you will have to work out what to do yourself), are in the announcement.


If you are one of the people on this mailing list that maintains service 
definitions of some kind, and PC-BSD/TrueOS is within your remit, please 
consider also disabling these services by default.

Re: Systemd unit parser and s6 generator

[Jonathan de Boyne Pollard]

[...]  We've discussed internally if we change that process and try to 
write a systemd unit parser, because all units are there in Ubuntu. If 
we could catch 90% of all cases, we need, we would be happy. If it 
would take 2 weeks of work, that would be fine. Did somebody of you 
try to implement something? What are your thoughts?


So it seems that I am, years later, still the only person in the world 
to have done this (except not targetting skarnet toolsets). (-:


As the voice of experience:

Yes, you'll not get total coverage.  You'll get "good enough for most 
cases", though I couldn't put an exact percentage on that.


No, with the s6 tooling as it stands, you'll not be able to do 
everything.  One of the things that one has to add is support for 
systemd's idiosyncratic non-UCSPI TCP/UDP service handling. Contrast the 
--systemd-compatibility flag to the tcp-socket-listen program in the 
nosh toolset to what s6-tcpserver4-socketbinder in s6-networking does.


* https://jdebp.uk/Softwares/nosh/guide/commands/tcp-socket-listen.xml

* https://skarnet.org/software/s6-networking/s6-tcpserver4-socketbinder.html

That flag is used by convert-systemd-units to handle the conversion of 
socket units.


* https://jdebp.uk/Softwares/nosh/guide/commands/convert-systemd-units.xml

Most of the issues for s6 will be things like that, minor tweaks for 
(sometimes reasonable, sometimes downright absurd) systemd quirks and 
the odd missing tool.  I wrote a set-control-group-knob command, for 
example, used by convert-systemd-units to translate the various unit 
file settings that end up twiddling knobs on control groups.  I haven't 
seen a skarnet tool for doing that, yet; so that will be one necessary 
addition, for example.


* https://jdebp.uk/Softwares/nosh/guide/commands/set-control-group-knob.xml

s6-networking also does not appear to have UDP listening tools, which 
socket units for UDP sockets need.  Timer units need tools like 
time-pause-until and time-env-add, similarly.  Notice the 
--systemd-compatibility flag for that latter, too, and the notions like 
systemd month lengths.


* https://jdebp.uk/Softwares/nosh/guide/commands/time-pause-until.xml

* https://jdebp.uk/Softwares/nosh/guide/commands/time-env-add.xml

There is, furthermore, a lot of stuff that you'll find that s6/nosh/&c. 
service run scripts do a lot better than systemd can.  As you'll see 
from the convert-systemd-units(1) manual page, I added quite a number of 
things to the systemd unit file syntax for things that one can do in the 
aforementioned, such as UCSPI access control mechanisms (UCSPI being 
entirely alien to systemd) for starters.

Re: [svlogd] / -ttt / why UTC?

[Jonathan de Boyne Pollard]

Yes. You cannot set your system clock to TAI, unless you want wildly 
incorrect results from time() and similar system calls. Setting it 10 
seconds earlier than TAI is the best you can do; and that's what the 
right/ timezones expect.




In my world time() returns the SI seconds since the start of 1970. 
Since TAI and UTC were off for fractions of a second from 1970 to the 
end of 1972 might be true, but my applications dont care for that time 
so much...




M. Bercot's point, which you edited out, is that what xe means by 
"TAI-10" is exactly that, because what you say here is untrue. TAI was 
different from UTC by over 10 seconds, not by merely fractions of 1 
second, and the "start of 1970" in one was not the "start of 1970" in 
the other.  People don't use TAI in Unices and Linux-based operating 
systems.  They use what is more accurately termed TAI-10; because there 
aren't versions of the timezone support data files (supplied "out of the 
box") that count SI seconds and use "start of 1970" TAI, only ones that 
count SI seconds and use "start of 1970" UTC, i.e. "start of 1970" TAI-10.


* https://unix.stackexchange.com/a/327403/5132

And whereas on an operating system like OpenBSD one should expect both 
"right" and "posix" timezones to be supported, and any inconsistencies 
in applications to be corrected with patches in ports, on Linux-based 
operating systems the hotch-potch nature of the various language runtime 
libraries and application repositories does not given one nearly as much 
confidence that it's all correct.


Well done for trying to push corrections back to the authors; but that 
makes you in a better position than most to agree with M. Bercot that 
the state of the applications and Linux-based operating systems as 
supplied leaves a little to be desired for TAI-10 operation.  But as M. 
Bercot and I both say, the major single offender here is {x,}ntpd.



I think, i will write socklog&svlogd myself...



Personally, I just go with the Bernstein timestamps.  Then I can convert 
to whatever human-readable form that I like, without touching the raw 
log files or having to worry about parsing log files with 
locale-dependent formats in them; even reading the same log entries in 
multiple different timezones and locales if I really want to.

Re: Why are there 3 programs in s6 that do the same thing?

[Jonathan de Boyne Pollard]

Laurent Bercot:


[...] direct adaptations from daemontools' setuidgid and envuidgid [...]



There is a similar history, and a similar spread of tools in the nosh 
toolset for roughly the same reasons.  There was a need for a split 
toolset where one could look up things in the system account database, 
do stuff (in particular stuff that obscured access to the system account 
database), and then later change the process user/group IDs.  There was 
also a need for tools that did not manipulate the user ID but only the 
group IDs.  Hence envgid, envuidgid, getuidgid, setgid-fromenv, and 
setuidgid-fromenv; with setuidgid effectively being envuidgid followed 
by setuidgid-fromenv (except that it does not touch the environment).


Uwe Ohse did the first setuidgid-fromenv tool, I believe, as a patch to 
daemontools back in 2001.  Several people have developed tools in this way.


* http://jdebp.uk./Softwares/nosh/guide/commands/setuidgid.xml

* http://jdebp.uk./Softwares/nosh/guide/commands/envuidgid.xml

* https://ohse.de/uwe/patches.html

Re: runit: run process in a tty

[Jonathan de Boyne Pollard]

Kian Kasad:


Unable to open file: /dev/tty2: Operation not permitted.



Operating systems do a whole bunch of stuff when connecting user 
programs to terminals, including changing the permissions of the 
terminal device file to allow the logged-in user access.  If you check 
the terminal device file of a terminal where you have actually logged 
in, you will find that it is owned by you, and that the device files of 
terminals where you have not logged in are owned by the superuser.  This 
is one of several things that the login process does.


Taking ly's supplied systemd service unit and converting it using the 
nosh toolset's convert-systemd-units command 
(http://jdebp.uk./Softwares/nosh/guide/commands/convert-systemd-units.xml), 
one obtains the following, which demonstrates that there are extra steps 
involved and provides at least a pointer to how a "run" script for a 
runit service should be constructed.



% convert-systemd-units --no-systemd-quirks ./ly.service
convert-systemd-units: WARNING: ./ly.service: Unused setting: [install] alias = 
display-manager.service
%
% system-control print-service-scripts ./ly
start:#!/bin/nosh
start:#Start file generated from ./ly.service
start:true
stop:#!/bin/nosh
stop:#Stop file generated from ./ly.service
stop:true
run:#!/bin/nosh
run:#Run file generated from ./ly.service
run:#TUI display manager
run:vc-get-tty /dev/tty2
run:open-controlling-tty --revoke
run:vc-reset-tty --hard-reset
run:/usr/bin/ly
restart:#!/bin/sh
restart:#Restart file generated from ./ly.service
restart:exec true# ignore script arguments
%


Notice how the service invokes the open-controlling-tty program 
(http://jdebp.uk./Softwares/nosh/guide/commands/open-controlling-tty.xml) 
to open the terminal device file as standard I/O and to make it the 
service's controlling terminal.  It knows what terminal device because 
that has been set up by the vc-get-tty program 
(http://jdebp.uk./Softwares/nosh/guide/commands/vc-get-tty.xml) which 
was passed the name "/dev/tty2".


Notice also how the service runs the "ly" program as the superuser.  It 
does not drop privileges by switching to an unprivileged account.


Furthermore, notice that there are two separate places where one tells 
"ly" what terminal to use: in the service definition, as here, and in 
the program's own configuration file.

Question about s6 on Stack Exchange

[Jonathan de Boyne Pollard]

Anyone who enjoys explaining s6-rc might enjoy answering 
https://unix.stackexchange.com/q/613225/5132 .


Answers go in the box under "Your Answer", of course.  (-:

Re: Following log directories

[Jonathan de Boyne Pollard]

John W Higgins wrote:

I meant nothing towards s6 - but daemontools does not deal with leap 
seconds (or at least it cetainly looks that way from my foolish 
viewpoint).


daemontools very much *does* deal in leap seconds.  The little-known 
requirement for daemontools, which *is* in its manual and has been all 
along albeit that many people have overlooked this, is that one use one 
of the Olson "right" timezones.


* http://jdebp.uk./Softwares/nosh/guide/timestamps.html

* http://cr.yp.to/daemontools/tai64nlocal.html

Re: Following log directories

[Jonathan de Boyne Pollard]

John W Higgins:

One thing to remember about tools like this is that s6 is very precise 
with it's implemenation of TAI64N - it uses leap seconds.


For the case of follow-log-directories, note that it does not decode the 
timestamps into times.  As long as they follow the simple (and obvious) 
constraints in its manual, the tool is happy.

Re: Readiness notification exemplars

[Jonathan de Boyne Pollard]

Brett Neumeier:
It's maybe a little heavy-handed, but is there any technical reason 
not to just pick an arbitrary FD, like say 3, and just always use that 
for a particular daemon?


Further to what M. Bercot said, see 
https://unix.stackexchange.com/a/331104/5132 .

helpcovid and s6

[Jonathan de Boyne Pollard]

Help is welcome at https://github.com/bstarynk/helpcovid/issues/34 .

Re: Readiness notification exemplars

[Jonathan de Boyne Pollard]

Serge E. Hallyn:


If making changes to daemons were going to palatable, [...]



Clearly, it *is* palatable, given that a few people have been adding the 
systemd mechanism to their programs for several years, now.   
Pierre-Yves Ritschard's code and Cameron Norman's code come straight out 
of actual service programs.



Serge E. Hallyn:

you are arguing for parsing stdout for a text message and/or using 
pidfiles (written to an fd)




No, M. Bercot is not.  Quite the opposite, in fact.  The s6 readiness 
mechanism demands *neither* pidfiles *nor* standard output.  It's *any* 
file descriptor (as defined by the service), and *just a linefeed* 
followed by closing the descriptor.


I'm nodding at almost everything M. Bercot is saying here, by the way.

* http://jdebp.uk./FGA/unix-daemon-readiness-protocol-problems.html

Re: keeping sites off

[Jonathan de Boyne Pollard]

Laurent Bercot:


 What I do is:

 *

run a tinydns on another IP address (if you only have 1 nic, you
can still attribute several IPs to it)

 *

fill that tinydns with sink data for the things I want to block

 *

configure my dnscache to query my internal DNS server for the
zones I want to block. In your case, if you tell your dnscache
that your internal DNS server is authoritative for the
facebook.com zone, any query for graph.facebook.com will go to
your internal server.

 *

no /etc/hosts manipulation needed.



This is the third-best way to achieve this.  The second-best way is a 
variant.  In the third-best way there's still a lot of setup and 
on-going maintenance involved in the servers/ directory of the dnscache 
service, as one adds/removes domain names.  In the second-best way, 
tinydns also serves up the root of the DNS namespace, delegating to the 
next level of servers in the same way that the public root content DNS 
servers do, and dnscache is simply configured to start with that root 
content server.  There's no further on-going dnscache work in this 
approach, just the maintenance of the entries in the tinydns database.


A side-benefit is that this also makes your traffic to the root content 
DNS server entirely private and not susceptible to network outages.  
Whilst it applies less to real existing domain names, this in particular 
helps with some of the other stuff that tends to leak out in DNS lookups 
(caused by a whole bunch of things from the DNS client library search 
path, through non-existent top-level domains, to applications passing 
human-form IP addresses to DNS lookup), which would otherwise end up as 
negative responses from the public root content DNS servers.  Moreover, 
properly dnscache should always be providing split-horizon DNS service 
for address-to-name lookups of all of your RFC 1918 IP addresses, and 
name-to-address lookups of any internal subdomains for your LAN(s), 
which are both more private stuff that the outside world should never 
see.  This also can achieve that, as a further side-benefit, simply with 
further additions to the tinydns database, and without need to configure 
split-horizon in dnscache.


In the service bundles that accompany the nosh toolset, I supply a 
tinydns@127.53.0.1 service bundle that does all of these.  Its Makefile 
populates its database with RFC 1918 IP address stuff, the root data 
pulled from ICANN, and administrator-controlled extra stuff (where the 
re-pointed wildcard domain name data would go).


* http://jdebp.uk./Softwares/nosh/guide/services/djbdns.html

The best way, however, is to realize that using the DNS for this breaks 
every non-HTTP(S) protocol, and do this for HTTP(S) only. I've myself 
done this in two ways in the past: with a custom proxy.pac/wpad.dat file 
that the WWW browsers load, redirecting all of the relevant domains in 
ECMAScript in the WWW browser itself; and with a fully-fledged proxy 
HTTP server that I wrote, that used a database of URL patterns (not just 
domain names) and how they should be handled in HTTP, including (as one 
of several possibilities) rewriting them into temporary redirects to a 
small static content HTTP server that served up coloured rectangles as 
placeholder images amongst other things.  These are the basic approaches 
of most non-toy WWW advert-blockers, you will find.

Re: runit SIGPWR support

[Jonathan de Boyne Pollard]

Alex Suykov:


Just for reference, I checked apcupsd sources.



That was not nearly enough research, and your conclusion is ill-founded.

The SIGPWR signal, from before Linux even existed, has always meant 
"something has happened with the power".  In AT&T Unix System 5 books it 
is conventionally described as "power fail/restart".  The "something" 
can include *that the power has been restored*.  And indeed, this has 
been its long-time use, on AT&T Unix and on Linux.   Miquel van 
Smoorenbug's original powerd daemon (which later became Tom Webster's 
genpowerd), for just one example, sent a SIGPWR to process 1 and 
beforehand wrote *what that meant* into /etc/powerstatus, which could be 
"LOW", "FAIL", or "OK".  A. B. Prasad's powerh, a trap handler for 
snmptrapd, did the same.  And there are several others over the years.


* https://linuxgazette.net/issue83/prasad.html

(Indeed, on several operating systems, including HP-UX and SunOS, SIGPWR 
sent to a process other than process #1 *only* meant "power has been 
restored", and did not have the power fail meaning at all.  Programs 
were expected to reinitialize after power comes back in their SIGPWR 
handlers.  A full-screen TUI program would re-draw its UI, for example, 
on the assumption that the local terminals had lost power as well and 
were currently showing blank screens.)


It is quite wrongheaded to think that this can be completely changed 
into a "shut down and power off" command, for that reason and others 
besides.  Other reasons include that existing systems *already define 
different signals to mean that*.  There is an existing mechanism 
already.  Several, in fact.  They need more than one signal, too.  
systemd (and the nosh toolset's system-manager which has compatible 
signalling here) uses two of the real-time signals to mean "shut down 
and power off", one to trigger the service changes, and the other to 
finalize the procedure.  There's no convincing case for either the 
systemd authors or me to change, given that SIGPWR has already meant 
something else for a long time and changing it would conflict with 
existing programs that we want to interoperate with.  There's a strongly 
convincing case to *avoid* changing SIGPWR, in fact.


The simple truth is that if you are using SIGPWR to mean only a "power 
failed" event, you aren't using it correctly.  van Smoorenburg init went 
and looked into /etc/powerstatus and acted accordingly, and still does 
so *to this day*, albeit that the file is now located under /var/run.  
(The systemd people and I both provide infrastructure only.  We define a 
target that is activated, but not what services that target invokes; and 
we leave it up to the third-party services to determine what the power 
change event actually is.)  Send a SIGPWR to process 1 without writing 
/run/powerstatus means that you'll get the last power change event set 
by the person that *did* use the signal properly.


Moreover, not only is there an existing mechanism in such programs, 
there are *several* existing mechanisms.  And they don't even all use 
signals.  The program being run could be anything, not just 
system-manager, or runit-init, or systemd, or finit, or even one of the 
several specialized programs designed solely to be process #1 of a 
container.  *Even just those* do not all use signals.


* https://unix.stackexchange.com/a/191875/5132

In the case of van Smoorenburg init and Joachim Nilsson's finit, there 
is a private API for commanding system state changes, idiosyncratic to 
just those softwares (and not compatible even across the twain), 
involving sending messages along a FIFO that process #1 is reading the 
other end of.  Ironically, Karl M. Hegbloom augmented genpowerd back in 
1998 to optionally use this API, which is more expansive than the 
comparatively small signal API that van Smoorenburg init has.  To 
properly use van Smoorenburg init or Nilsson finit in a container, you 
have to configure the container manager to *do something else instead of 
sending signals at all* to send it system management commands such as 
"shut down and halt/restart/poweroff/powercycle", as the API for 
commanding these is not signals.  The right place to handle all of these 
variances is in the container manager's configuration settings.


* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=23007#25

If you want to have some sort of "shut down and 
halt/restart/poweroff/powercycle" API between a container manager and 
process #1 inside a container, slightly absurd that the idea is for 
something that isn't a fully-fledged virtual machine with a virtual 
power supply, it is the program that is run that dictates the protocol 
to you, *not* you that dictates to the program being run.  The correct 
course of action is to tailor the container manager to the particular 
program, not try to make all programs, decades after the fact, suddenly 
discard and incompatibly overwrite the long-established, event not 
command, m

Re: runit SIGPWR support

[Jonathan de Boyne Pollard]

Laurent Bercot:
 Of course, but once the fd is closed, /dev/console should not have 
any impact on the process, so would a kbrequest still reach it?


Yes.  First: This is a kernel virtual terminal thing not a console 
thing.  Strictly speaking, it is doing it wrongly to access it through 
the console device, which is not necessarily a KVT.  Second: The 
mechanism does not require an open file descriptor.  It requires that 
the target process never terminate, because there's no symmetrical 
kernel API function to disable the mechanism once it has been enabled, 
but it does not require that that process have any particular file 
descriptors open.

Re: runit SIGPWR support

[Jonathan de Boyne Pollard]

Laurent Bercot:
What piece of software sends SIGRTMIN+3 to pid 1 when you're not 
running systemd? 


* 
http://jdebp.uk./Softwares/nosh/guide/commands/system-control.xml#SYSTEMCONTROL


(-:

Re: Handle ENOTDIR in pathexec_run

[Jonathan de Boyne Pollard]

Jacob Vosmaer:


The patch below makes runit ignore this type of bad path entry.
What do you think?



It's not good enough.

I was all set to apply this to djbwares, as the runit code for pathexec 
is actually Daniel J. Bernstein's original code.  But it highlighted 
some basic problems of running execve() in a loop to do a PATH search.


Simply put: One wants to (largely) ignore the error if there's a problem 
with the directory prefix from the search path; and one wants to retain 
the error if there's a problem with the final pathname component, the 
name that is being searched for or the actual node that it names.  
Unfortunately, execve() can return EACCESS in both cases, and simply 
deciding to exit the loop based upon nothing but the errno value is 
therefore an unworkable strategy.


If you look at the BSD C library in FreeBSD, you'll see that it attempts 
to distinguish between the two cases in its execvpe() by calling stat() 
on error.  If the stat() succeeds, then the problem cannot be with the 
directory prefix.  So it remembers the EACCESS, which must be a problem 
with the file itself.  If the stat() fails, then the problem is with the 
directory prefix (or with symbolic link traversal, hence the use of 
stat() not lstat()) and it largely ignores the EACCESS.


If you look at the internal_execve() function in the next release of the 
nosh toolset, you'll see that I've used a different design and made use 
of (relatively, they still being over a decade old) newer API functions, 
instead.  The code uses openat() on the path prefixes with O_DIRECTORY 
(and O_CLOEXEC), if that fails largely ignoring that error and 
continuing to the next PATH component.  It then uses, relative to that 
file descriptor, openat() with O_EXEC and then fexecve() for the final 
component (which it also this way does not have to do string 
concatenation with), retaining that error if that fails.  Things are 
slightly complicated by some trickiness relating to executable scripts 
with interpreters, but that's the gist of it.


pathexec_run() needs similar attention.

Re: Debian and init freedom (logging freedom?)

[Jonathan de Boyne Pollard]

fungal-net:


While upgrading my antiX-sid (Debian) with s6 and 66 installed today

… the system enforced Debian policy on you, which you had violated by 
using UIDs less than 100 for your private "log" and "s6log" accounts.  
Debian Policy §9.2.2 reserves UIDs less than 100 to Debian, for global 
assignment.


* 
https://www.debian.org/doc/debian-policy/ch-opersys.html#uid-and-gid-classes


Observe in contrast that the nosh toolset employs UIDs in the ordinary 
dynamic range for log accounts.

Re: s6 usability

[Jonathan de Boyne Pollard]

Oliver Schad:


A booting tools should be in /bin - full stop!



That is historically untrue.  The real world has not actually worked in 
the way that some people think.


* https://unix.stackexchange.com/a/448799/5132

Re: s6 usability

[Jonathan de Boyne Pollard]

Samuel Holland:


Then, with the symlinks, s6 could "provide" virtual/daemontools.

I do this with the nosh toolset already, for Debian.  Several of the 
"shim" and "-run" packages, which are separate from the main tools 
packages, "provide" stuff.  I even have  a things such as a dummy 
"nosh-logrotate-shims" package that provides "logrotate", which is 
required by things like the "rabbitmq-server" package but isn't actually 
needed at all in order to run rabbitmq-server under nosh service 
management.  And a "nosh-run-bcron" that "provides" "bcron-run".


* http://jdebp.uk./Softwares/nosh/debian-binary-packages.html#Toolsets

* http://jdebp.uk./Softwares/nosh/debian-binary-packages.html#run

Re: runit patches to fix compiler warnings on RHEL 7

[Jonathan de Boyne Pollard]

Jan Braun:


2) runit has manpages. s6 has HTML. :(

Daniel J. Bernstein had something to say on that subject, two decades 
ago.  See the "Notes" section of http://cr.yp.to/slashdoc.html .


I generate both manual pages and HTML from a common DocBook XML master 
in the nosh toolset.  And the DocBook XML is itself readable directly 
with a WWW browser. 
http://jdebp.uk./Softwares/nosh/guide/commands/setuidgid-fromenv.xml is 
a copy of one such DocBook XML master, for example.  It's on the WWW, 
and the packages also install it locally, for off-line reading.


M. Pape did some of the manual pages for some operating system's 
versions of daemontools, converting M. Bernstein's HTML pages into 
roff.  For djbwares I converted everything into DocBook XML, and the 
same holds for djbwares as for the nosh toolset.  There is a DocBook XML 
master that one can view in a WWW browser directly (both on-line and 
off-line), generated HTML pages, and generated manual pages readable 
with man. 
http://jdebp.uk./Softwares/djbwares/guide/commands/setuidgid.xml is a 
copy of one such DocBook XML master, for example.  This is the source 
for the "man setuidgid" manual and the source for 
http://jdebp.uk./Softwares/djbwares/guide/setuidgid.html .


I even filled in the manual pages that M. Pape hadn't done and that M. 
Bernstein hadn't originally written in HTML, and updated some of the 
doco.  See 
http://jdebp.uk./Softwares/djbwares/guide/commands/caldate_easter.xml 
and http://jdebp.uk./Softwares/djbwares/guide/commands/dnscache.xml for 
examples of that.

Re: runit patches to fix compiler warnings on RHEL 7

[Jonathan de Boyne Pollard]

Some of those are common.  I looked at similar stuff in daemontools, 
where runit got some of this code from, when I packaged it up with some 
of the other Bernstein softwares some years ago.  However, you have 
missed the point of HASSHORTSETGROUPS.  There's no point in having 
conditionally compiled code selected by it that uses gid_t in both 
codepaths.  If you are going to use gid_t, you do not need the 
HASSHORTSETGROUPS mechanism in its entirety.  Think about what it does.


* http://jdebp.uk./Softwares/djbwares/

In the nosh toolset, I have no such mechanisms.  The code uses the 
 types, with a std::vector for the call to setgroups() 
in setuidgid-fromenv for example.  The only similar mechanism picks 
between the very old waitpid() function and the newer (but still old, 
because it was SVR4) waitid() function.  And only OpenBSD ever needs the 
code to use the former, in practice.

Re: runit patches to fix compiler warnings on RHEL 7

[Jonathan de Boyne Pollard]

Laurent Bercot:

It looks like several distributions have their own version of runit; 
they are maintained by the distros themselves.


Further to all that:  I believe, although things may have changed, that 
the Debian maintainer for runit is open to patches.


* https://tracker.debian.org/pkg/runit

Re: s6-log can create current with 640?

[Jonathan de Boyne Pollard]

Laurent Bercot:
The real question is, why is there a "umask" binary that's not the one 
from execline? Non-chainloading non-builtin umask is nonsense, just 
like non-chainloading non-builtin cd. 


Not quite.  People find uses for these things, and as the SUS rationale 
points out, for every potentially useless external equivalent of a 
(non-special) built-in command someone has come up with an arcane actual 
use for it.  Even "cd".


* 
https://pubs.opengroup.org/onlinepubs/9699919799/xrat/V4_xcu_chap01.html#tag_23_01_07


* https://unix.stackexchange.com/q/50058/5132

The POSIX model is therefore that all non-special built-ins are also 
available as executables; or, rather, that all of the standard utilities 
that are not special built-ins are simply *available* (via execvp(), 
find -exec, env, and *all of the other* ways that standard utilities can 
be invoked), and whether they are built-in or not, to a shell or 
otherwise, is an implementation detail as far as actually invoking the 
utility is concerned.  (Very few shells truly conform to this, but the 
Watanabe shell largely does in its non-native mode.)


* https://unix.stackexchange.com/q/347188/5132

* https://unix.stackexchange.com/q/496259/5132

nosh, not being a shell, is not bound by this.  Its built-ins are found 
before a PATH search is enacted.  So it actually does what M. Geraghty 
mistakenly thought execline was doing.  It finds its own built-ins in 
preference to finding external commands; so invoking "umask" (without 
directory name prefix) within a nosh script will always invoke the 
built-in chain-loading one, irrespective of PATH. (Several of the 
non-conformant shells do this, too, giving their built-ins unequivocal 
precedence over a PATH search.  Some even document this behaviour.  But 
it is not standard-conformant for a shell.)


* https://unix.stackexchange.com/a/496377/5132

* http://jdebp.uk./Softwares/nosh/guide/commands/exec.xml

execline always searches PATH.  It does not have built-ins like nosh or 
shells, in the first place.  There's a quirk about when changes to PATH 
take effect, and it does not quite have the POSIX semantics for when 
PATH is not set, but other than that it actually has the POSIX model 
behaviour: Commands are located using PATH, and what command one invokes 
by the name "umask" is entirely determined by what PATH lists.


* https://unix.stackexchange.com/a/448799/5132

Re: s6-log can create current with 640?

[Jonathan de Boyne Pollard]

Dewayne Geraghty:

I did try umask 037 but that just broke the pipe.


That was probably because you were using the wrong umask tool.  It has 
to be the chain-loading one in a nosh or an execlineb script.


% cat /var/sv/deluged/service/run
#!/bin/nosh
#Deluge BitTorrent client
envuidgid -- deluge
userenv-fromenv
umask 037
setuidgid -- deluge
deluged --do-not-daemonize
%

* http://jdebp.uk./Softwares/nosh/guide/commands/umask.xml

* http://skarnet.org/software/execline/umask.html

Re: chpst -u and supplementary groups

[Jonathan de Boyne Pollard]

My inability to see the issue came from the fact that all other 
similar programs (I'm aware of) do in fact add the supplementary groups.


Then you are not aware of Bernstein daemontools, where setuidgid does 
not.  (-:


# /package/admin/djbwares/command/setuidgid operator id
uid=2(operator) gid=5(operator) groups=5(operator)
#

* http://jdebp.uk./Softwares/djbwares/guide/commands/setuidgid.xml

Setting only one group was the behaviour of the original tool. Setting 
the supplementary groups as well is behaviour that others added to their 
toolsets later.  Bruce Guenter (in daemontools-encore) and I added it as 
an optional behaviour for setuidgid.


# /package/admin/nosh/command/setuidgid operator id
uid=2(operator) gid=5(operator) groups=5(operator)
# /package/admin/nosh/command/setuidgid --supplementary operator id
uid=2(operator) gid=5(operator) groups=5(operator),1298(log)
#

* http://jdebp.uk./Softwares/nosh/guide/commands/setuidgid.xml

* http://untroubled.org/daemontools-encore/setuidgid.8.html

Re: interesting claims

[Jonathan de Boyne Pollard]

Jeff:


where can i learn how a "correct" init has to operate ?



See https://unix.stackexchange.com/a/197472/5132 for starters.

nosh version 1.40

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.40 .

 *

   http://jdebp.eu./Softwares/nosh/

 *

   http://jdebp.info./Softwares/nosh/

 *

   https://github.com/freebsd/freebsd-quarterly/blob/master/2018q4/nosh.md

This version sees changes to the doco, improvements to network 
configuration, and a change to machine ID generation.



FreeBSD binaries



I plan for this to be the last release with binaries built on FreeBSD 
10.  I am going to upgrade the build machine.



|ifconfig|



 *

   https://unix.stackexchange.com/a/504084/5132

There is now an |ifconfig| command in the toolset, with a command-line 
interface and output similar to the FreeBSD |ifconfig|. It is primarily 
intended for use on non-FreeBSD systems, to provide a FreeBSD-like 
|ifconfig| where one does not have the actual FreeBSD tool.  The 
|ifconfig@*| services generated by the external configuration import 
subsystem make use of it on Linux operating systems, allowing the 
services themselves to be pretty much the same across platforms.


It handles multiple (unlabelled) addresses per interface and both IP 
version 6 and IP version 4 addressing, which are two of the 
long-standing complaints against the old |ifconfig| programs from GNU 
inetutils  and NET-3 net-tools 
.  It has no notion of 
overwriting a single "primary" address.  It has a |broadcast1| flag for 
calculating the broadcast address from the prefix length and address.  
It prefers the new (since 1993) notation for IP version 4 network 
masks.  It can do the FreeBSD style of EUI-64 address assignment for IP 
version 6 with an |eui64| flag.


And it colours its output if writing to a terminal.  (-:


Machine ID generation

=

FreeBSD from 2007 onwards used the SMBIOS system UUID from the machine 
firmware as a fallback source for a machine ID. |setup-machine-id| prior 
to this release of the toolset would do the same for compatibility.  
This has now been removed from |setup-machine-id|. The privacy problems 
that it entails have turned out to outweigh what little utility it had.


Systems that would have fallen back upon the SMBIOS system UUID will now 
fall back to creating UUIDs using the C library.  Note that the FreeBSD 
C library still uses MAC addresses to create UUIDs.  The OpenBSD and GNU 
C libraries use CSPRNGs.


There is also now an |erase-machine-id| command that resets all of the 
machine ID storage locations set by |setup-machine-id| to a nil UUID.  
The |machine-id| service now calls |erase-machine-id| at shutdown.


Thus: Machine IDs (when using the supplied service bundles) now have a 
lifetime from bootstrap to shutdown, will not persist across reboots, do 
not reveal the SMBIOS system UUID and are not constant and correlatable 
because of it even when explicitly wiped, and can still reveal MAC 
addresses on FreeBSD.


The new |machine-id|(7) manual page lists some of the known users of 
machine IDs, explains where machine IDs are stored, and gives some of 
the history of machine IDs.



Square mode

===

Square mode is now switchable in |console-terminal-emulator|, using DEC 
Private Mode 1369. |console-control-sequence| has a |--square| option 
for changing it.



Other tools

===

|ucspi-socket-rules-check| has gained the ability to check |uid/self/| 
and |gid/self/| subdirectories when handling UCSPI-UNIX connections.



Doco



The Guide now includes the original command manuals, written in DocBook 
XML.  These are directly readable using a GUI WWW browser and the 
supplied stylesheet.  The conversions to HTML are still supplied, but 
reading the original DocBook XML format is better.


TUI WWW browsers such as lynx cannot read DocBook XML.  Their deficiency 
has inspired a new |console-docbook-xml-viewer| tool that parses and 
displays the manual pages with a simple full-screen interface on a 
terminal.  This can of course display other DocBook XML manual pages as 
well.



External configuration import improvements

==

The external configuration import subsystem now allows various 
extensions in a |/etc/network/interfaces| file, including |ipv4ll| 
stanzas (which will cause |avahi-autoipd| services to be set up) and 
|eui64| stanzas.  A "|broadcast +|" setting is now transformed into the 
aforementioned |broadcast1| flag for |ifconfig|.


It also now once more treats |false| for the login shell as signifying a 
non-personal user account.



More service bundles



There are a few more service bundles in this release, including ones for 
Laurent Bercot's mdev, for two-ftp, and for NcFTPd.

djbwares version 9

[Jonathan de Boyne Pollard]

The djbwares package is now up to version 9 .

 *

   http://jdebp.eu./Softwares/djbwares/

 *

   http://jdebp.info./Softwares/djbwares/

This version sees changes to the doco and to the DNS and HTTP servers.


FreeBSD binaries


I plan for this to be the last release with binaries built on FreeBSD 
10.  I am going to upgrade the build machine.



Doco


All of the manuals are now DocBook XML, and the hodgepodge admixture of 
manuals from three sources is gone.  A stylesheet is supplied for 
reading the manuals directly using a GUI WWW browser.  They can also be 
read using the |console-docbook-xml-viewer| tool from version 1.40 of 
the nosh toolset.


The long-missing |tcp-environ| manual is now present.


HTTP servers


Hand in hand with the documentation improvement, |httpd| now has content 
types for the |.xml| and |.xhtml| filename extensions.



DNS servers
===

 *

   http://jdebp.eu./Softwares/djbwares/qmail-patches.html#any-to-cname

 *

   http://jdebp.info./Softwares/djbwares/qmail-patches.html#any-to-cname

The |ANY| query type in the DNS has never meant ALL, and has never 
really been useful.  Pretty much only one software even made use of 
|ANY| for non-testing purposes.  That was qmail, which I patched not to 
do so back in 2003.


Although the diagnostic tools still support sending |ANY| queries, with 
one exception, in this release all of the DNS server softwares now 
synthesize non-responses, containing an invariant |HINFO| resource 
record set, to |ANY| queries.  The specialized content DNS servers 
simply return such responses straightforwardly, as they do not have to 
worry about |CNAME| chains, which they do not ever construct.  The other 
DNS servers have to handle |CNAME| chains.


The general-purpose |tinydns| and |axfrdns| content DNS servers will 
continue to process |CNAME| chains as before, but will return a 
synthesized |HINFO| resource record set at the end of the chain. 
|dnscache| also processes |CNAME| chains as before, again returning the 
chain with a synthesized |HINFO| resource record set at the end.  It no 
longer allows |ANY| queries as a loophole for retrieving cache contents, 
and will not issue |ANY| queries from its back end.


The diagnostic tool that is the exception is |tinydns-get|, whose 
operation is supposed to exactly replicate what |tinydns| does.  It, 
too, now synthesizes non-responses when an |ANY| lookup is requested.

Tesla autopilot, runit, and svlogd

[Jonathan de Boyne Pollard]

So …

* https://news.ycombinator.com/item?id=19193572

… where are and who did the modifications to runit to add -L and -M 
options to svlogd?

Re: No ifconfig

[Jonathan de Boyne Pollard]

 *

   https://lists.debian.org/debian-user/2017/08/msg01613.html

 *

   https://news.ycombinator.com/item?id=17152738

 *

   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=274269

 * https://news.ycombinator.com/item?id=17151922

Since I needed an |ifconfig| with a more BSD-like interface /anyway/ so 
that I didn't have to maintain highly divergent scripts, you are going 
to gain in the future.


|ifconfig| from GNU inetutils 
:


   jdebp % inetutils-ifconfig -l
   enp14s0 enp15s0 lo
   jdebp % inetutils-ifconfig lo
   loLink encap:Local Loopback
  inet addr:127.0.0.1  Bcast:0.0.0.0  Mask:255.0.0.0
  UP LOOPBACK RUNNING  MTU:65536  Metric:1
  RX packets:9087 errors:0 dropped:0 overruns:0 frame:0
  TX packets:9087 errors:0 dropped:0 overruns:0 carrier:0
  collisions:0 txqueuelen:1000
  RX bytes:51214341  TX bytes:51214341

   jdebp %

|ifconfig| from NET-3 net-tools 
:


   jdebp % ifconfig -l
   ifconfig: option `-l' not recognised.
   ifconfig: `--help' gives usage information.
   jdebp % ifconfig lo
   lo: flags=73  mtu 65536
inet 127.0.0.1  netmask 255.0.0.0
inet6 ::1  prefixlen 128  scopeid 0x10
inet6 ::2  prefixlen 128  scopeid 0x80
loop  txqueuelen 1000  (Local Loopback)
RX packets 9087  bytes 51214341 (48.8 MiB)
RX errors 0  dropped 0  overruns 0  frame 0
TX packets 9087  bytes 51214341 (48.8 MiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

   jdebp %

|ifconfig| from an unreleased version of the nosh toolset 
:


   jdebp % ifconfig -l
   enp14s0 enp15s0 lo
   jdebp % ifconfig lo
   lo
link up loopback running
link address 00:00:00:00:00:00 bdaddr 00:00:00:00:00:00
inet4 address 127.0.0.1 prefixlen 8 bdaddr 127.0.0.1
inet4 address 127.53.0.1 prefixlen 32 bdaddr 127.53.0.1
inet6 address ::2 scope 0 prefixlen 128
inet6 address ::1 scope 0 prefixlen 128
   jdebp %

FreeBSD |ifconfig| (on a different system) for comparison:

   JdeBP % ifconfig lo0
   lo0: 
flags=ffde8149
 metric 0 mtu 16384
options=63
inet6 ::2 prefixlen 128
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
inet 127.0.0.1 netmask 0xff00
inet 127.53.0.1 netmask 0xff00
nd6 options=61
   JdeBP %

The |ifconfig| from an unreleased version of the nosh toolset 
 on that other system:


   JdeBP % ifconfig lo0
   lo0
link up loopback drv_running running promisc multicast ppromisc monitor 
staticarp renaming
nd6 performnud auto_linklocal noradr
 rxcsum txcsum hwcsum rxcsum_ipv6 txcsum_ipv6
link address lo0 metric 0 mtu 16384
type 24 linkstate 0 physical 0 baudrate 0
inet6 address ::2 scope 0 prefixlen 128
inet6 address ::1 scope 0 prefixlen 128 bdaddr ::1 scope 0
inet6 address fe80::1 scope 3 prefixlen 64
inet4 address 127.0.0.1 prefixlen 8 bdaddr 127.0.0.1
inet4 address 127.53.0.1 prefixlen 8 bdaddr 127.53.0.1
   JdeBP %

svscan and supervise

[Jonathan de Boyne Pollard]

Kelly Dean:


Surely this is a common question.


It's a common redesign.

In the original daemontools, |supervise| knows nothing at all about the 
difference between "log" and "main" services.  Linking up the standard 
I/Os with pipes, to connect "main" services to "log" services, and 
knowing the directory layout that delineates them, is entirely the 
domain of |svscan|, which in its turn knows nothing at all about the 
control/status API and about supervision.  M. Bercot has stuck with this 
design.


As laid out in this Frequently Given Answer 
, three of the other 
toolsets did not, and went down the path of tighter integration of the 
twain. They baked in a relationship between "main" and "log" services. 
In the original daemontools design, it was theoretically possible to run 
|supervise| under something else, that set up different relationships.  
Indeed, in the original daemontools |svscan| came along (in version 
0.60) about two years after |supervise| did, /originally/ in early 
versions people were expected to run |supervise| directly, and it was 
more like |daemon| in its operation.  This was until it became apparent 
that "dæmonization" is a fallacy, and simply does not work on modern 
operating systems where login sessions have gone through too many 
one-way trapdoors for escaping to a dæmon context to be viable.  Running 
dæmons outwith login session context /right from the top/ became the 
way, with |svscan| and (the also since much-replaced 
) 
|svscanboot|.  This was of course how service management had been done 
on AIX since 1990 and on AT&T Unix since 1988 
.


My nosh toolset did not stick with the original design in this regard, 
either.  But I did not go for tighter integration, which I thought to be 
the wrong thing to do. |service-manager| 
 
manages a whole bunch of services, does all of the waiting and spawning 
in a single process, which is conventionally also a subreaper, and 
provides all of the control/status APIs.  But the knowledge of the 
relationships amongst those services, and of directory structures, is 
entirely /outwith/ that program.


It is, rather, in programs like |system-control| 
 (in 
its |start| and |stop| subcommands) and |service-dt-scanner| 
. 
They decide the policy, which services' standard I/Os are plumbed 
together and how a "main" service indicates its "log" service. They 
instruct |service-manager| with the |service/| and |supervise/| 
directories, passing it open file descriptors for them, and what to 
plumb to what, and it just runs the mechanism.  They even implement /two 
different/ policies, one with ordering and dependency processing and a 
full /service bundle/ mechanism and the other more like the old 
daemontools and s6 /scan directory/.  There is no reason that a third 
tool could not implement a third policy still.


There is not even a requirement of a 1:1 relationship between "main" and 
"log" services, and indeed the set of pre-supplied service bundles has 
fan-in arrangements for a few of the short-lived single-shot services 
that run at bootstrap/shutdown, with them sharing a single logger. The 
pre-supplied per-user services have three fan-in sets 
, 
too.

Re: Generic interrupt command?

[Jonathan de Boyne Pollard]

Colin Booth:


As documented here: https://www.skarnet.org/software/s6/s6-svstat.html

s6-svstat -p /path/to/service | xargs kill SIGNAL

You can thank Jos Backus for similar functionality in the nosh toolset 
since 2013, with program-readable output that one can use existing tools 
to pull arbitrary fields from.


   % svshow --json /var/sv/* 2>/dev/null | jq '."/var/sv/bcron-sched".MainPID'
   1861
   % svshow --json /var/sv/* 2>/dev/null | jq 
'."/var/sv/bcron-sched".RunTimestamp'
   4611686019976326000
   % svshow --json /var/sv/* 2>/dev/null | jq 
'."/var/sv/bcron-sched".DaemontoolsEncoreState'
   "running"
   % svshow --json /var/sv/* 2>/dev/null | jq 
'."/var/sv/bcron-sched"."Wanted-By"'
   [
  "/etc/service-bundles/targets/server"
   ]
   %

Re: How best to ensure s6-managed services are shut down cleanly?

[Jonathan de Boyne Pollard]

Laurent Bercot:

The question is, how does systemd decide to proceed with the rest of 
the shutdown?


It waits for |s6-svscan| for up to 90s, putting the infamous cylon 
warrior and "A stop job is running for s6" message on the console.  
After 90s, it starts forcibly killing stuff, not necessarily in the 
right order because it does not know that PostgreSQL should be killed 
before |s6-svscan| and that main services are best taken down before log 
services.


No, it will not wait forever for |s6-svscan| to exit.  That is not a way 
to block it.


I arrange things differently for running |service-manager| under systemd 
:


   % grep ExecStop /usr/local/lib/systemd/system/system-control-normal.service
   ExecStop=/bin/system-control start --verbose shutdown
   %

|system-control| 
 has 
all of the logic that knows to try harder if a |TERM| signal does not 
stop a service within 60s, and the |start| of |shutdown| stops running 
normal services because the |shutdown| service has |conflicts/| 
relationships with them.  None of this logic is in the service manager 
itself, which does not need to know about timeouts and alternative 
signals, it comprising mechanism not policy.


systemd will still try sending |TERM| signals to the service manager and 
force-killing stuff out of order, but because of an |After=| ordering 
only /after/ the |ExecStop| of |system-control-normal.service| has had 
its chance to shut things down in an orderly fashion.  systemd does not 
even begin taking down the service manager until after |system-control| 
has attempted to shut down all managed services.

Re: s6 problems logging

[Jonathan de Boyne Pollard]

Peter Pentchev:

If it is not easy to change the program itself, the stdbuf(1) utility 
[...]


I am certain that M. MacLennan is capable of changing M. MacLennan's own 
program.  After all, you are replying to a message where xe said that xe 
already did.  (-:


Injecting things into the code of the running program via pre-loads 
should be a last resort, not on an equal footing with adjusting the 
program's logging.  In any case, the Bernstein |ptybandage| 
 achieves 
this /without/ injecting code into the target programs.

svscan --help

[Jonathan de Boyne Pollard]

Laurent Bercot:

s6-svscan is pretty mild in that aspect.


In fairness, the original Bernstein program was somewhat more forgiving:

> /package/admin/djbwares % command/svscan --help
> svscan: fatal: unable to chdir to --help: file does not exist
> /package/admin/djbwares %

People used this program wrongly so often with the Bernstein tools that 
I gained a Frequently Given Answer about it in 2002.


* http://jdebp.eu./FGA/daemontools-spurious-supervise-directories.html

I take the view that if the user gives a command that takes a specific 
number of arguments the wrong number of arguments, that's an error.  A 
quick if (1 != args.size()) test saves this support pain, and having to 
explain this *yet again*.


> /package/admin/nosh % command/svscan 1 2
> svscan: FATAL: One directory name is required.
> /package/admin/nosh % command/svscan
> svscan: FATAL: One directory name is required.
> /package/admin/nosh % command/svscan /dev/null
> svscan: FATAL: /dev/null: Not a directory
> /package/admin/nosh %

I defined the directory argument as mandatory in service-dt-scanner, 
a.k.a. svscan, though.


> /package/admin/nosh % command/svscan --help
> Usage: svscan [-?] [--help] [--usage] [--input-activation] {directory}
>
> Main options:
> --input-activation Use input activation for log services.
> /package/admin/nosh %

It's not exactly the same program, after all.

* http://jdebp.eu./Softwares/nosh/guide/commands/service-dt-scanner.xml

* http://jdebp.eu./Softwares/djbwares/guide/commands/svscan.xml

Users should indeed read the doco.  It is written to be read.  (-:

Re: Essex: A simple command line interface for managing s6 services, using the s6 toolset

[Jonathan de Boyne Pollard]

The |cat| subcommand is the one that seems to get most people.  I have 
seen more objections to that in systemd's |systemctl| than to any of the 
others, because it really isn't concatenation as people are used to.  I 
provide it in my toolset because I provide migration paths for people 
familiar with systemd.  I would not have chosen |cat| myself, if I had 
had the choice.  (It is actually |grep| under the covers, not |cat|, for 
starters.)  If you are /not/ looking to have similar commands as a 
systemd migration path, which seems to be the case from the other 
subcommands that you have, you would probably do well to come up with a 
better name than |cat|.


I went with |reset| rather than |sync|. |preset| sets the /enable/ state 
from configuration information. |reset| sets the /start/ state from 
configuration information, and as the manual says can be thought of as 
"reset to however the service is configured to be at bootstrap" with 
respect to start/stop. |sync| is not a bad name, but |system-control| 
 can 
do /system/-level stuff (e.g. |poweroff|) as well as /service/-level 
stuff, and there is a definite quite different meaning to |sync| when 
one is operating at system level which would not be a good idea to 
overlap.  You're providing service management and not system management, 
and contrastingly are probably alright here.


Your |reload| is not what people have come to know and to think of by 
that name, and will probably confuse people. What people have come to 
know by that name is sending a "reload" signal (of some kind, the exact 
signal varies) to the service telling the service to /keep running/ but 
reload stuff. What you have is more akin to what people know as 
|condrestart| or |try-restart|, although it is not quite those either, 
because (for one thing) they do not have the side-effect of starting 
currently stopped services as your |reload| subcommand does.  See the 
Debian /Policy Manual/ 
 
and the Fedora wiki 
 
for the long standing meanings of the |try-restart| and |reload| 
subcommands.


Placing the generation of the information that your |reload| operates on 
actually in the |run| script itself is a bit of a layering violation.  
Really, a framework to detect changes to |run| scripts should have no 
effect on the |run| scripts contents themselves, certainly not the 
requirement that they be modified to inject special extra commands.  You 
end up with user problems, with users setting up vanilla |run| scripts 
that they get from elsewhere 
 
and then not being able to successfully employ your tools on top of 
that.  You end up with mechanism problems, with a /single change/ to a 
|run| script resulting in potentially /repeated/ termination signals (if 
|reload| is called more than once, which people inevitably /will/ do) 
until the part of the |run| script that regenerates the signature 
actually re-runs, and takes.


Observe how I designed service bundles.  They are layered on top of 
|supervise/| and |service/| directories.  Before/after orderings, 
wants/conflicts dependencies, and the rest are all /outwith/ the 
scripts, and do not require modifying them.  Only |system-control| has 
dealings in them.  The scripts, the lower-level |service-control| 
/|svc|, 
and indeed the |service-manager| 
 
itself, have no notion of /any/ of that.  Indeed, it is one of /two/ 
ways of deciding what services to run, the other of which would be 
adversely affected if all of the scripts had to incorporate tooling for 
the first mechanism.  Maintaining layering allows drop-in replacement, 
and the administrator a choice of the |system-control| way or a 
daemontools-style |service-dt-scanner| 
/|svscan| 
way.


Maintaining layering allows further forms of drop-in replacement, too.  
My nosh toolset does not require that |run| scripts use the toolset's 
own chain-loading utilities.  It is just as happy with |run| scripts 
that use execline, or the perp tools, or runit's |chpst|.  The service 
manager only cares that it's a program that it can |fexecve()|. Whereas 
your tooling forces execline for everything, because you've built it 
around having a snippet of execline code embedded in an execline script 
as a part of the mechanism.


For a restart-if-the-|run|-script-has-changed framework, I suggest a 
similar approach.  The simplest approach is surely to have a per-service 
|Makefile.essex|, or a set of redo |.do| scripts, that embodies the 
if-file-X-changed-take-action-Y lo

smaclennan/doorknob

[Jonathan de Boyne Pollard]

 *

   The plural of "address" is "addresses".  "-ii" is not how any word
   pluralizes in English.

 *

   The way to monitor such a directory nowadays is with
   |kevent(EVFILT_VNODE)| or |inotify_*()|.

 *

   Long-known multi-user spool security precautions: /Always/ check the
   |d_type|; /always/ |fstatat()|/|fstat()| the spool file and check
   for |S_ISREG()|.

 *

   Long-known multi-user spool security precautions: Spool areas should
   be beneath a non-world-accessible parent directory, and the program
   that dumps into the spool should be set-group-ID to a group (or a
   /non-owner/ user) that has search access on the parent in order to
   reach the spool directory beneath.  Not doing this is an error that
   was initially made in Postfix years ago, avoiding all set-ID
   programs without realizing that set-ID is a necessarymechanism for
   secure multi-user spooling when it is in this form. (See Bruce
   Guenter's bcron <http://untroubled.org/bcron/> for an example of an
   alternative way that multi-user spooling can be structured using a
   submission server and UCSPI-UNIX.)

   The spool-processing dæmon itself does not need to run under the
   aegis of this group, if it is simply started up already in its spool
   directory using |chdir|
   <http://jdebp.eu./Softwares/nosh/guide/commands/chdir.xml> (or cd
   <http://skarnet.org./software/execline/cd.html> or equivalent) in
   the |run| script.  (This also means that it does not need to
   hardcode the location of its spool directory.  Its spool directory
   is its working directory, where it works.)

 *

   Other security precautions: Dæmons such as this should /always/
   |setuidgid|
   <http://jdebp.eu./Softwares/nosh/guide/commands/setuidgid.xml> away
   from the superuser in their |run| scripts to an account dedicated to
   the dæmon.  (Compare the |run| script in Bruce Guenter's nullmailer
   <http://untroubled.org/nullmailer/>.)

 o

   Jonathan de Boyne Pollard (2019). "Limiting services: Running
   under the aegises of unprivileged user accounts
   <http://jdebp.eu./Softwares/nosh/guide/limiting-services.html#users>".
   /nosh Guide/. Softwares.

 o

   https://unix.stackexchange.com/questions/450251/

 *

   Other spool-processing dæmon security measures: The dedicated user
   account has no need to own /anything/, neither the spooled files nor
   the spool directory itself.  It needs only read+write+execute
   permission on the spool directory, and read permission on the spool
   files.  Having ownership permission as well permits compromised
   dæmons to change ACLs and permissions.

 o

   Jonathan de Boyne Pollard (2019). "Log service security:
   Dedicated log user accounts
   
<http://jdebp.eu./Softwares/nosh/guide/log-service-security.html#users>"./nosh
   Guide/. Softwares.

 *

   Debian family operating system conventions:  That has not been the
   way to write a van Smoorenburg |rc| script for Debian family
   operating systems since 2014.

 o

   Petter Reinholdtsen (2014-02-09). init-d-script
   
<https://manpages.debian.org/stretch/sysvinit-utils/init-d-script.5.en.html>.
   /File formats manual/. Debian.

 *

   C language standards: |stdout| is fully buffered if it is not an
   interactive device, which a pipe is indeed not. |stderr| is of
   course /not/ fully buffered.

 *

   Unix conventions: Logs go to standard error.

nosh version 1.39

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.39 .

* http://jdebp.eu./Softwares/nosh/

* 
https://www.freebsd.org/news/status/report-2017-07-2017-09.html#The-nosh-Project 



* http://jdebp.info./Softwares/nosh/

I missed announcing 1.38, so this announcement will cover both versions.

These versions see a major addition to the user-space virtual terminal 
subsystem, various other changes in several areas, the completion of 
some items mentioned as placeholders in version 1.37, and some bug fixes.



Completed placeholders
==

make-read-only-fs is now fully implemented, and is no longer a placeholder.


More service bundles


There are several more additions to the set of service bundles supplied 
with the toolset: connman, ofono, dundee, cntlm, minidlna, powertop, 
alsa-state, alsa-restore, unattended-upgrade-shutdown, apt-daily-update, 
apt-daily-upgrade, LCDd, phpsessionclean, tinysshd, watchman, rngd, 
isnsd, isnsdd, usbmux, and VBoxBalloonCtrl.  atd is now a Linux-only 
service, with the BSDs now having an atrun service.



More packages
=

The new nosh-run-bcron, nosh-bcron-as-cron-shims, nosh-debian-crontab, 
and nosh-debian-crontab-anacron packages deal in running the services 
and providing the data files for various cron toolsets.  The former two 
deal in bcron, running its services and providing the crontab command as 
an alias for bcrontab; and the latter two (only available for Linux 
operating systems) deal in Debian's /etc/crontab file.


The new nosh-openrc-shims package contains shims for OpenRC's rc-service 
and rc-update commands.  And the new nosh-run-via-open-rc package 
contains OpenRC scripts for running the service manager.


The new nosh-linux-shims package contains shims for commands to be found 
in the non-portable util-linux toolset, such as setterm (more on which 
later).


The Debian desktop and server base -run packages no longer preset ntpd 
and openntpd, on the grounds that a range of such services exist and 
these are not necessarily the installed softwares.



More tools
==

New commands include getuidgid, userenv-fromenv, setgid-fromenv, envgid, 
printenv, setlogin, console-decode-ecma48, console-control-sequence, 
console-flat-table-viewer, console-input-method, and 
local-stream-socket-connect.


The userenv command is now a combination of two of these new commands, 
getuidgid and userenv-fromenv.  It has also gained options for not 
setting SHELL and USER/LOGNAME.


setlogin sets the login account that is associated with a kernel 
session, as returned by the logname command.


printenv is roughly equivalent to the conventional tool of the same 
name, except that it is a nosh/exec built-in command and that it 
supports several forms of output (including properly quoted rc.conf 
form, NUL-terminated form, and envdir form) in addition to the 
conventional human-readable form.  This built-in command makes a common 
idiom easier.  When combining clearenv, read-conf/envdir, and printenv 
to read a configuration setting, before the advent of the built-in 
command one had to employ `command -v printenv` (because clearenv unsets 
PATH).  Now one can invoke it as simply printenv.


One common use of this idiom is by the toolset's own build system and by 
the external configuration import subsystem, to read things like the 
amalgamated /etc/system-control/convert/rc.conf and an os_version file.  
Further to this, the amalgamated rc.conf now has an os_version setting 
on Linux operating systems, consolidating the code for obtaining that in 
one place.


console-flat-table-viewer is a full-screen TUI viewer for various sorts 
of common flat database tables.  It decodes the vis(3) encoding that is 
employed in various FreeBSD system tables.  It also handles tables that 
use the standard ASCII US, RS, GS, and FS characters.  File separators 
permit a form of continual update and redisplay if used in combination 
with pipes.


local-stream-socket-connect is the AF_LOCAL socket equivalent of 
tcp-socket-connect.



Improvements to existing tools and bug fixes


The Z shell completions now function better, and now cover a lot more of 
the commands in the toolset.


systemd service unit conversion has been modified to make use of the new 
environment commands.  The conversion tool in particular makes use of 
these when converting per-user Desktop Bus services.  The 
EnvironmentUser extension has been replaced by an EnviromnentUserOnly 
extension, so that User and Group are consistently the sources of the 
user account and primary group. Additional settings now supported by 
convert-systemd-units include RuntimeDirectoryGroup, 
RuntimeDirectoryPreserve, WantsMountsFor, AfterMountsFor, and 
RequiresMountsFor.


convert-systemd-units now also supports %T, %V, and %E expansions and 
snippets files.


By analogy to \S, the login-banner command now also recognizes the \N 
sequence.


The extern

Re: nosh: User-space virtual terminal test and questions

[Jonathan de Boyne Pollard]

The blue/green problem only manifested itself with certain terminal 
types, which I was not regularly using.


Since you have 1.39 built, and as long as you built from today's 
snapshot, you can be the second person in the world to enjoy this:


   while true ; do printf '\f' ; top -b -n 1 | sed -e '1,6s/^/#/' ; sleep 1 ; 
done | console-flat-table-viewer --header-count 1

Substitute

   top -b all | sed -e '1,8s/^/#/'

on FreeBSD, or indeed

   procstat -a

Notice the |ascii| format, too.

So what the world needs now is for M. Bercot to give us a Linux+BSD 
version of |s6-ps| that uses US and RS, or |vis()| encoding.  (-:


The FreeBSD people definitely need to give us a |procstat| that uses 
|strvis()|, given that the output of |procstat| is explicitly documented 
as being a table and |vis()| encoding is actually documented for this 
type of table in FreeBSD.  (See the manual for |fstab|, for example.)

Re: nosh: User-space virtual terminal test and questions

[Jonathan de Boyne Pollard]

Guillermo:

The combination of vboxvideo with |console-fb-realizer| was explosive, 
I got a (guest) kernel panic.


|console-fb-realizer| does not employ the arcana of framebuffer I/O, it 
just using little more than the mode setting API and memory mapped I/O, 
so I'm initially going to lay that one at the door of the driver.  I 
have had poor experiences with it myself.  Try the |--80-columns| option 
(which prevents |console-fb-realizer| trying to switch to some of the 
more exciting high-resolution modes) and ensure that your driver exactly 
matches your kernel version.


I currently test |console-fb-realizer| on real hardware.  (-:

Guillermo:

I [...] managed to deal with the BSDness of the requirements on font 
and keyboard map files with the help of FreeBSD's SVN repository :)


For those reading this, there is a whole chapter in the _Guide_ on the 
multiplicity of places whence one can obtain fonts, keyboard maps, and 
input methods.  The toolset does not come with them for two major 
reasons.  First, it is a deliberate design decision to use existing 
formats and not require new ones be created from scratch.  Second, they 
come under a variety of copyright licences, and it is simpler to just 
keep those separate.  Quite a few of them are already packaged up for 
operating systems.  I discovered /yet another/ collection of input 
method files just recently ... in a Debian package.


Guillermo:

1) What is the proper way (if any) to switch between kernel VTs and 
user-space VTs?


|console-fb-realizer| does not have hot key chords for initiating KVT 
switching that get stripped out of input processing and enacted 
locally.  All such chords are sent down the input FIFO to a /user-space/ 
VT multiplexor.  It could have, as there is room for such a specialized 
type of actions in the keyboard map.  Since I need to do more work on 
having multiple keyboard maps in a single |console-fb-realizer| 
instance, I might look into it.  But at the moment 
|console-multiplexor-control| pointed at a KVT is the tool, and I am not 
going to hold off 1.39 specifically for this.  I was going to do that 
work in 1.40.


Guillermo:

2) Key combinations with Alt (e.g. Alt + f or Alt + b to move forward 
or backward one word on Bash) did not work with neither 
|console-fb-realizer| nor |console-termio-realizer|. Any idea why, or 
does this just happen to me?


I have an idea why.  I have put improving it on the to-do list. It's not 
you.  (-:


Guillermo:

3) More strangely, Ctrl + x (e.g. to exit from GNU nano) did not work 
with |console-termio-realizer|, but did with |console-fb-realizer|.


Control+X is ␘ (cancel, U+0018) which cancels an in-progress ECMA-48 
escape or control sequence.  This is documented in ECMA-48:1991 § 8.3.6 
and in § 4.3.5 of DEC's _VT520/VT525 Video Terminal Programmer Information_.


|console-termio-realizer| decodes its input with a fully-fledged ECMA-48 
decoder, that implements the DEC/ECMA-documented behaviours of both ␘ 
and ␛ on the decoding state machine; rather than with a perennially 
incomplete collection of ad-hoc pattern recognitions as 
|console-ncurses-realizer| does.  It's the same decoder that 
|console-terminal-emulator| uses for output.  You can see ␘ in action 
with the new |console-decode-ecma48| tool in 1.39, which also uses the 
same decoder.  Type in an incomplete CSI sequence by hand, and cancel it 
with ␘.  It will be discarded.  The same will happen if you print one 
and cancel it, as output to the terminal emulator.


Although it would be entirely legitimate for a terminal to send ␘ in the 
middle of (say) a DECFNK control sequence in order to cancel it, it 
would be unusual, so I'll add another exception to ECMA-48 decoding for 
this, that |console-termio-realizer| can select.


ECMA-48 is of course nothing to do with the input protocols that 
|console-fb-realizer| speaks with its input devices.


Guillermo:

4) With |console-termio-realizer|, green is blue and blue is green :D 
Not with |console-fb-realizer|, though.


Part of my usual Z shell prompt is green, so I am confident that I would 
have noticed it being blue in error.  (-:


What terminal type is |console-termio-realizer| outputting to?  Is this 
a GUI terminal emulator with a colour palette that has been altered from 
the standard colour set?  Is it a 24-bit-colour-capable terminal?  Use 
|console-control-sequence --foreground blue|console-decode-ecma48| from 
version 1.39 on that terminal.  What do you see?  SGR 34?  SGR 38;5;4?  
Or SGR 38;2;... ?

Re: s6-ps

[Jonathan de Boyne Pollard]

That said, I would probably write my own portable tool if it came to 
that, as my gripes with |ps| are not to everyone's tastes.


 *

   It is the 21st century.  The FreeBSD, OpenBSD, and NetBSD |ps|
   programs all use |-| as the command-line option introducer, and have
   done since /at least/ BSD 4.4 Lite in 1994
   . 
   A "BSD personality" that gives me the interface from the 1980s is

   not /actually/ a BSD personality that gives me |ps| as it exists and
   as I use it on the BSDs.  This is neither compatibility nor
   interoperability.

 *

   The defaults are wrong.  Much of my GUI is processes that run in
   dæmon context.  By default, I /want/ to see processes in sessions
   that have no controlling terminal.  The |-x| option is the wrong way
   around.

 *

   The defaults are wrong.  Most of the time I want to see all users'
   processes.  The |-a| option is the wrong way around.  (Yes, most of
   the time I want to "see processes from other users" in Windows, too.)

 *

   Indeed, even in the unusual case where I am interested in only my
   own processes, my own processes /include/ set-UID processes spawned
   by processes running as my account, such as |urxvtd|, which is /my/
   Unicode rxvt dæmon.

 *

   The defaults are wrong.  Most of the time I do not want to see
   kernel processes.

 *

   When I use both the |comm| and |args| columns, I don't need an extra
   copy of the former put in brackets in the latter.

 *

   When I use the |comm| and |args| columns with |f|, I don't need both
   columns to have the tree diagram added to them, especially as it
   usually then obscures what is in the actual column, that I wanted to
   see.  Notice how FreeBSD |ps| only puts the tree diagram in if
   |args| is the last column.  Do that.

 *

   Your "BSD personality" should be giving me |-d| anyway, not |f|, as
   that is the actual option on the BSDs.  I don't want a tool with
   Multiple Personality Disorder and Jack-of-All-Trades option parsing,
   though.  I want a |ps| tool that is self-consistent and logical,
   with /one/ personality across all of the platforms that it works on,
   and /one/ way of doing most things (albeit allowing both short and
   long command-line options).

 *

   In fact, give me the tree diagram as its own, first class, column,
   as well.

 *

   My terminal emulators speak UTF-8 and have switchable fonts. Their
   grave accent sometimes really looks like a grave accent; and their
   minus sign sometimes really looks like a minus sign.  Give me proper
   line drawing characters.  It's ridiculous giving me a "BSD
   personality" from the 1980s and yet a TUI that does not even do TUI
   line drawing as well as it was done on MS/PC/DR-DOS in the 1980s.

 *

   The defaults are wrong.  It is the 21st century.  I haven't used a
   terminal with only the choices of 80 or 132 columns in years.  Stop
   giving me 132 columns as the primary alternative behaviour.

 *

   The Single UNIX Specification has standardized the |COLUMNS|
   environment variable as a way for me to tell you how many columns I
   want, for many years, now.  Don't give me idiosyncratic command-line
   option mechanisms of your own. Don't give me "It's unspecified.". 
   Give me the |COLUMNS| environment variable.


 *

   The defaults are wrong.  I'm piping your output through |less|,
   which knows how to sideways scroll.  Release your right margin when
   standard output is not a terminal device.  I do not need the final
   column to be of "undefined" width.  I need it to be defined, as
   having unlimited width.  For any column type, not just one or two of
   them.

 *

   Indeed, if I am piping output give me the option of releasing the
   width restrictions on all columns, not just the last one. I might
   want to have both the |env| and |args| columns, with neither truncated.

 *

   Don't give me |-u| , |-v| , |-l| , |-f| , |-j| and tell me that they
   are "user", "virtual", "long", "full", and "jobs", especially if you
   aren't going to give me long options or aren't even going to
   document what those are.  Give me at the least a |--format| option
   that takes |user|, |virtual|, |long|, |full|, and |jobs|.  Indeed,
   give me a |--format| option that has /all/ of the shorthands from
   |procstat|.  (But that, unlike |procstat|, can combine with |-o| and
   itself rather than give me a mutually-exclusive choice amongst
   things none of which are /quite/ what I want.)

 *

   In fact, give me just |--output| for selecting columns individually
   and |--format| for adding predefined handy sets of columns, and
   allow them to combine as I see fit.

 *

   Give me |--no-headers| too.

 *

   Give me the option of TAB rather than SPC as column separator whilst
   you are at it.  Then I can run it through tools that understand
   tables-of-one-line-per-record-with-TAB-separated-fields text. (No

Re: s6-ps

[Jonathan de Boyne Pollard]

Laurent Bercot:
I just wasn't aware of people besides me actually using s6-ps. :) 


I used it as an example of not being a mess.  (-:

s6 and s6-rc are actually ports/packages in FreeBSD and s6 is a package 
in Debian.  Alas, the Debian world has not yet caught up with the other 
toolsets, and the third-party Debian packaging for s6-linux-utils and 
the others has yet to make it into Debian's own repository.


* https://svnweb.freebsd.org/ports/head/sysutils/s6/

* https://svnweb.freebsd.org/ports/head/sysutils/s6-rc/

* https://packages.debian.org/sid/s6

* https://github.com/multiplexd/s6-packaging

You could, in theory, make s6-ps not Linux-specific.  It would require 
interposing a layer that abstracted reading the information out of the 
kernel, though.  It's perhaps not worth it, because (for example) the 
FreeBSD ps is not nearly as encumbered by the mess that results from 
multiple personalities and Jack-of-All-Trades option parsing, albeit 
that it still has confusing column headings and does not have an env column.


* https://unix.stackexchange.com/questions/492513/

Re: s6-ps

[Jonathan de Boyne Pollard]

Laurent Bercot:

 That functionality:
 - isn't available in procps.


I think that you should not let that stop you.  (-:

Laurent Bercot:

I modelled the available columns from procps' ones.


I think that you should not take that any further than you have 
already.  This is the right point to stop.  (-:


You have a tool whose usage is a lot less messy, and a fair degree more 
straightforward, than that ps.  You're fortunate *not* to have all of 
the quirky behaviours that I mentioned in the Stack Exchange answer.  
Having the three pieces of information in three columns with just three 
labels, without all of the ifs and buts, and having the environment 
strings be a column in their own right, and not an option that modifies 
another column's behaviour, is far simpler to explain.

s6-ps

[Jonathan de Boyne Pollard]

* https://unix.stackexchange.com/a/492346/5132

It occurs to me that s6-ps would gain from having an "exe" column.

setterm and the SAF

[Jonathan de Boyne Pollard]

* https://unix.stackexchange.com/a/491883/5132

* http://jdebp.eu./Softwares/nosh/guide/TerminalCapabilities.html

* http://jdebp.eu./Softwares/nosh/guide/TERM.html

It is a toolset for managing terminals as well.  (-:

* http://jdebp.eu./FGA/unix-service-access-facility.html

As was this.

Re: Log rotation issue with runit

[Jonathan de Boyne Pollard]

Dmitry Bogatov:

After some investigation I found code, that caused issue, but it seems 
that it was written with some purpose, yet I fail to understand that 
purpose.



What you need are manuals that tell you about this mechanism. (-:

* http://jdebp.eu./Softwares/nosh/guide/cyclog.html#COMPATIBILITY

* http://skarnet.org./software/s6/s6-log.html#Logdirs

* http://jdebp.eu./Softwares/djbwares/guide/multilog.html#ROTATION

* http://smarden.org/runit/svlogd.8.html#sect4

Re: Log rotation issue with runit

[Jonathan de Boyne Pollard]

Alex Efros:

I'm not sure is it good idea to include .u files in usual rotation 
process at all, and especially handle them just like .s files. If 
several crashes happens for some reason in a short period of time this 
will result in deletion of all log files except last (say) 10 .u 
files, which are usually small and may contain just one line. 
Replacing 10MB of last logs with 10 last log lines doesn't sounds like 
a nice idea.


Then cap the log directory by total file size rather than file count.  
M. Bercot and I both provide tools with this feature.  "I want to keep 
the last 10MiB of logs." is exactly what it is for.


* http://jdebp.eu./Softwares/nosh/guide/cyclog.html

* http://skarnet.org./software/s6/s6-log.html

Re: Log rotation issue with runit

[Jonathan de Boyne Pollard]

Laurent Bercot:
If loggers kept appending to "current" instead, it could have corrupt 
information in the middle of a "current" file, which would them be 
archived as a .s (intact file), which wouldn't be good. 


Examples:

* https://unix.stackexchange.com/questions/237321/

* https://askubuntu.com/questions/356981/

More Answers

[Jonathan de Boyne Pollard]

* https://unix.stackexchange.com/a/489949/5132

This may be of interest to people looking for some (brief) comparative 
analysis.  Including the further reading.  (-:

Re: runit don't restart service.

[Jonathan de Boyne Pollard]

Yoshinori Sato:


root 10647 10592 0 7月13 ? 00:00:00 /opt/LifeKeeper/sbin/runsv lkcheck

terminate service
[root@nodeA ec2-user]# kill 10647 12521

That is not terminating a service.  That is terminating the service 
supervision subsystem.  If you kill the process that actually does the 
restarting, you really should not expect that process to then restart 
anything.  The behaviour of runsv in response to SIGTERM is documented 
in its manual; as are the ways to properly tell it to terminate a 
service, as opposed to what you did here.


Use the runit manual in preference to the LifeKeeper manual here, too.  
In particular learn the use of runit commands like


# /opt/LifeKeeper/sbin/sv status /opt/LifeKeeper/etc/service/*

rather than the clumsy and error-prone grepping of the process list that 
is given in the LifeKeeper manual.

Re: has anyone gotten nosh to build on Void Linux?

[Jonathan de Boyne Pollard]

Guillermo:


 *

There is a |curses.h| file in both Void's 'standard' headers
directory and in a subdirectory named '|ncursesw|' (for the narrow
character and wide character versions of the library,
respectively). Normally, that would be |/usr/include/curses.h| and
|/usr/include/ncursesw/curses.h|.

On that score:  Those of you who have peeked at 1.38 will possibly have 
noticed the manual for the new |console-termio-realizer| command, which 
has a section explaining how it realizes onto terminals.  I have some 
things that I am mulling over.


 *

   The |chkservice| command (also new, and which you might also have
   spotted) uses wide character ncurses.  I am wondering whether it
   should employ |console-termio-realizer|'s mechanism instead.

 * The |monitor-fsck-progress| command uses wide character ncurses, but
   there is no real use case for it running against a terminal that
   does not understand ECMA48 control sequences.  I am wondering
   whether it should employ |console-termio-realizer|'s mechanism instead.
 *

   The only programs that do not explicitly use /wide-character/
   ncurses (and so require the additional slightly different
   development tooling) are |service-status|, and the |start|, |stop|,
   and |reset| subcommands of |system-control|.  And they only employ
   it for colourization, which the ncurses and terminfo models aren't
   particularly good for in the first place.  I am wondering whether
   they too should employ |console-termio-realizer|'s mechanism instead.

The general upshot would be that only |console-ncurses-realizer| ends up 
using wide character ncurses.  Only the one place for patches, and only 
the one flavour of curses; with the tradeoffs that are given in the 
|console-termio-realizer| manual.

Re: has anyone gotten nosh to build on Void Linux?

[Jonathan de Boyne Pollard]

Guillermo:


 *

The |wall|(1) program is available in Void's 'standard'
executables directories.

Because that's what nosh's build system and executables are going to 
expect, as far as I know.


For the sake of other readers: note that it is not the build system that 
employs the |wall| program.  (That would be bizarre.)  It is the 
|shutdown| program, at runtime, as described in its manual.  This is not 
a requirement specific to the nosh toolset's |shutdown| program. The old 
|shutdown| program in FreeBSD and OpenBSD has the same requirement, for 
example, as did the |shutdown| program in 386BSD.  Using the |wall| 
program to do its job rather than duplicating (a fork of) its code into 
|shutdown| (which the van Smoorenburg 
, 
upstart 
, 
and systemd 
 
|shutdown| programs all have) has a long history in the Unix world.

Re: has anyone gotten nosh to build on Void Linux?

[Jonathan de Boyne Pollard]

As someone porting to another operating system, you need also to pull up 
the /_nosh Guide_/  and 
read the _Packages and Ports_ chapter.  There is a section in that 
chapter of things that require attention (in addition to figuring out 
your operating system's required development packages) when porting to 
another operating system.  You'll notice from the current source that 
several people have already told me about their particular operating 
systems.  This is stuff that needs to be figured out just once per 
operating system, as long as the first person to do so tells me the 
right answers.  (-:

Re: has anyone gotten nosh to build on Void Linux?

[Jonathan de Boyne Pollard]

Chris Brannon:

When I run package/compile from the root of the unpacked source tree, 
I get:



redo: ERROR: all: Cannot find .do file to use.


You need to make your operating system POSIX-conformant.  I make fairly 
liberal use of some standard utilities in the build process including 
|pax 
|, 
|uname 
|, 
|mv 
|, 
and |expr 
| 
amongst others.  A POSIX-conformant operating system is a fairly minimal 
expectation for a build environment.  (It is not as though the build 
process is requiring, for example, the Bourne Again shell and its 
non-POSIX extensions for building from source on host operating systems; 
as some softwares do.) The source package page 
 notes the 
*non*-standard tools that are needed.


This information is pre-encoded in the |package/debian/control| file 
that I supply for Debian Linux, to be automatically checked by 
|dpkg-buildpackage|.  (There's no equivalent list that I supply for the 
BSDs; but it is also the case that the BSDs provide nearly all of this 
stuff, including the non-POSIX |install| tool, out of the box, in 
base.)  If you are building on a different Linux operating system, that 
is a good hint, with two caveats: Your Linux operating system won't 
necessarily package things up the same way as Debian or have the same 
package dependency tree, and Debian mandates a group of tools as 
standard without their needing to be mentioned in a 
|package/debian/control| file 
. 
Furthermore: Thomas Caravia has already worked out the build 
requirements for Arch Linux in terms of pre-requisite Arch packages, and 
the doco for Archnosh is a further source of clues for building on other 
Linux operating systems.

Re: [RFC] prctl: (Please do not!) Deprecate non PR_SET_MM_MAP operations

[Jonathan de Boyne Pollard]

In case you weren't aware, programs can make use of a facility on Linux 
to make their ps listings contain appropriate command-line arguments and 
environment strings. There's also a similar kernel API on FreeBSD.  This 
is used by the built-ins in nosh to good effect, even more so in the 
forthcoming version 1.38, allowing system adminstrators to debug things 
more easily from ps listings by showing them the actually effective 
states of the chained argument and environment vectors rather than what 
they were at the beginning of a chain of built-in commands.


* https://unix.stackexchange.com/a/438007/5132

* https://unix.stackexchange.com/a/432681/5132

The subject of deprecating and removing the API that permits this came 
up recently on the Linux Kernel Mailing List.  I raised my concerns, and 
the Linux Kernel developers have been laudably responsive on the 
subject.  As I said to them, I would welcome a better API, that allows 
setting these things atomically, start and length together as a pair.


* https://lkml.org/lkml/2018/4/3/739

* https://lkml.org/lkml/2018/5/6/127

* https://lkml.org/lkml/2018/5/7/78

* https://lkml.org/lkml/2018/5/7/79

* https://lkml.org/lkml/2018/5/7/80

* https://lkml.org/lkml/2018/5/7/81

Parsing /etc/os-release

[Jonathan de Boyne Pollard]

For those who might be parsing /etc/os-release, and for those who have 
seen the way that the nosh toolset does it, in its build system and in 
the external configuration import subsystem:


* https://unix.stackexchange.com/a/433245/5132

uschedule service bundles supplied with the nosh toolset

[Jonathan de Boyne Pollard]

Guillermo:

the 1.38 development version


On that subject: If you are tempted to play with the new uschedule 
service bundles, note what is written in the new section in the Guide 
about uschedule.

G++ warnings for set-control-group-knob from nosh-1.37

[Jonathan de Boyne Pollard]

The macro generates no warning in the GNU C library in Debian 9, but I 
should have caught that overload that was not passing by reference.  It 
is fixed now, thank you. The in-development source archive for 1.38 is 
in the usual place.

cloux

[Jonathan de Boyne Pollard]

s6 and runit get a mention at http://cloux.org/init/ .

It's not a very technical or in-depth treatment, resorting more to 
quotations.  For example: There's a lot more to say about system 5 rc, 
which it conflates with system 5 init.  And AIX had more run levels than 
the eleven that it claims.


* http://jdebp.eu./FGA/system-5-rc-problems.html

It misses entirely that s6's "unusual service commands" are direct 
descendants of Bernstein's svc command and not really unusual at all.


* http://cr.yp.to/daemontools/svc.html

* http://untroubled.org/daemontools-encore/svc.8.html

* http://b0llix.net/perp/site.cgi?page=perpctl.8

* http://www.skarnet.org/software/s6/s6-svc.html

* http://jdebp.eu./Softwares/nosh/guide/svc.html

And that runit's "weird svlogd logger filter configuration syntax" is a 
direct descendant of Bernstein's multilog filter.


Also, the command that it is hunting for in another section is systemctl 
list-units  --state=failed .

Re: nosh version 1.37

[Jonathan de Boyne Pollard]

Guillermo:

Did someone else help with that?


No.  I am reliant upon you and anyone else who pipes up.

Guillermo:

if you are going to go this route:


Yes.  I thought about it and it seemed that doing otherwise places the 
burden on the wrong person.  The other approach is to defer running the 
*.do files until the end-user runs the import subsystem in xyr 
$HOME/.config/service-bundles/convert/ directory. That's placing the 
burden of working out how GNOME confd, dbus-daemon, and others are built 
for any given operating system, on every single end-user.  The burden of 
that should be just on the shoulders of the person doing the porting to 
the operating system in question and making the relevant operating 
system packaging.


There's one more to go.  Have a look at the Packages and Ports chapter 
on the copy of the nosh Guide on the WWW site.  (I might move this doco.)

Re: nosh version 1.37

[Jonathan de Boyne Pollard]

Guillermo:

redo-ifchange[2]: ERROR: services/dbus-broker.service: Not done.
redo-ifchange[2]: ERROR: services/dbus-daemon.service: Not done.
redo-ifchange[2]: ERROR: services/system-wide.conf: Not done.
redo-ifchange[2]: ERROR: systemd/service-manager.socket: Not done.
redo-ifchange[2]: ERROR: convert/per-user/at-spi-dbus-bus.service: Not 
done.

redo-ifchange[2]: ERROR: convert/per-user/gconfd.service: Not done.
redo-ifchange[2]: ERROR: convert/per-user/per-user.conf: Not done. 


A snapshot of the 1.38 development is in the usual place.  See how far 
that gets you.

Re: nosh version 1.37

[Jonathan de Boyne Pollard]

Guillermo:
But the 7 .do scripts I talked about are executed (indirectly) by the 
package/compile script from the source package, and they just create 
symbolic links. To be precise, they are executed by the source/all.do 
script, because they are named in 'echo' commands that feed their 
output to the 'xargs -r redo-ifchange' invocation.


Hmmm.  You're right.  Maybe that should be deferred.  I shall think 
about this.


As the release notes said, some of these are works in progress.  I was 
expecting hiccoughs in the reorganized external config import, which I 
had to rush to meet the deadline.  (I am also expecting that we have 
have to work out what the non-Debian equivalents for dpkg-divert are, if 
any.  It turns out that dbus-daemon has got worse, which has had 
knock-on effects.)  Get me those Gentoo equivalences, and I can tweak 
this stuff for 1.38.

Re: nosh version 1.37

[Jonathan de Boyne Pollard]

Guillermo:

This is going to happen for every [GNU/]Linux distribution that is 
not  Debian, Arch, CentOS or RHEL. It does not... uh... look very 
portable :/




That is a reflection of reality.  It isn't very portable.  But you are 
misidentifying what it is.  It is the location of these various off-PATH 
executables, which have at least four different possible locations; and 
the name of the user that runs the system-wide Desktop Bus broker.  Yes, 
the people who have packaged these up don't agree on the same off-PATH 
locations; and their names are not very portable; and they don't agree 
on system account names.  If you can work out which of the existing four 
Gentoo's packaging for things like at-spi-bus-launcher is most like, 
like M. Caravia did for Arch Linux, I'll put in some mappings for 
Gentoo.  But do not blame the messenger for the news that GNOME confd is 
in four different places across Arch Linux, RedHat Linux, Debian/Ubuntu 
Linux, and FreeBSD/TrueOS.


You might find that Gentoo mainly or wholly follows the pattern of one 
of the others.  But from mine and M. Caravia's experience of operating 
systems not even being entirely *self* consistent in this area, with no 
overall pattern being followed even for Desktop Bus softwares on a 
single operating system, you probably shouldn't raise your hopes about 
that.  (-:


And yes, M. Caravia originally worked these out as a set of patches for 
Arch Linux, working out what should happen for arch:* .


Guillermo:

Additionally, the convert/per-user/*.do scripts' 'read_os' function 
calls 'exec' via absolute path /bin/exec instead of relative path 
../../exec, which is not going to work if nosh isn't already installed 
(chicken and egg).




There's no chicken and egg problem.  The external formats configuration 
import requires (amongst others) the nosh-exec package.  There is no 
requirement in the opposite direction, so simply install nosh-exec 
first.  This happens automatically with the provided packaging for 
FreeBSD/TrueOS and Debian, and with M. Caravia's Archnosh packaging, 
because of nosh-bundles depending from nosh-exec.  If one installs the 
former, the package managers will have made sure that the latter is 
installed too.


Moreover that is not the relative path from your 
$HOME/.config/service-bundles/convert/ directory to /bin .  Not that 
there of course *is* a stable single relative path for such a thing, 
given that home directories can be anywhere from /export/home/guillermo 
to /var/lib/mysql .  Nor that, as mentioned, such a relative path is 
needed, given that one installs the toolset before running the external 
configuration import.

nosh version 1.37

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.37 .

 *

   http://jdebp.eu./Softwares/nosh/

 *

   
https://www.freebsd.org/news/status/report-2017-07-2017-09.html#The-nosh-Project

 *

   http://jdebp.info./Softwares/nosh/

Some of the changes in this release are works in progress, that you will 
see fully realized in version 1.38 or later.


Changes include:

 *

   There is a new chapter in the /nosh Guide/ for those wishing to make
   packages and ports of other softwares, or add service bundle support
   to existing packages and ports.

 *

   The external formats configuration import subsystem has been
   reorganized a bit.

 o

   Nothing uses the |JAVA_HOME| import system any more, where
   service bundles explicitly have their |JAVA_||HOME| variables
   set by configuration import, although it is retained.  All
   service bundles instead use the |find-matching-jvm| mechanism to
   auto-detect a JVM matching their chosen criteria at start time.

 o

   The per-user services import is now in two parts. System-wide
   import sets up a |$HOME/.config/service-bundles/convert/|
   subdirectory for each (real user) user account; and each user
   can then use that, which contains a subordinate per-user
   configuration import mechanism, to set up imported per-user
   service bundles for things.

 o Per-user service source files for Desktop Bus and other services
   are now in their own subdirectory, as are converted keyboard
   maps for the userspace virtual terminals.
 *

   |static-networking| external format configuration import has been
   enhanced to set up |snort@/interface/| services and to handle
   |ipv6_cpe_wanif| and |ipv6_activate_all_interfaces| from |/etc/rc.conf|.

 *

   There is a new |make-read-only-fs| chain loading tool that is a
   placeholder for now.  It is used in some service bundles generated
   by the |convert-systemd-units| tool, which now recognizes and
   converts |CPUAffinity|, |ProtectHome|, |ProtectSystem|,
   |ReadWriteDirectories|, |ReadOnlyPaths|, and |InaccessiblePaths|
   settings.

 *

   Per-user management has been augmented, finally fixing the problem
   of |system-control| locating the per-user manager by giving the
   per-user manager an optional listening FIFO open file descriptor,
   which it uses to listen for user-wide state change commands.
   |system-control --user| |halt|/|normal|/|sysinit|/&c. now send
   commands via this FIFO, and each user's |user-services@/username/|
   service bundle now uses |fifo-listen| to set up the FIFO and creates
   the |per-user-manager/| subdirectory in |/run/user|.

 *

   There are some more service bundles in the collection that comes
   with the toolset: clickhouse-server, hue, udhcpc-log, minissdpd,
   rtkit-daemon, accounts-daemon, gdm3, speech-dispatcher, gdomap,
   blueman-mechanism, and sysvipc.

 *

   The per-user configuration import now recognizes and sets up
   per-user service bundles for a whole lot more per-user services.

 *

   On FreeBSD/TrueOS systems |setup-machine-id| now writes
   |/usr/local/etc/machine-id|.

 *

   The userspace virtual terminal services, the multiplexor and the
   terminal emulators, no longer run under the aegis of the |daemon|
   system account.  Rather, they now have their own dedicated accounts
   under whose aegides they run.  To go with that, there is now a
   |user-vt-realizer| group to which users can be added to grant them
   realizer (i.e. front-end I/O) access to the system-wide userspace
   virtual terminals.

 *

   A common build problem across several toolsets that occurs if one
   has set a |CDPATH|, has been fixed.  Various tweaks have also been
   made to make life easier for Archnosh and ports to other operating
   systems.

djbwares version 7

[Jonathan de Boyne Pollard]

djbwares is now at version 7.

 *

   http://jdebp.eu./Softwares/djbwares/

 *

   http://jdebp.info./Softwares/djbwares/

There are only a few changes.  A common build problem across several 
toolsets that occurs if one has set a |CDPATH|, has been fixed. 
|dnscache| now has a |FORWARDFIRST| mode.  And a bug in |tcpserver| that 
manifests itself when |tcpserver| inherits no open standard I/O file 
descriptors has been fixed.

redo version 1.4

[Jonathan de Boyne Pollard]

redo is now at version 1.4

 * http://jdebp.eu./Softwares/redo/

The only change from 1.3 is a belt-and-braces protection mechanism that 
prevents cleanup code from being told to delete a parent directory.

The shape of things to come

[Jonathan de Boyne Pollard]

To let you know in advance, Thomas:

In 1.37, a pre-relase source archive for which will be available for you 
later today in the usual place, I am making several changes.


Several of your patches are going to be shorter or even entirely moot.  
The build system will try to explicitly pick some .service.arch-linux 
files when it detects Arch Linux.  Please check that I have got these right.


I finally need to have pre-install and post-remove maintenance actions, 
because of D-Bus in Debian 9.  So I have re-vamped the maintenance 
script generation.  You should find it easier to map the new system onto 
Arch Linux, as the Debian file naming system is now closer to the BSD 
one.  Please adjust as required.  Have a look at the new 
package/debian/gencontrol for pointers to adjustments for your 
scriptletbuilder.  Notice that there's now no occasion where there is a 
wholesale override script, nor is there any occasion where there is no 
.funcs file.  Also beware that you'll need to patch out dpkg-divert in 
nosh-service-management.*.extra and the dbusprefix in package/stage.

Re: nosh build problems (Re: Compatibilities between runit and s6 (re: )

[Jonathan de Boyne Pollard]

Guillermo:


Jonathan de Boyne Pollard:

What does line 843 of your |/usr/include/curses.h| actually say? 
Because that's not the location of that function declaration in that 
header in either Dickey ncurses or Acton pdcurses as far as I can 
see. So what curses are you actually using?


Dickey ncurses' |configure| script has options |--enable-const| and 
|--disable-const|. [...]


I am aware.  It doesn't shift that declaration onto line 843 of that 
header. It was quite a few lines further down a few years ago 
<https://github.com/ThomasDickey/ncurses-snapshots/blame/5246b41942b2734b012b118903ca869b1a1fae2f/include/curses.h.in#L854>, 
and is even further down nowadays.  So either this is a significantly 
older Dickey ncurses, older even than what is in Debian 8 
<https://anonscm.debian.org/cgit/collab-maint/ncurses.git/tree/include/curses.h.in?h=jessie#n854>, 
or it is something else.

Re: Compatibilities between runit and s6 (re: Incompatibilities between runit and s6?)

[Jonathan de Boyne Pollard]

Charlie Brady:


OK, so this statement is incomplete:

It also does not tell you that you need other POSIX-standard commands 
such as |mv| and |ln|.   But you need them, too.  I draw the line at 
listing every command, that POSIX-conformant systems have had to have 
for two decades and that are also listed in the Linux Standard Base 
, 
that is required.  That would be ridiculous.  I also don't mention 
things that /operating system doco/ says are fundamental for building 
packages, like build-essential 
 
and base-devel 
. 
Pointing out that you need |redo|, on the other hand, is pointing out 
something that is not POSIX, not in the LSB, and not in the operating 
system doco.  On those grounds, I should add that you need |xmlto|.

Re: nosh build problems (Re: Compatibilities between runit and s6 (re: )

[Jonathan de Boyne Pollard]

Charlie Brady:


/usr/include/curses.h:843:31: error:   initializing argument 1 of 'char*
tigetstr(char*)' [-fpermissive]

service-status.cpp:148:21: error: invalid conversion from 'const char*' to
'char*' [-fpermissive]
s = tigetstr(setaf);


This is a problem that I thought was confined to OpenBSD, and its 
not-|const|-correct ncurses library.  Certainly this has never been a 
problem on Debian nor on FreeBSD, and M. Caravia did not report any lack 
of |const|-correctness on Arch Linux.  What does line 843 of your 
|/usr/include/curses.h| actually say?  Because that's not the location 
of that function declaration in that header in either Dickey ncurses or 
Acton pdcurses as far as I can see.  So what curses are you actually using?

Re: Compatibilities between runit and s6 (re: Incompatibilities between runit and s6?)

[Jonathan de Boyne Pollard]

Charlie Brady:


Sorry, doesn't work for me:

You need a POSIX-conformant system with all of the POSIX utilities, 
including |pax| 
.  
The Debian |package/debian/control| file automatically ensures this on 
Debian, with its |Build-Depends:| information.  The BSDs have these 
tools /as standard/ in the base operating system, out of the box.  
Thomas Caravia both notes this in the Archnosh doco (in the section 
headed "Dependencies") and declares it as a |makedepends| in the 
|PKGBUILD| itself so that it is automatically ensured on Arch.  You are 
going to have to do this yourself on your operating system, too.

Re: Compatibilities between runit and s6 (re: Incompatibilities between runit and s6?)

[Jonathan de Boyne Pollard]

Thomas Caravia:


I'm not familiar with rpm but my packaging is just this:


You have prompted me to fill in a long-standing dangling hyperlink.

* http://jdebp.eu./FGA/slashpackage.html

Re: Compatibilities between runit and s6 (re: Incompatibilities between runit and s6?)

[Jonathan de Boyne Pollard]

Jonathan de Boyne Pollard:

The |service-manager| manual page from version 1.37 or later of the 
nosh toolset contains  a full breakdown of the control/status API in a 
|supervise/| directory, including the daemontools-encore extensions 
and the nosh service-manager's own extensions.  I am hoping to get 
version 1.37 released within the next week.  In the meantime, I have 
pushed the new manual page to the on-line copy of the /nosh Guide/ 
<http://jdebp.info./Softwares/nosh/>.  You can find the HTML-form 
manual page for |service-manager| hyperlinked from several pages in 
the /Guide/, including the _new Interfaces_ page.



Charlie Brady:

Do you know whether anyone has documented how to build on a non-Debian 
linux? An rpm spec file would be ideal, but just simple non-Debian 
centric build instructions would be better than nothing.


Yes.  I have.  The slashpackage-style build instructions, which are of 
course /not/ Debian-specific nor even /Linux-/specific, are actually 
right there on the source package WWW page.


The only thing that is Debian-specific is the step that gets one from a 
/built/ slashpackage-style package to Debian binary packages.  This is 
in the supplied |package/debian/rules| command.  But even that is not 
actually Debian-specific, as there is a similar |package/bsd/rules| 
command that does the same thing on the BSDs.


Thomas Caravia constructed the Archnosh PKGBUILD the same way. It does 
the slashpackage-style build, then takes the built slashpackage-style 
package and parcels it out into a set of Arch binary packages, with a 
light dusting of patches to a handful of service bundles.  It even 
mostly re-uses the Debian maintenance scripts.


(On that note, Thomas: You may enjoy the start that I have made in 1.37 
to a _Packages and Ports_ chapter in the /nosh Guide/.  Yes, I currenly 
plan for the |.p|/|.func| system to get at least a short mention.)


I was only expecting you to read the manual for doco on the common 
control/status API that you were interested in, not build the software.  (-:

Re: Compatibilities between runit and s6 (re: Incompatibilities between runit and s6?)

[Jonathan de Boyne Pollard]

Charlie Brady:

There is a certain level of compatibility between daemontools and 
runit, and I presume the same exists for s6. The devil is in the detils.


The |service-manager| manual page from version 1.37 or later of the nosh 
toolset contains  a full breakdown of the control/status API in a 
|supervise/| directory, including the daemontools-encore extensions and 
the nosh service-manager's own extensions.  I am hoping to get version 
1.37 released within the next week.  In the meantime, I have pushed the 
new manual page to the on-line copy of the /nosh Guide/ 
.  You can find the HTML-form manual 
page for |service-manager| hyperlinked from several pages in the 
/Guide/, including the _new Interfaces_ page.

Re: Incompatibilities between runit and s6?

[Jonathan de Boyne Pollard]

Laurent Bercot:

Also, admittedly, I simply did not want to read the systemd unit file 
documentation to understand how to start a s6 supervision tree from 
systemd. I will do the effort and come up with a small unit file 
suitable for this.




* https://lists.debian.org/debian-user/2014/09/msg01715.html

* http://jdebp.info./FGA/inittab-is-history.html

* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=752075#82

Re: cannot set terminal process group

[Jonathan de Boyne Pollard]

DGSJ:


Well, Crtl+C doesn't work, [...]

It seems to be the time of year for this question.  I just answered 
another variant of it, complete with daemontools-style chain loading 
tools.  Read the further reading and other questions, too.


* https://unix.stackexchange.com/questions/410318/

* https://unix.stackexchange.com/questions/255385/

* https://unix.stackexchange.com/questions/338214/

nosh version 1.36

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.36 .

 *

   http://jdebp.eu./Softwares/nosh/

 *

   
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

 *

   http://jdebp.info./Softwares/nosh/


   More Java tools

This release comes with the |find-default-jvm| and |find-matching-jvm| 
tools, which will set up the |JAVA_HOME| environment variable to point 
to a default/matching JVM directory, using the FreeBSD/TrueOS and Debian 
conventions for locating JVM directories.  To match these, 
|convert-systemd-units| now recognizes |JVMDefault|, |JVMVersion|, 
|JVMOperatingSystem|, and |JVMManufacturer| extensions to the systemd 
unit file format.



   Tool improvements

|convert-systemd-units| now recognizes a |MachineEnvironment| extension 
to the systemd unit file format, which controls the generation of an 
invocation of |machineenv|.  It also now recognizes and translates 
|RDMAHCAHandlesMax| and |RDMAHCAObjectsMax| settings.


The |unshare| command now has flags for specifying process ID and user 
ID namespaces on Linux.


The |setup-machine-id| command now correctly falls back to the old D-Bus 
files on FreeBSD, which it had not been doing because of a bug.



   New system management features

In support of an initiative by Warner Losh, there is support for power 
cycling via hardware and a kernel that support it.


The system manager treats |SIGRTMIN+6|, unused in the systemd system, as 
a request to invoke a new |powercycle| service bundle; and 
|SIGRTMIN+16|, similarly unused, as the underlying actual powercycle 
request, which it translates to either |RB_POWERCYCLE| if it is present 
in the C library headers, or |RB_AUTOBOOT| if it is not. There is a new 
|system-control powercycle| subcommand, which defaults to sending these 
signals.


Note that the binary packages are currently built on a system that lacks 
|RB_POWERCYCLE| in the C library.


The compatibility |shutdown|, |reboot|, |halt|, and |poweroff| commands 
all now sport a new |-c|/|--powercycle| option.  There are new 
|fastpowercycle| and |powercycle| commands.  The |system-control init| 
subcommand now sports a new |c|/|C| argument, by analogy to |h|/|H|. And 
this is of course thus reflected automatically in the compatibility 
|telinit| command and the |initctl-read| server.



   Service bundles

Fixing an oversight in 1.35, the per-user |dbus| services are now 
renamed to |dbus-daemon| too.


There are a few more service bundles, including ones for |jenkins|, 
|apacheds|, |udisks2|, and |ndppd|.


The |linux-utmp| service bundle has been retired, in favour of a unified 
|utx| service bundle, which was previously FreeBSD-only, that operates 
across platforms.  In support of this, there is a new 
|login-update-utmpx| command, and a new |freebsd-shims| package that 
aliases that to the |utx| command on non-FreeBSD platforms.

SpaceFM using nosh service management

[Jonathan de Boyne Pollard]

For a wider readership:

> 
> I recently had issues where Firefox and the SpaceFM file manager would
> hang when started. Apparently they were calling dbus-launch.
> 

If they were calling dbus-launch, that means that they were unable to contact a
Desktop Bus broker and tried to run their own broker in the same session as
themselves.  Run system-control status dbus and system-control --user status
dbus to check that the system-wide and your per-user Desktop Bus brokers are
running and to see their logs.  Also look at /run/dbus/ and whether you have a
DBUS_SYSTEM_BUS_ADDRESS variable pointing at a different place.

I cannot speak to Firefox but a quick check with SpaceFM reveals that, once it
can contact a Desktop Bus broker, it tries to contact and auto-start the
org.freedesktop.UDisks2 server over the system-wide Desktop Bus.  So you'll need
a system-wide org.freedesktop.UDisks2 service that runs that server.  I've added
such a service bundle for 1.36.

Re: New reboot flag: -c for 'power cycle'

[Jonathan de Boyne Pollard]

Warner Losh, FreeBSD and embedded systems developer, has just invented a 
new shutdowngoal, in addition to the ones that we already know.  In 
addition to the conventional reset, power off, halt, and kexec goals; xe 
has added a power-off-and-then-on-again goal.  Xe has named it power 
cycle, and its use case is systems with hardware that can trigger a full 
power cycle under the kernel's controland where an ACPI reset or even an 
EFI ResetSystem() of type EfiResetColdjust do not cutthe mustard.


It involvesanew flag to the FreeBSD reboot() system call, RB_POWERCYCLE; 
new FreeBSD kernel functionality for enacting this with hardwares that 
support it; new -c (for "cycle") flags to the various reboot, halt, 
poweroff, and shutdown commands; and a new signal sent to process #1.


* https://marc.info/?l=freebsd-arch&m=150886821625848&w=2

* https://reviews.freebsd.org/D12777

This of course impacts those of us who write process #1 programs.

Since SIGWINCH to process #1 is already taken by Linux, I have adjusted 
my softwares to use SIGRTMIN+7 and SIGRTMIN+17 for the signals to 
process #1 for this, ready for when Linux eventually catches up with 
this idea.  (-:


(There is an on-going discussion of whether FreeBSD's old init program 
should use real-time signals, too.  It currently does not.But that is a 
side issue here. I have also added to my softwares powercycle and 
fastpowercycle commands and subcommands, a "c" or "C" argument to the 
compatibility telinit program (by analogy to the existing letters, 
although I might have to pick another letter), a powercycle target 
(again by analogy to existing targets), and a --powercycle long option 
equivalent for -c .)


Please do not use SIGRTMIN+7 and SIGRTMIN+17 for an incompatible purpose.

Re: New reboot flag: -c for 'power cycle'

[Jonathan de Boyne Pollard]

Warner Losh:

I was completely unaware of SIGWINCH being used like this on Linux. It 
didn't pop up in the quick research I did before implementing this.


But it would have been nice to know this sooner, but I'm OK with later 
since it isn't much later.




Some sources of information for you, and for anyone else interested:

* The "System event response" section of my system-manager(1). 
(http://jdebp.eu./Softwares/nosh/guide/system-manager.html or 
file:///usr/local/share/doc/nosh/system-manager.html if you have the 
nosh Guide installed)


* The SIGNALS section of Miquel van Smoorenburg's init(8). 
(http://svn.savannah.gnu.org/viewvc/sysvinit/sysvinit/trunk/man/init.8)


* The SIGNALS section of Gerrit Pape's runit(8). 
(http://smarden.org/runit/runit.8.html)


* The SIGNALS section of systemd(8). 
(https://www.freedesktop.org/software/systemd/man/systemd.html)


* The doco for signal handling in Joachim Nilsson's finit. 
(https://github.com/troglobit/finit/blob/master/doc/signals.md)


* Felix von Leitner (2004-09).  Speeding up the Linux boot process with 
minit. https://www.fefe.de/minit/minit-linux-kongress2004.pdf.  11th 
Linux Kongress.  p. 14.


* https://unix.stackexchange.com/a/191875/5132

* https://unix.stackexchange.com/a/197472/5132

* https://www.mail-archive.com/supervision@list.skarnet.org/msg01344.html

Re: execlineb's import conflicts with ImageMagick

[Jonathan de Boyne Pollard]

Guillermo:


This is known, see the 'execline-2.3.0.0' section of this announcement:

* 
https://www.skarnet.org/cgi-bin/archive.cgi?2:msp:1687:blbgbjnpebcmhkmelgjd



And these:

* https://lists.debian.org/debian-user/2017/10/threads.html#00164

* https://lists.debian.org/debian-devel/2017/10/threads.html#00127

Re: bug in tcpserver?

[Jonathan de Boyne Pollard]

Laurent Bercot:
s6-tcpserver doesn't have this bug. :P 


Neither do tcp-socket-accept, nor 
local-{stream,seqpacket}-socket-accept.  (-:


Just for the Hell of it, I have patched the djbwares' tcpserver to have 
if (0 != s) and if (1 != s) conditions, for version 7 whenever that will be.

nosh version 1.35

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.35 .

 *

   http://jdebp.eu./Softwares/nosh/

 *

   
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project

 *

   http://jdebp.info./Softwares/nosh/


   Networking

As I mentioned a week or so ago, the external configuration import 
subsystem now converts a Debian-style /etc/network/interfaces 
configuration file, via rc.conf settings, into the native networking 
subsystem.


There is also a whole new /Networking/ chapter in the /nosh Guide/, 
which explains this and several other things, including how Plug and 
Play integration interoperates with the networking services and what the 
native networking subsystem encompasses, to the level of what service 
does what and to what purpose.


Work on the Plug and Play integration is on-going, and I hope to have 
yet more for this, and indeed for other parts of the networking 
subsystem, in version 1.36.



   Packages

There are some Debian packages that declare that they need the logrotate 
package, even though they do not when run under nosh service 
management.  For their benefit there is now a nosh-logrotate-shims 
Debian package that is simply a dummy package that satisfies this need 
without setting up a spurious and unnecessary logrotate system.



   Service bundles

There are a few more service bundles, including ones for sysstat and 
elasticsearch.  The existing service bundles for things such as unbound, 
clamav, and freshclam have been augmented and fixed in response to user 
feedback.  And a bug that incorrectly resulted in the ldconfig service 
being disabled has been fixed.


The dbus services, the system-wide one and the per-user one(s), have 
been renamed to dbus-daemon.  This is because of the existence of a 
dbus-broker service bundle.  This is a placeholder for if the 
dbus-broker people ever fix it so that it works. dbus-broker does not 
provide a working system right now.  It is currently not possible to 
substitute dbus-broker for dbus-daemon on non-systemd systems, because 
dbus-broker is very tightly tied in to systemd's idiosyncratic D-Bus 
control interface.  It /only/ speaks the systemd-specific protocol, and 
knows no other way of stopping and starting services, not even the 
service command.  (In contrast dbus-daemon can still be configured to 
demand-start services using simple service management commands 
.)

Gentoo wiki

[Jonathan de Boyne Pollard]

For those people who had not noticed: Someone (I shall say no more. (-:) 
has been writing on the Gentoo wiki recently.


* https://wiki.gentoo.org/wiki/S6_and_s6-rc-based_init_system

* https://wiki.gentoo.org/wiki/Runit

* https://wiki.gentoo.org/wiki/S6

* https://wiki.gentoo.org/wiki/Daemontools-encore

Archnosh 1.35 networking

[Jonathan de Boyne Pollard]

Thomas:


[...] the new networking documentation [...]

This will benefit a general readership, as well as you looking towards 
Archnosh 1.35.


When you upgrade to 1.35, there will be two ways of configuring things.  
You can either write a Debian-style /etc/network/interfaces file, or you 
can employ a FreeBSD-like rc.conf.  Both styles will work on FreeBSD, 
TrueOS, Debian, and (I hope) Arch. The former you will find is 
translated into the latter.  For more detail on that translation 
process, see the new doco.  Here is an example from one of my machines.


jdebp % cat /etc/network/interfaces
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

auto lo
iface lo inet loopback
iface lo inet static
address 127.53.53.1
netmask 255.0.0.0
scope host
iface lo inet6 loopback
iface lo inet6 static
address ::2/128
scope host
allow-auto eth1
iface eth1 inet dhcp
iface eth1 inet6 auto
allow-auto rename2
iface rename2 inet dhcp
iface rename2 inet6 auto

jdebp % sed -ne '/etc.network.interfaces/,/console-setup:/p' 
/etc/system-control/convert/rc.conf
# Converted from /etc/network/interfaces:
network_interfaces="eth1 rename2 lo "
ifconfig_eth1="AUTO DHCP inet  "
ifconfig_rename2="AUTO DHCP inet  "
ifconfig_lo="AUTO inet  127.0.0.1 "
ifconfig_lo_ipv6="inet6  ::1 "
ifconfig_lo_aliases="inet 127.53.53.1 netmask 255.0.0.0
inet6 ::2/128"
# Converted from /etc/default/console-setup:
jdebp %

djbwares version 6

[Jonathan de Boyne Pollard]

djbwares is now at version 6.

* http://jdebp.eu./Softwares/djbwares/
* http://jdebp.info./Softwares/djbwares/

The main changes here are in dnscache and ftpd.

dnscache now has a built-in  resource record for localhost, like it 
already had a built-in A resource record.  I've slightly improved the 
way that it caches  resource record sets, to match the way that it 
was handling A resource record sets.  And it now caches SOA resource 
records.  There are also some minor improvements to the logging to 
decode SRV, A, and  records rather than print them in raw 
hexadecimal format.


The changes to ftpd were motivated by my pointing several WWW browsers 
at a publicfile FTP site and discovering that the WWW browsers adhere to 
the RFCs far less than they used to at the turn of the century.  You can 
read some of the saddening discoveries in the Hall of Shame.  I have 
enhanced publicfile ftpd to support OPTS, FEAT, SIZE, EPSV, and HOST; to 
interoperate better with some faulty FTP ALGs that cannot cope with an 
FTP server that one does not need to log in to; to interoperate better 
with some faulty WWW browsers that misuse CWD as a type testing 
mechanism; and to log things more clearly in order to diagnose such 
faults from server logs.  HOST support means that ftpd supports virtual 
hosting on FTP, which is explained in the manual, although it is hard to 
find any FTP client that employs this.


* http://jdebp.eu./FGA/web-browser-ftp-hall-of-shame.html
* http://jdebp.info./FGA/web-browser-ftp-hall-of-shame.html

A further minor addition is a host command, a subset of the host 
commands from ISC and from KnotDNS that uses the same DNS client library 
from djbdns as all of the other djbdns query tools do.  Of course, the 
conventional djbdns client tools have a simpler syntax and more regular 
behaviours than the host command, and are preferable.  Moreover, the 
subset excludes rarities that djbdns has never supported, such as non-IN 
class queries.

Re: s6-svscanboot, how to exit?

[Jonathan de Boyne Pollard]

> Type=forking

No.

> StandardOutput=tty

No.

> RemainAfterExit=yes

No.

> SysVStartPriority=99

No.

You're a long-time daemontools user, you say.  Why on Earth do you think that
Type=forking is right?  Why on Earth do you think that good daemons fork and
exit parent?  Why on Earth do you think that svscan -- any svscan, from
daemontools-encore svscan to s6-svscan -- forks and exits parent?  Or needs a
TTY?

* http://jdebp.eu./FGA/inittab-is-history.html

* http://jdebp.eu./FGA/unix-daemon-readiness-protocol-problems.html

* http://untroubled.org/daemontools-encore/svscan.8.html

* http://www.skarnet.org/software/s6/s6-svscan.html

* https://packages.debian.org/sid/runit-systemd

* https://packages.debian.org/sid/daemontools-run

* https://wiki.archlinux.org/index.php/Runit#Installation

* https://wiki.gentoo.org/wiki/Runit#Starting_the_supervision_tree

*
https://framagit.org/taca/archnosh#nosh-service-manager-only-nosh-run-via-systemd

Re: s6-svscanboot, how to exit?

[Jonathan de Boyne Pollard]

Jean Louis:
> Well I made rc-local.service

Do not do that.  Creating a two steps backwards compatibility mechanism, in
order to run s6-svscan under systemd, is daft.

Jean Louis:
> I just used recipe from stackexchange.com

You're not reading the right parts of Stack Exchange, obviously.

* https://askubuntu.com/a/700401/43344

* http://jdebp.eu./FGA/inittab-is-history.html

* https://unix.stackexchange.com/a/247543/5132

* https://unix.stackexchange.com/a/333003/5132

* https://news.ycombinator.com/item?id=14695970

Re: nosh version 1.34

[Jonathan de Boyne Pollard]

Guillermo:

> 
> So, system-control has a new but undocumented 'is-service-manager-client'
> subcommand?
> 

Yes.  It's part of an incomplete to-do list item; and might change.

Re: nosh git tree

[Jonathan de Boyne Pollard]

Suggestions are welcome.  You can put them on the Supervision mailing list if
you like.  Please bear in mind the design principles laid out in the Guide
http://jdebp.eu./Softwares/nosh/guide.html , though.

Bug reports and real world experiences are also welcome, and in fact more
interesting.  Several recent developments have come from bug reports.  The
recent changes for the per-user service bundle for gpg-agent came about from a
bug report from a person who has a more up-to-date version of gpg-agent than is
available on any of my test operating systems, for example.  Again, you can use
the Supervision mailing list if you like.

There is a Roadmap in the WWW blurb http://jdebp.eu./Softwares/nosh/  with some
of the items on my own to-do list on it.  The to-do list itself has somewhat
more on it.  Currently, for example, I am further improving the external
configuration import, so that /etc/system-control/convert/rc.conf aggregates
more stuff on non-BSD systems.

Arch users might want to contribute to Archnosh http://repo.or.cz/archnosh.git .
 Gentoo users might want to contribute to the Gentoo doco
https://wiki.gentoo.org/wiki/Nosh  and bring it up to the Archnosh level.
 Debian, Ubuntu, FreeBSD, and TrueOS users note that I supply full package
repositories http://jdebp.eu./Softwares/package-repositories.html  already.
 There is no git repository, but the curious can see old source archives via
GOPHER or FTP there.  The FTP listing is determined by the server's directory
order, but they are neatly segregated in the GOPHER menu.  It is of course the
same repository being served over HTTP, FTP, and GOPHER, this being publicfile.
 Yes, GOPHER
http://www.mail-archive.com/supervision@list.skarnet.org/msg01657.html .  (-:

nosh version 1.34

[Jonathan de Boyne Pollard]

The nosh package is now up to version 1.34 .

* http://jdebp.eu./Softwares/nosh/
*
https://www.freebsd.org/news/status/report-2015-07-2015-09.html#The-nosh-Project
* http://jdebp.info./Softwares/nosh/

Once again, there are a few more service bundles.  The most interesting ones in
this version are perhaps the finish-update and finish-install targets, designed
to be invoked the first bootstrap after an update or install has been done, and
the users target, which is used to auto-start per-user subsystems at bootstrap.
 Several NFS service bundles are now common across operating systems.  And the
OpenVPN service bundles are now split into separate client and server services.

Several minor bugs have been fixed here and there: a duplicated newline in
line-banner that was throwing off publicfile FTP service; a problem with
recordio on FreeBSD/TrueOS; and a problem with attempts to use slashes in
environment variables in service bundle environment directories.

The user-space virtual terminal emulator now implements the Xterm extensions to
DECSCUSR, and the framebuffer realizer can display the resultant cursor shapes.
 This can be made use of by programs such as Neovim.

There are now separate service bundles and nosh-run- packages for running eudev
and systemd-udev, because the two are now significantly divergent.

The various utilities for changing the process environment no longer use the GNU
C library/BSD C library functions for doing so, and so no longer suffer from the
concomitant memory leaks that their manual pages used to warn about.

The convert-systemd-units tool has been slightly enhanced, for the benefit of a
fix that has been made to the per-user gpg-agent service.

The external configuration imports system has been extended.  It now deals with
importing the hostname configuration value, taking that responsibility away from
and simplifying the set-dynamic-hostname utility.  It now imports various Debian
and other kernel virtual terminal settings, from /etc/kbd/config,
/etc/default/console-setup, and /etc/vconsole.conf .  And network configuration
import now can set up services for both dhcpcd and dhclient.
 /etc/system-control/convert/rc.conf now contains more settings on Linux
operating systems as a result, including dhclient_program.

Runit and perp doco improvement for chpst and runuid

[Jonathan de Boyne Pollard]

This week's little kerfuffle over systemd and the way that its mechanism to drop
superuser privileges works (see https://news.ycombinator.com/item?id=14681377
and the headlined bug report there) has led me to observe that whilst the doco
for nosh, s6, daemontools, and daemontools-encore all state that their various
setuidgid programs will abend on being given an unknown/invalid account, the
doco for chpst in runit and runuid in perp do not come out and explicitly say
the same.

runuid's doco implies it, albeit indirectly; and I have double-checked (not that
I was in much doubt) that chpst actually has this behaviour.  So it might be
worth improving the doco for chpst and runuid to actually say this outright.

Re: s6 as a systemd alternative

[Jonathan de Boyne Pollard]

Charles Duffy
> - Integration with the linux-only cgroups mechanism for managing CPU,
> memory, and I/O throughput limits

As you say, this is just an exercise in chain loading tools and conversion to
scripts that employ them.

* https://news.ycombinator.com/item?id=11846083

* http://jdebp.eu./Softwares/nosh/guide/move-to-control-group.html

Re: s6 as a systemd alternative

[Jonathan de Boyne Pollard]

Casper Ti. Vector:

> 
> (Normally Jonathan would be replying to this point, but I still do not see
> him in this thread, so I rashly take this job ;)
> 

Goodonyer.  Thank you.

I have been distracted.

* https://github.com/neovim/neovim/pull/6816
https://github.com/neovim/neovim/pull/6816

I have got back to 1.34 and added some more bits and pieces this week.  Looking
at the changelog, it is probably time to release that and begin work on 1.35.

Davin McCall's comparison of service and system management

[Jonathan de Boyne Pollard]

https://github.com/davmac314/dinit/blob/master/doc/COMPARISON#L71

It's a shame that this gets a number of things wrong about several of the
systems.

Re: small proxy

[Jonathan de Boyne Pollard]

Jorge Almeida:
> As said in the first mail, I need to redirect some targets to a ssh tunnel,
> and let everything else go its way. Rather than using a proxy, the solution
> seems obvious: [...]

... use a Proxy Auto-Configuration script, as that is the function of PAC
scripts.

* http://jdebp.eu./FGA/web-browser-auto-proxy-configuration.html

Re: A dumb question

[Jonathan de Boyne Pollard]

Francisco Gómez García:
> And the person who told me the "it is too old" thing used arguments 
> such as "it will eventually not compile" - while I doubt a small codebase 
> on plain C will stop compiling in the near future, it is indeed something 
> to keep an eye on.

That person needs to learn from history.  We have actually *already been
through* the "it does not compile" moment in the daemontools world.  It was the
well-known errno problem; "plain C" that was not, in fact, standard-conformant
C.  It was patched fairly rapidly, years ago.  Moreover there are plenty of
people who have, severally and independently, brought the Bernstein libraries up
to date with respect to C1999 and C2011, including Felix von Leitner, Laurent
Bercot, and ... well ... me.

* http://jdebp.eu./Softwares/djbwares/

Francisco Gómez García:
> Good to know there's s6 as well [...]

Even freedt is still available.

* http://jdebp.eu./FGA/daemontools-family.html

Re: A dumb question

[Jonathan de Boyne Pollard]

Laurent Bercot:
> Lots of work have been put into s6 in the last few years, more than in any
> other supervision suite, with the possible exception of nosh;

(-:

Laurent Bercot:
> the fact that Void Linux has not yet switched to s6 is proof that runit is
> still working well enough for them and they don't feel the need to change.

Did you ever hear anything as a result of
http://www.mail-archive.com/supervision@list.skarnet.org/msg01570.html ?

Re: A dumb question

[Jonathan de Boyne Pollard]

Laurent Bercot:
> It is also true that it would benefit from closer maintenance.  
> Gerrit Pape, runit's author, is still around and still reads this list, 
> but is not as active as he was a few years ago (typically during 
> runit development). Fortunately, there are not many feature requests.

The Debian maintainer is now Dmitry Bogatov.

* https://tracker.debian.org/pkg/runit

Re: A dumb question

[Jonathan de Boyne Pollard]

Laurent Bercot:
> > "It's old software. Its last version is from 2014."
> 
>  Old? Let's see...

A more amusant counterexample is perhaps the Debian Almquist shell, put into
widespread use on Debian and Ubuntu a decade ago in order to speed up bootstrap.
 M. Almquist first published it in May 1989.  Its Debian maintainer is Gerrit
Pape.

* https://wiki.ubuntu.com/DashAsBinSh

* https://wiki.ubuntu.com/DashAsBinSh/Spec

* https://lists.debian.org/debian-release/2007/07/msg00027.html

* https://wiki.debian.org/DashAsBinSh

* https://tracker.debian.org/pkg/dash