Re: Issues with lseek(2) on a block device

[Thor Lancelot Simon]

On Sun, Feb 25, 2024 at 04:16:03AM -, Michael van Elst wrote:
> t...@panix.com (Thor Lancelot Simon) writes:
> 
> >Probably not a good idea to start with lseek() because if you _do_
> >encounter a tape device, seeking to SEEK_END could take you an extremely
> >long time.
> 
> lseek() doesn't move the tape. But the open() or close() may trigger
> tape operations like loading or winding. Special files are special.

Ah, yes, that's the MTEOM ioctl -- approximately.

I have it stuck in my mind that SEEK_END did this, too, back when I
was a literal tape jockey for 9 track tapes on an 11/750 and a
collection of TK and QIC-150 drives.  But this could be an invention
of memory.

-- 
  Thor Lancelot Simont...@panix.com

  "The inconsistency is startling, though admittedly, if consistency is to
   be abandoned or transcended, there is no problem." - Noam Chomsky

Re: Issues with lseek(2) on a block device

[Thor Lancelot Simon]

On Sun, Feb 25, 2024 at 04:41:37PM +, Sad Clouds wrote:
> 
> The whole idea of Unix was "everything is a file" and the sooner people
> get rid of magic ioctls the better.

But everything is *not*, actually, a file, and forcing things that
are not files into file-like shapes is not always the right design
choice.

As simple as possible, but no simpler.

-- 
  Thor Lancelot Simont...@panix.com

  "The inconsistency is startling, though admittedly, if consistency is to
   be abandoned or transcended, there is no problem." - Noam Chomsky

Re: Issues with lseek(2) on a block device

[Thor Lancelot Simon]

On Thu, Feb 22, 2024 at 12:08:14PM +, Sad Clouds wrote:
> Hello, thanks to everyone who responded with their suggestions. Using
> various non-portable ioctls I can device size on most platforms, for
> both block and raw devices.
> 
> This is more convoluted than a single lseek() call, but it is what it
> is. If anyone wants to do something similar, then the following example
> can give you a good start.

Probably not a good idea to start with lseek() because if you _do_
encounter a tape device, seeking to SEEK_END could take you an extremely
long time.

Thor

Re: Issues with lseek(2) on a block device

[Thor Lancelot Simon]

On Wed, Feb 21, 2024 at 09:20:55PM +, Taylor R Campbell wrote:
> > Date: Wed, 21 Feb 2024 10:52:55 +
> > From: Sad Clouds 
> > 
> > Hello, for most operating systems determining the size of a block
> > device can be done with:
> > 
> > lseek(fd, 0, SEEK_END);
> > 
> > However, on NetBSD this does not seem to work.
> 
> Internally, this is happens for more or less the same reason that
> stat(2) doesn't return the size of a block device in st_size.
> 
> Each file system on which device nodes can reside implements its own
> VOP_GETATTR (used by both lseek(2) and stat(2)), and most of them use
> the _inode_ size (more or less) rather than querying the block device
> that the device node represents for its physical size.
> 
> I think this is a bug, and it would be great if stat(2) just returned
> the physical medium's size in st_size -- currently doing this reliably
> takes at least three different ioctls to handle all storage media, if
> I counted correctly in sbin/fsck/partutil.c's getdiskinfo.

I am not sure this can be done for all block devices.  Tapes have block
devices, and open-reel tape drives do not really know the length of the
loaded media, while on any other tape drive supporting compression, there
may really be no such size.

Re: Issues with lseek(2) on a block device

[Thor Lancelot Simon]

On Thu, Feb 22, 2024 at 05:47:55AM +, Sad Clouds wrote:
> On Wed, 21 Feb 2024 12:48:34 -0800
> Jason Thorpe  wrote:
> 
> > 
> > > On Feb 21, 2024, at 2:52???AM, Sad Clouds  
> > > wrote:
> > > 
> > > Hello, for most operating systems determining the size of a block
> > > device can be done with:
> > > 
> > > lseek(fd, 0, SEEK_END);
> > 
> > On what operating systems does this do what you claim?
> > 
> > > However, on NetBSD this does not seem to work.
> > 
> > It doesn???t work on macOS, either:
> > 
> > thorpej-mbp:thorpej 541$ sudo ./lseek /dev/disk4
> > Password:
> > lseek(fd, 4096, SEEK_SET) = 4096 bytes
> > lseek(fd, 0, SEEK_END)= 0 bytes
> > thorpej-mbp:thorpej 542$ 
> > 
> > -- thorpej
> > 
> 
> Hello, it works on Linux, FreeBSD, Solaris and it also works on NetBSD

I don't think that's correct for all devices on Linux - generally, on
Linux systems I have to use the BLKGETSIZE64 ioctl to reliable determine
the size of a block device.

Re: [PATCH] HTTPS/TLS CA certificates in base

[Thor Lancelot Simon]

/usr/share/certs/mozilla/server for all of those trusted by Mozilla
>   for server authentication (i.e., TLS, including HTTPS).
> 
>   Certificates marked DISTRUST_AFTER any date are excluded; there's no
>   standard mechanism to pass this information through to OpenSSL as
>   far as I know.
> 
>   (As a potential convenience, the email and code-signing ones are
>   also symlinked at /usr/share/certs/mozilla/email and
>   /usr/share/certs/mozilla/code, but nothing uses those directories at
>   the moment.  Certainly /usr/share/certs/mozilla/code is _not_ used
>   for code-signing any kind of NetBSD updates.)
> 
> - OpenSSL-based applications use the directory /etc/openssl/certs or
>   the file /etc/openssl/certs/ca-certificates.crt to find TLS CA
>   certificates, as they do already -- no change to the
>   application-visible interface.
> 
> - New command certctl(8) manages /etc/openssl/certs and
>   /etc/openssl/certs/ca-certificates.crt as a cache of hashed
>   symlinks.
> 
>   . Simple configuration file /etc/openssl/certs.conf specifies search
> path (default: /usr/share/certs/mozilla/server), or `manual' to
> prevent certctl(8) from touching /etc/openssl/certs at all so you
> can manage it another way (e.g., security/mozilla-rootcerts or
> security/ca-certificates in pkgsrc).
> 
>   . `certctl untrust /path/to/foo.pem' excludes that certificate from
> /etc/openssl/certs, by symlinking it in /etc/openssl/untrusted.
> 
>   . If /etc/openssl/certs is damaged or lost, `certctl rehash'
> rebuilds it from the configuration file and the list of
> exclusions.
> 
>   The command-line syntax is mostly compatible with FreeBSD's
>   certctl(8), but FreeBSD's certctl(8) treats /etc/ssl/certs and
>   /etc/ssl/untrusted as _both_ configuration _and_ a cache, whereas
>   this proposal treats /etc/openssl/certs strictly as a cache for the
>   configuration in /etc/openssl/certs.conf and /etc/openssl/untrusted.
> 
>   I believe this proposal is more useful than the FreeBSD semantics:
>   it means, e.g., you can run `certctl rehash' (e.g., automatically
>   via postinstall(8)) after an upgrade and it won't forget your
>   `certctl untrust' decisions.
> 
> - When run in DESTDIR=/, postinstall(8) runs `certctl rehash' to make
>   sure /etc/openssl/certs is populated.
> 
>   Note: This is limited to DESTDIR=/, and will not happen in
>   cross-built destdirs, because it requires running openssl(1), and we
>   don't have openssl(1) hooked up to the tools build, nor do we
>   currently require the build environment to support it.
> 
>   If we changed either of those we could lift the restriction of
>   DESTDIR=/ for postinstall(8) to run `certctl rehash'.  (An
>   alternative would be to test whether `type openssl' succeeds, but
>   that strikes me as a little sketchier.)



-- 
  Thor Lancelot Simont...@panix.com

  "The inconsistency is startling, though admittedly, if consistency is to
   be abandoned or transcended, there is no problem." - Noam Chomsky

Re: alloca again

[Thor Lancelot Simon]

On Tue, Oct 04, 2022 at 02:22:23AM +, David Holland wrote:
> On Tue, Oct 04, 2022 at 12:41:07AM +0300, Valery Ushakov wrote:
>  > On Mon, Oct 03, 2022 at 18:32:47 +, David Holland wrote:
>  > 
>  > > On Mon, Oct 03, 2022 at 03:45:06PM +0200, Roland Illig wrote:
>  > >  > > This is the current alloca definition in the libc headers:
>  > >  > > 
>  > >  > > #if defined(_NETBSD_SOURCE)
>  > >  > > #if defined(alloca) && (alloca == __builtin_alloca) && \
>  > > 
>  > > Note that the part you didn't quote is
>  > >defined(__GNUC__) && __GNUC__ < 2
>  > > and all it does is change the argument type to int from size_t.
>  > > 
>  > > This makes it highly unlikely that it ever failed, for an assortment
>  > > of reasons, not the least of which is that we removed critical parts
>  > > of gcc2 support from /usr/include years ago so it's not been exercised
>  > > in at least that long.
>  > 
>  > ITYM gcc1?  Note "less than" in  __GNUC__ < 2
> 
> Oops! Yeah.
> 
> Was gcc1 still in use even in 1993? I can't remember.

Yes, because gcc2 was unstable for many architectures, and had performance
regressions for others.

-- 
 Thor Lancelot Simon t...@panix.com
"Somehow it works like a joke, but it makes no sense."
--Gilbert Gottfried

Re: Set USB device ownership based on vedorid/productid

[Thor Lancelot Simon]

On Mon, Feb 15, 2021 at 12:15:28PM -0500, Mouse wrote:
> > Does NetBSD provide any framework that allows USB device
> > ownership/permissions to be autmatically set on USB
> > VendorId/DeviceId?
> 
> As far as I know it doesn't; a quick look at 8.0's manpages didn't show
> me anything.  (9.1 is not easy for me to check right now; today I'm on
> the wrong job for that.)

I don't think so, because these IDs aren't exposed to the right layer
of the autoconf machinery.  They're used internally in the driver match
routines.  Hoisting them up to the config file seems like a big task and
in some ways a regression.

But I wonder if, for the "generic" drivers like ugen, they could be
added as optional locators, passed down into ugen's match/attach so
a specific ugen instance could be pinned down to a specific vendor and
device ID.  Then you'd get, let's say, ugen9 whenever you had vid 0x5150
did 0x1337, and you could adjust the permissions on /dev/ugen9 accordingly
in /dev and away you'd go.

I can see how this would work for static config but I am not really
up to date enough on the modern world of NetBSD kernel modules and boot
time configuration to understand how or even whether it could work well
there.

Thor

Re: Waiting for Randot (or: nia and maya were right and I was wrong)

[Thor Lancelot Simon]

On Fri, Jan 15, 2021 at 12:21:24AM +, Taylor R Campbell wrote:
> 
> What about them?  Systems without usable microphone noise are no worse
> off than they would have been without nia's suggestion.

There are actually a couple of things that can be done here.

1) It's likely better to max out the gain on a _disconnected_ audio input
   so you're getting mostly amplifier noise than to use real ambient audio.

2) Some audio devices have their own clocks not derived from the system clock.
   This clock can be sampled against the system clock to construct a
   clock-skew entropy source.

I think either of these is best done in the hardware device driver itself;
but honestly, it's hard to knock the simplicity of the quick and dirty
approaches that do it all in userspace.

Thor

Re: Summary of man-page formatting

[Thor Lancelot Simon]

On Sat, Nov 21, 2020 at 07:50:02AM +, David Holland wrote:
> On Sun, Nov 15, 2020 at 03:10:08AM +0100, Kamil Rytarowski wrote:
>  > I'm going to summarize the situation with formatters in the NetBSD base.
> 
> A couple more points:
> 
>  > [...] old groff in base.
> 
> There's an additional reason to not update groff besides licensing,
> which is that recent groff depends on perl.

Heck, at least it doesn't depend on adb.

Thor

Re: Proposal to drop MKCATPAGES

[Thor Lancelot Simon]

On Sun, Oct 25, 2020 at 03:45:56PM -0400, Mouse wrote:
> 
> I once had an hp300 with all of 5M of RAM.  Years ago, when I had it
> running, thorpej told me it was quite possibly an instance of the
> slowest machine then supported by NetBSD.  (Amusingly, at the same time
> I had an alpha that he said was possibly an instance of the fastest.)

I think -- with the possible exception of the 725 or 730 if anyone ever
made them work, they were very hard to get going even with 4.3! -- the
Sun 2/50 is likely the slowest machine that ever ran NetBSD.  I don't think
we support any 68010-based HP300 models, though I am not entirely sure.

Was the Australian machine kre mentioned one of those bizarre dual-68000
designs that ran two CPUs in lockstep to handle non-restartable
instructions?  I've heard about them but never seen one.

Thor

Re: Proposal to drop MKCATPAGES

[Thor Lancelot Simon]

On Sun, Oct 25, 2020 at 02:13:47AM +0200, Kamil Rytarowski wrote:
> 
> I recall catpages to used in 80286 UNIX (Coherent) and the catpages are
> probably just applicable for such constrained environments that cannot
> host any text formatters.

The issue was the speed of the text formatters.  I vividly recall on
the 11/750 waiting multiple seconds for nroff to chew through man
pages before I could read them.

Of course groff's even slower, but mandoc is faster -- than groff, at
least, dunno about heritage nroff.  Is there a noticeable delay with
mandoc even on our slowest supported hardware?  It might very well be
fine.

Thor

Initial entropy with no HWRNG

[Thor Lancelot Simon]

On Tue, May 12, 2020 at 10:00:20AM +0300, Andreas Gustafsson wrote:
>
> Adding more sources could mean
> reintroducing some timing based sources after careful analysis, but
> also things like having the installer install an initial random seed
> on the target machine (and if the installer itself lacks entropy,
> asking the poor user to pound on the keyboard until it does).

As Peter Gutmann has noted several times in the past, for most use cases
you don't have to have input you know are tied to physical random processes;
you just have to have input you know it's uneconomical for your adversary
to predict or recover.

This is why I fed the first N packets off the network into the RNG; why I
added sampling of kernel printf output (and more importantly, its *timing);
etc.  But the problem with this kind of stuff is that there really are use
cases where an adversary _can_ know these things, so it is very hard to
support an argument that _in the general case_ they should be used to
satisfy some criterion that N unpredictable, unrecoverable (note I do
*not* say "random" here!) bits have been fed into the machinery.  The data
I fed in from the VM system are not quite the same, but in a somewhat similar
situation.

That said, I also added a number of sources which we *do* know are tied to
real random physical processes: the environmental sensors such as temperature,
fan speed, and voltage, where beyond the sampling noise you've got thermal
processes on both micro and macro scales, turbulence, etc; and the "skew"
source type which, in theory, represents skew between multiple oscillators
in the system, one of the hybrid analog-digital RNG designs with a long
pedigree (though as implemented in the example "callout" source, less-so).

Finally, there's a source type I *didn't* take advantage of because I was
advised doing so would cause substantial power consumption: amplifier noise
available by setting muted audio inputs to max gain (we can also use the
sample arrival rate here as a skew source).

I believe we can and should legitimately record entropy when we add input
of these kinds.  But there are three problems with all this.

*Problems are marked out with numbers, thoughts towards solutions or
 mitigations with letters.*

1) It's hard to understand how many bits of entropy to assign to a sample from
   one of these sources.  How much of the change in fan speed is caused by
   system load as a factor (and thus highly correlated with CPU temperature),
   and how much by turbulence, which we believe is random?  How much of the
   signal measured from amplifier noise on a muted input is caused by the
   bus clock (and clocks derived from it, etc.) and how much is genuine
   thermal noise from the amplifier?  And so forth.

   The delta estimator _was_ good for these things, particularly for things
   like fans or thermistors (where the macroscopic, non-random physical
   processes _are_ expected to have continuous behavior), because it could
   tell you when to very conservatively add 1 bit.  If you believe that at
   least 1 bit of each 32-bit value from the input really is attributable to
   entropy.  I also prototyped an lzf-based entropy estimator, but it never
   really seemed worth the trouble -- it is, though, consistent with how
   the published analysis of physical sources often estimates minimum
   entropy.

   A) This is a longwinded way of saying I firmly believe we should count
  input from these kinds of sources towards our "full entropy" threshold
  but need to think harder about how.

2) Sources of the kind I'm talking about here seldom contribute _much_
   entropy - with the old estimator, perhaps 1 bit per change - so if you
   need to get 256 bits from them you may be waiting quite some time (the
   audio-amp sources might be different, which is another reason despite
   their issues, they are appealing).
  
3) Older or smaller systems don't have any of this stuff onboard so it does
   them no good: no fan speed sensors (or no drivers for them), no temp
   sensors, no access to power rail voltages, certainly no audio, etc.

B) One thing we *could* do to help out such systems would be to actually run
   a service to bootstrap them with entropy ourselves, from the installer,
   across the network.  Should a user trust such a service?  I will argue
   "yes".  Why?

B1) Because they already got the binaries or the sources from us; we could
simply tamper those to do the wrong thing instead.

Counterargument: it's impossible to distinguish the output of a
cryptographically-strong stream cipher keyed with something known
to us from real random data, so it's harder to _tell_ if we subverted
you.

Counter-counter-argument: When's the last time you looked?  Users
who _do_ really inspect the sources and binaries they get from us
can always not use our entropy server, or run their own.

B2) Because we have already arranged to mix in a whole 

Re: getrandom and getentropy

[Thor Lancelot Simon]

On Mon, May 11, 2020 at 04:16:12PM -, Michael van Elst wrote:
> n...@netbsd.org (nia) writes:
> 
> >is insisting). All of that depends on assumptions and trust - it
> >does no measurement of the value of the entropy being provided.
> 
> Previously we could trust in random processes, whether the entropy
> estimation was scientific or not. We could also chose what source
> to trust.
> 
> Now we put all trust in loading a constant file.

That is not the case, because the data from the supposedly "random
processes" is still mixed into the pool.  We just don't make unsupportable
claims about how much entropy it contributed.

Re: getrandom and getentropy

[Thor Lancelot Simon]

On Mon, May 11, 2020 at 04:28:51PM +0300, Andreas Gustafsson wrote:
> 
> For the OpenBSD strategy to work, the system needs to actually refuse
> to run if the seed can't be loaded (or full entropy achieved in some
> other way).  NetBSD doesn't do that.  As long as there is any way

Well, no.  The system can also handwave the question of whether the
"seed" actually contains any entropy at all.  We did not go that way.

Thor

Re: getrandom and getentropy

[Thor Lancelot Simon]

On Mon, May 11, 2020 at 09:53:31AM +0300, Andreas Gustafsson wrote:
> 
> OpenBSD guarantees that there is an entropy seed from the boot loader,
> which is very different from NetBSD's "best effort".  Was this not
> already the case when the getentropy API was introduced?

I think you need quotes around "entropy" on that seed, there...

Thor

Re: getrandom and getentropy

[Thor Lancelot Simon]

On Sun, May 10, 2020 at 08:22:42PM +0300, Andreas Gustafsson wrote:
> Joerg Sonnenberger wrote:
> > We don't warn people about unavailable of 127.0.0.1/:: during very early
> > boot and a number of other issues either. If your application is running
> > during system initialisation, you are supposed to be somewhat aware of
> > the limitations in that case.
> 
> That's nonsense.  If the man page promisies "high quality entropy",
> that needs to apply no matter when the application is run.

If the man page promises "gravity will pull everything up" that needs to
apply no matter when the application is run.

The notion that /dev/random unblocking because a "delta estimator" that
attributes entropy to pretty much every sample submitted to it said it
saw some entropy is actual evidence that an application was delivered
"high quality entropy" is, as far as I'm concerned, roughly like the
notion that a ball flying upwards out of my hand is evidence that
gravity reversed directions.

-- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: PATCH libatomic

[Thor Lancelot Simon]

On Fri, May 08, 2020 at 10:24:43PM +, m...@netbsd.org wrote:
> 
> The indirection only applies to the first call. The magic is within
> rtld.

If it's not binary patch, no magic can avoid at least one level of
indirection.

-- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: PATCH libatomic

[Thor Lancelot Simon]

On Fri, May 08, 2020 at 10:03:06AM +0200, Kamil Rytarowski wrote:
> On 08.05.2020 02:14, Thor Lancelot Simon wrote:
> > 
> > Not without performance penalty for every atomic operation, unless you 
> > propose
> > to do this by binary patch as is done in the kernel.
> 
> There is atomic penalty, but it is the contract and design of this (C
> and C++) feature. Atomics can be legitimately lock-free or non-lock-free
> and this is a feature.

Adding an extra test or, worse, an indirect branch before every atomic
operation makes it far worse than it has to be.  An uncontended locked
transaction on the bus may cost nothing.  An indirect branch followed
by the same transaction consumes all sorts of microarchitectural resources
will not just be slower but also impact the performance of _other_ code
too.  That's why the kernel binary-patches out lock prefixes instead of
using indirection for atomics.

Thor

Re: PATCH libatomic

[Thor Lancelot Simon]

On Fri, May 08, 2020 at 01:51:16AM +0200, Kamil Rytarowski wrote:
> A runtime detection could be a part of ifunc (is it ready for NetBSD?).
> 
> The standard C/C++ feature is to detect whether atomic operations are
> real (lock-free) through atomic_is_lock_free(). This is a feature, not a
> bug (as claimed by some people). atomic_is_lock_free() can be overloaded
> in libatomic and detect CPU type in runtime and redirect either to real
> CPU intrinsic of lock-free fallback.

Not without performance penalty for every atomic operation, unless you propose
to do this by binary patch as is done in the kernel.

Thor

Re: getrandom and getentropy

[Thor Lancelot Simon]

On Sun, May 03, 2020 at 10:48:41AM +0200, Kurt Roeckx wrote:
> 
> You might want to read https://lwn.net/Articles/808575/

To be blunt (I think it's past time for it):  I trust Taylor reading the
code more than I trust you reading stuff you found with Google.

Thor

Re: getrandom and getentropy

[Thor Lancelot Simon]

On Sat, May 02, 2020 at 06:07:54PM +0200, Kurt Roeckx wrote:
> 
> It's my understanding that it never blocks because the bootloader
> provides entropy. Be time time the first user can call genentropy,
> it has already been seeded.

Horsepuckey.  You can't know the bootloader always has entropy to
provide.

-- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: Solving the syslogd problem

[Thor Lancelot Simon]

On Tue, Feb 04, 2020 at 08:36:56AM -0500, Thor Lancelot Simon wrote:
> 
> Moving part of the system to /usr was a *necessary evil* when it was done.
> There is no real rhyme nor reason to what's in /bin vs /usr/bin, even less
> to /sbin vs /usr/sbin, except "huh, I need _this_ and I'm willing to make
> / a little bigger to hold it".  But why shouldn't / just be big enough to
> hold all of /usr?  Because of header files and static libraries and other
> toolchain components?  I'm going to submit that if you have a machine

As several people have reminded me privately: optional packages.  But there is
no real reason we couldn't keep /usr/pkg (or, simply, /pkg; or, more
conventionally, /opt) as its own filesystem while undoing the historical
mistake of splitting /bin and /lib into two pieces and scattering them across
the namespace.

Thor

Re: Solving the syslogd problem

[Thor Lancelot Simon]

On Tue, Feb 04, 2020 at 07:38:34AM +1030, Brett Lymn wrote:
>
> and mounted.  So, so what if you get / first and then have to wait for
> the rest of the fsck's to happen vs a fsck of a single large file
> system?  At the end of the day it will take about the same amount f time
> to get the machine to a usable state.

That.  To me the argument that we should have /usr split from / "because
it takes too long to fsck huge filesystems" makes sense only if one's
put totally inappropriate stuff in /usr.

I can see entirely reasonable arguments for splitting out /home and /var
and /tmp in the default partitioning.  But spraying the system's executables
and libraries out across two filesystems so half of them are in / and half
of them are in /usr "to make fsck faster"?  To me it just stinks of "that's
how it was in my Golden Youth and I want it that way FOREVER!".

Moving part of the system to /usr was a *necessary evil* when it was done.
There is no real rhyme nor reason to what's in /bin vs /usr/bin, even less
to /sbin vs /usr/sbin, except "huh, I need _this_ and I'm willing to make
/ a little bigger to hold it".  But why shouldn't / just be big enough to
hold all of /usr?  Because of header files and static libraries and other
toolchain components?  I'm going to submit that if you have a machine
suitable for development, then you have a machine where the time to fsck
a few directories of libraries and header files *in the rare instances
when you're booting single-user with a r/w /* will not in fact be large.

Remember, we already split out the "heavy hitters" (/home, /var) to
paths (and thus filesystems, if you like)  of their own.  I really can't see
why we shouldn't fix the mess with every directory in / being mirrored
over into /usr, which was acknowledged as an unfortunate compromise when
it was made.

Re: Solving the syslogd problem

[Thor Lancelot Simon]

On Wed, Jan 29, 2020 at 11:33:22AM +0100, Joerg Sonnenberger wrote:
> On Tue, Jan 28, 2020 at 09:21:23PM +, Roy Marples wrote:
> > To fix this, I suggest that we split syslogd into syslogd and 
> > syslogd-network.
> 
> We could also do a much simpler and more radical decision and stop
> splitting / and /usr. Of all the partitioning choices available, it
> truely seems to be a pointless legacy from extremely constrained
> hardware with a significant cost to maintain.

This is elegant and I would like to see it.  Just remove /usr entirely and
collapse its contents into / - no /usr/bin, no /usr/lib, etc.

Thor

Re: dhcpcd privilege separation user

[Thor Lancelot Simon]

On Thu, Jan 23, 2020 at 09:36:29PM +, Roy Marples wrote:
> Hi List!
> 
> dhcpcd-9 is almost ready for release and the big feature for it is privilege
> separation. The question is which system user should it use on NetBSD? Other
> BSD's have the _dhcp user - which shares DHCP serving on OpenBSD at least.
> 
> Is this to generic and we should have a _dhcpcd user or is _dhcp satisfactory?

I don't think it's a great idea to share the user between the purposes, even
though few systems will simultaneously be DHCP server and client.

-- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: stdio.h's P_tmpdir definition - make it /tmp?

[Thor Lancelot Simon]

On Fri, Aug 30, 2019 at 02:40:06PM +1000, matthew green wrote:
> hi folks.
> 
> 
> a very long time ago in netbsd years tls@ patched GCC to use
> /tmp over /var/tmp, for the compiler temporaries.
> 
> i tried to keep these patches, but they got lost at some
> point and when i recently wanted to deal with it (i had one
> machine with some several hundred /var/tmp/cc*.o files that
> were mostly over a year old), i discovered it's not nearly
> as simple as a quick patch.
> 
> most of the toolchain consumers that create temp files don't
> use $TMPDIR from the env but use stdio.h's P_tmpdir defintion


Well, this is... well... I'm embarassed I missed it.  I think
you should change it now.

Only thing to watch out for I suppose is that vi's expectation
of where vi.recover goes doesn't change -- though that should
never have even been in /var/tmp in the first place.

Thor

Re: Add curses_version() in curses(3)

[Thor Lancelot Simon]

On Wed, Aug 28, 2019 at 10:42:13PM -0400, Thor Lancelot Simon wrote:
> On Wed, Aug 28, 2019 at 01:54:54AM +0100, Roy Marples wrote:
> > > > > 
> > > > > Where 8.2 is taken from the .so version?
> > > > > 
> > > > > Roy
> > > 
> > > I think that it would be confused with NetBSD release and it could be
> > > meaningless/confusing for downstream users that just pick the code as is
> > > and can pick different major/minor.
> > 
> > Then I'm against this being added because it adds nothing of value.
> > 
> > I note that PD curses also supports curses_version() with something similar
> > to my proposal.
> 
> The problem with using the solib version is that it may differ on
> platforms derived from NetBSD even if the curses API is exactly the same.
> 
> This would, for example, have been the case on EQ/OS which I maintained
> at Coyote Point - we bumped all the shlib versions at least once because
> of a change to libc.

Sorry, hit send a little early and left out the ugliest detail.

The problem then really is that if, let's say, my EQ/OS libcurses had
reported 9.2 for a libcurses that was called 8.2 in NetBSD, when NetBSD
bumps its libcurses to 8.2 for real API-change reasons within libcurses,
my version will look like it has the API change but actually it won't.

You really don't want to use the solib version for this sort of thing
because it will not mean the same thing on different platforms for the
exact same version of the code implementing the API, because it must be
bumped every time the library's *ABI* changes -- and that must be done
every time some library upon which _it_ depends changes, in turn.

A version like __NetBSD_Version should be used.

- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: Add curses_version() in curses(3)

[Thor Lancelot Simon]

On Wed, Aug 28, 2019 at 01:54:54AM +0100, Roy Marples wrote:
> > > > 
> > > > Where 8.2 is taken from the .so version?
> > > > 
> > > > Roy
> > 
> > I think that it would be confused with NetBSD release and it could be
> > meaningless/confusing for downstream users that just pick the code as is
> > and can pick different major/minor.
> 
> Then I'm against this being added because it adds nothing of value.
> 
> I note that PD curses also supports curses_version() with something similar
> to my proposal.

The problem with using the solib version is that it may differ on
platforms derived from NetBSD even if the curses API is exactly the same.

This would, for example, have been the case on EQ/OS which I maintained
at Coyote Point - we bumped all the shlib versions at least once because
of a change to libc.

-- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: pthread_setname_np API is bad

[Thor Lancelot Simon]

On Fri, Aug 02, 2019 at 09:29:27PM -0700, Matt Turner wrote:
> 
> So great, you made your API incompatible with everyone else for zero
> gain. /o\

And so forth.

Is there some reason for the snotty attitude?  Whether you're right or
wrong you're not likely to persuade anyone that way.

Thor

Re: malloc() exceeds RLIMIT_DATA

[Thor Lancelot Simon]

On Tue, Jul 09, 2019 at 10:24:53AM -0700, Graham Percival wrote:
> On Tue, Jul 09, 2019 at 07:17:39AM +0200, Martin Husemann wrote:
> > The classical "data segment" (limited by RLIMIT_DATA) is not used much
> > nowadays in NetBSD. Especially malloc() does not use it.
> > 
> >  RLIMIT_DATA The maximum size (in bytes) of the data segment for a
> >  process; this defines how far a program may extend its
> >  break with the sbrk(2) system call.
> > 
> > 
> > In ancient times malloc allocated memory via sbrk(2), but nowadays it uses
> > mmap(2) with anonymous memory.
> 
> I have no complaint about the internal implementation of malloc().  But POSIX
> says that:
> 
> RLIMIT_DATA
> This is the maximum size of a data segment of the process, in bytes. If

I think you don't grasp what Martin's telling you.

POSIX does not precisely define "data segment of the process"; nor does it
state that there cannot be more than one.

Our implementation doesn't really have such; so, though it might be desirable
to ensure a process cannot allocate more than RLIMIT_DATA bytes, it is not
a violation that it cannot, so long as said allocations don't come from a
single "data segment of the process" (which in our implementation they do
not).

The concept of a process having a single "data segment" managed with sbrk()
is very old and is not really relevant to modern Unix implementations.  If
POSIX wants to redefine RLIMIT_DATA in some way more useful in the modern
era, this might be good; but as it stands, the problem is that the limit
is enforced, but it does not, perhaps, limit what many users initially
think it ought to.

Thor

Re: Patch to make reentrant by default

[Thor Lancelot Simon]

On Fri, Apr 26, 2019 at 11:36:00PM +0200, Kamil Rytarowski wrote:
> 
> I propose a patch that:
> 
> 1. Removes alternative versions of  depending on preprocessor
> namespace definitions. All code since now is assumed to be
> reentrant/threading safe.

Benchmark results?  Pessimizing the single-threaded case is kind of a big
deal, and I'm shocked to see it proposed in a very long message that
seems to include _no_ measurements of performance at all.

Thor

Re: More compatibility for refuse

[Thor Lancelot Simon]

On Fri, Apr 12, 2019 at 03:46:15AM +0200, Emmanuel Dreyfus wrote:
> Thor Lancelot Simon  wrote:
> 
> > Gluster on Linux hitting 9Gb/sec for sequential I/O
> 
> Well at that performance level, avoiding copies is certainly a must, but
> I cannot test that because I lack the hardware (10G ethernet for that
> slide?)

10G ethernet is pretty common these days.  But I think the important thing
is that we know causing multiple copies back and forth through a userspace
daemon is an anti-pattern; we should not do more of it when we could do less.

Thor

Re: More compatibility for refuse

[Thor Lancelot Simon]

On Fri, Apr 12, 2019 at 01:55:39AM +0200, Emmanuel Dreyfus wrote:
> 
> On the performance front, it is true that the perfuse stack will cause
> data to be copied back and forth. When I started it, I thought that I
> would quickly have to add shared memory tricks to avoid copying data,
> but it has not been an obvious requirement: both GlusterFS and LTFS have
> decent performances, probably limited by other factors: network latency
> for GlusterFS and LTO drive throughput for LTFS.

I don't know what "decent" is, but, the issue isn't latency; it's
throughput -- for example, 
https://image.slidesharecdn.com/glusterperformancegdsoct2016-161007213244/95/state-of-gluster-performance-8-638.jpg?cb=1475875989
shows Gluster on Linux hitting 9Gb/sec for sequential I/O.  If we're not in
that neighborhood, I don't think adding additional memory copies is a great
idea.

The other thing to keep in mind is that the system's only got so much memory
bandwidth.  Whatever's consumed by needless copies in the filesystem isn't
available for the rest of the machine's workload.  Admittedly, 9Gb/sec is
only about 10% of the memory bandwidth available on a midrange x86 server
CPU these days; but still, that's not insignificant.

Thor

Re: Detecting keypress on console

[Thor Lancelot Simon]

On Sat, Mar 23, 2019 at 08:28:38AM +0100, Emmanuel Dreyfus wrote:
> Thor Lancelot Simon  wrote:
> 
> > Hold your nose and TIOCSTI.
> 
> That let's me inject data as if it was typed on the terminal right? You
> suggest I steal input from console and reinject it using TIOCSTI
> afterwards?

It's horrible, but, it should work.

This describes every use of TIOCSTI ever.

Re: Detecting keypress on console

[Thor Lancelot Simon]

On Fri, Mar 22, 2019 at 04:11:31PM +, Emmanuel Dreyfus wrote:
> Hello
> 
> I would like to execute a scipt when a function key is pressed on the
> console. Is there a mechanism to detect keypresses on wscons?  If I 
> make a daemon that opens /dev/console it will steal the input from
> getty/login/sh, which is not what I intend.
> 
> Any idea?

Hold your nose and TIOCSTI.

Re: [PATCH] dump -U to specify dumpdates device

[Thor Lancelot Simon]

On Wed, Mar 20, 2019 at 03:53:41PM +, Emmanuel Dreyfus wrote:
> On Wed, Mar 20, 2019 at 03:58:01PM +0100, Edgar Fuß wrote:
> 
> > Any prior art in one of the other BSDs or in systemd?
> 
> According to FreeBSD and OpenBSD man pages, no prior art, but the
> flag was not used for soemthing else there!

IIRC Irix had this, both for EFS and XFS.

-- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: colorls in base

[Thor Lancelot Simon]

On Fri, Feb 15, 2019 at 09:47:37PM +0100, Kamil Rytarowski wrote:
> 
> Colors nowadays are industry standard and increase readability.

Your unqualified statement is objectively false.

How do you know that the user's terminal isn't using the same
color -- or all too close to it -- for background as you've chosen
to use as syntax highlighting for directories?  Oops!  I guess I
didn't need to read any directory names at all.

I am *not* opposed to highlighting when ls (or vi, or...) is outputting
to a terminal.  I *am* opposed to the use of hardwired colors as
"highlights".  That practice is discriminatory against the color-blind
and also punishes those who (horrors) have chosen some background color
for their terminals from personal preference.

Among the worst offenders in this regard is vim, where on a properly
color-calibrated monitor with the terminal emulator set to a true
black background, dark blue is used for syntax highlighting, a blue
so dark it's basically unreadable.

Use abstract properties for highlighting syntax all you like -- bold,
reverse video, italic, underline, heck, make the text blink if you
have to.  Modern terminals let users map those to colors if they
choose anyway.

But please, don't pick some colors that look nice on *your* terminal
and inflict them on everyone else.

Thanks.

Thor

Re: Pressure sensors

[Thor Lancelot Simon]

On Mon, Dec 31, 2018 at 06:35:20PM +0200, Artturi Alm wrote:
> Hi,
> 
> i'd like to have the functionality given by the diff below, before
> cleaning up another diff making use of it.

LGTM - please make sure these are used as entropy sources, IIRC I
hardcoded the sensor types when I originally did the envsys entropy
work.  It looks to me like pretty much all the continuous sensors
(illuminance, pressure, etc.) could be used.

Thanks!

Thor

Re: Moving telnet/telnetd from base to pkgsrc

[Thor Lancelot Simon]

On Sat, Dec 15, 2018 at 10:08:00PM +, Taylor R Campbell wrote:
> 
> Given that a large fraction of respondents (though not all) indicated
> that their primary use of telnet is to test reachability of a server
> or manually enter SMTP or HTTP requests over the internet -- a use
> which is adequately served by the much smaller and much more
> confidence-inspiring usr.bin/nc -- I think this _does_ constitute a
> serious danger that warrants the scrutiny it is getting.

Apple removed it from OS X and it's a big pain in the ass.  I'd suggest
fix it (going at it with a big torch if necessary to remove likely dead
and likely dangerous code -- many/most of the options, even things like
linemode) and keep it.

I'd be glad to help.

I see far less reason to keep the daemon than the client.  But walking
up to a purportedly Unix system and finding that I don't have "telnet"
for quick TCP connection testing and have to use nc with all its weird
options that are inconsistent from version to version and annoying
behaviours like refusal to transmit ^C is, to me, even more repellent
than encountering color ls.

Thor

Re: Sample based profiling

[Thor Lancelot Simon]

On Sun, Aug 05, 2018 at 12:22:04AM +0200, Joerg Sonnenberger wrote:
> On Thu, Aug 02, 2018 at 09:26:00PM -0400, Thor Lancelot Simon wrote:
> > On Thu, Aug 02, 2018 at 01:46:35PM +0200, Joerg Sonnenberger wrote:
> > > On Wed, Aug 01, 2018 at 11:54:10PM -0400, Thor Lancelot Simon wrote:
> > > > On Tue, Jul 31, 2018 at 08:28:19AM +, Christos Zoulas wrote:
> > > > > In article <2c408f23-6eae-da00-dfb2-ebc7b66e6...@gmx.com>,
> > > > > Kamil Rytarowski   wrote:
> > > > > >
> > > > > >I'm for removal of gprof and deduplication of .a files in the base.
> > > > > >
> > > > > >While there, can we get rid of moncontrol(3), profil(2) and all the
> > > > > >corresponding code?
> > > > > 
> > > > > I constantly see proposals to remove things from base without 
> > > > > replacing
> > > > > the functionality first. Yes, the current state is poor but the 
> > > > > "hand hurts, cut hand" approach is worse.
> > > > 
> > > > Here it'd also be "cut arm", I think, since the kernel profiling that
> > > > relies on the same tools does still work, and, though limited, can do
> > > > some things out of the box that take nontrivial effort with the DTrace
> > > > FBT provider, the nearest alternative.
> > > 
> > > The kernel is quite different beast as it is self-contained.
> > 
> > But it uses the same userspace tools for analysis.
> 
> I haven't been talking about gprof, I could care less about that
> program itself. That said, I would expect copying the interceptor

Oh, FFS.  Please read the *very first line quoted above*.

I don't care whether you rip out -pg support for userland so long as you
say there's a better alternative - I trust you.  But, please. don't
touch tools shared with the kernel profiling unless and until there is
a fully equivalent and documented alternative in our source tree.

-- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: Sample based profiling

[Thor Lancelot Simon]

On Thu, Aug 02, 2018 at 01:46:35PM +0200, Joerg Sonnenberger wrote:
> On Wed, Aug 01, 2018 at 11:54:10PM -0400, Thor Lancelot Simon wrote:
> > On Tue, Jul 31, 2018 at 08:28:19AM +, Christos Zoulas wrote:
> > > In article <2c408f23-6eae-da00-dfb2-ebc7b66e6...@gmx.com>,
> > > Kamil Rytarowski   wrote:
> > > >
> > > >I'm for removal of gprof and deduplication of .a files in the base.
> > > >
> > > >While there, can we get rid of moncontrol(3), profil(2) and all the
> > > >corresponding code?
> > > 
> > > I constantly see proposals to remove things from base without replacing
> > > the functionality first. Yes, the current state is poor but the 
> > > "hand hurts, cut hand" approach is worse.
> > 
> > Here it'd also be "cut arm", I think, since the kernel profiling that
> > relies on the same tools does still work, and, though limited, can do
> > some things out of the box that take nontrivial effort with the DTrace
> > FBT provider, the nearest alternative.
> 
> The kernel is quite different beast as it is self-contained.

But it uses the same userspace tools for analysis.

Thor

Re: Sample based profiling

[Thor Lancelot Simon]

On Tue, Jul 31, 2018 at 08:28:19AM +, Christos Zoulas wrote:
> In article <2c408f23-6eae-da00-dfb2-ebc7b66e6...@gmx.com>,
> Kamil Rytarowski   wrote:
> >
> >I'm for removal of gprof and deduplication of .a files in the base.
> >
> >While there, can we get rid of moncontrol(3), profil(2) and all the
> >corresponding code?
> 
> I constantly see proposals to remove things from base without replacing
> the functionality first. Yes, the current state is poor but the 
> "hand hurts, cut hand" approach is worse.

Here it'd also be "cut arm", I think, since the kernel profiling that
relies on the same tools does still work, and, though limited, can do
some things out of the box that take nontrivial effort with the DTrace
FBT provider, the nearest alternative.

-- 
 Thor Lancelot Simon t...@panix.com
  "Whether or not there's hope for change is not the question.  If you
   want to be a free person, you don't stand up for human rights because
   it will work, but because it is right."  --Andrei Sakharov

Re: Remove dhclient(8) from base

[Thor Lancelot Simon]

On Thu, Jun 21, 2018 at 04:12:32PM +0100, Roy Marples wrote:
> 
> The release of NetBSD-8 is probably imminent, now is probably the best time
> to do this.

I would favor removing dhclient from base for NetBSD-8, even at this late
date.  

Thor

Re: Moving virecover to ~/

[Thor Lancelot Simon]

On Tue, Nov 14, 2017 at 05:35:54PM -0500, Izaac wrote:
> On Sat, Nov 11, 2017 at 08:10:57PM +0700, Robert Elz wrote:
> > It is unreasonable to inflict on people who don't want/need it, just
> > because it happens to be useful to you.
> 
> Elz?  Robert Elz?  The same one I chastised for a bunch of unnecessary
> changes to /bin/sh several weeks ago?

You looked like an ass then, and you're not doing much better now.

Thor

tac (was Re: GNU shellery)

[Thor Lancelot Simon]

On Tue, Oct 03, 2017 at 06:50:35PM +, David Holland wrote:
> On Mon, Oct 02, 2017 at 04:04:41AM +0700, Robert Elz wrote:
>  > ps: the tac in gnu coreutils is by no means "original" - the tac command
>  > way predates GNU - I forget who created it initially, but the real original
>  > (non GPL'd) version could probably be found if there was a good reason
>  > for that.
> 
> Huh, I thought they'd invented it. Still, it shouldn't matter; it's
> obviously a good idea and it's not like everything the FSF does is
> ipso facto automatically wrong.

A colleague and I spent lunchtime today digging into the prehistory of
"tac".

It appears tac was written by some unknown person, then rewritten
by the late Jay Lepreau of the University of Utah in 1985 and sent
to CSRG for inclusion on the "contrib" tape of 4.3BSD, but may or
may not have shipped with 4.3 -- I can't find the "contrib" sources
for 4.3 in TUHS, maybe I'm not looking right.

If indeed it was in 4.3 "contrib" (which most sites installed) I guess
that's where most of us remember it from.  Or it may have been in
Lites (which Jay worked on) or Research Unix, since evidently Rob
Pike used it (see below).

It appears the copyright history on the version in coreutils
is not correct, though it was probably legitimately donated to FSF
at some point: there was a flame war over dual-licensing of GPL/BSD
code on the OSKit list where Jay was active and it appears he
or the University of Utah lost patience and relicensed a great deal
of code as GPL including code that had formerly had BSD style licenses
only.

Here is the header comment from the version it appears Jay sent
to CSRG:

/*
 * tac.c - Print file segments in reverse order
 *
 * Original line-only version by unknown author off the net.
 * Rewritten in 1985 by Jay Lepreau, Univ of Utah, to allocate memory
 * dynamically, handle string bounded segments (suggested by Rob Pike),
 * and handle pipes.
 */

The historical versions took very different command-line arguments
from the GNU version.  Jonathan Kamens posted Jay's version to
alt.sources in 1991: 
https://groups.google.com/forum/#!search/alt.sources$20uta$20tac$20kamens/alt.sources/WKEKVn9i-JA/LHqlL0Ay__YJ

https://groups.google.com/forum/#!search/utah$20tac$20gpl$20bsd%7Csort:relevance/muc.lists.freebsd.hackers/JaNrZz8_3LU/pI5cvX552WgJ
also discusses this, down towards the bottom of the thread.

Anyway, it seems highly likely this program came from BSD
Unix or perhaps its cousin Research Unix.  Welcome back.

-- 
  Thor Lancelot Simont...@panix.com
 "The two most common variations translate as follows:
illegitimi non carborundum = the unlawful are not silicon carbide
illegitimis non carborundum = the unlawful don't have silicon carbide."

Re: GNU shellery

[Thor Lancelot Simon]

On Mon, Oct 02, 2017 at 02:48:12AM +0700, Robert Elz wrote:
> 
> With this, just commit it (don't forget the set lists).

What he said.

Thor

Re: Future shell work - comments reqyuested

[Thor Lancelot Simon]

On Thu, Jul 13, 2017 at 09:01:26AM -0400, Izaac wrote:
> On Thu, Jul 13, 2017 at 07:46:39AM +0700, Robert Elz wrote:
> > So, opinions?
> 
> Stop.
> 
> Leave /bin/sh alone.

I strongly disagree and note the obvious internal inconsistency in your
argument (such as it is):

> If you want features for a scripting environment, use a different shell.
> 
> The /bin/sh is an environment intended primarily to execute scripts.

In the direction you seem to be headed lies the true and vivid idiocy of
the Debian shell, which is *larger* than our shell (depending on platform
and compiler flags, etc.) yet strips out functionality every other modern
shell offers, such as command line editing -- for purely doctrinaire
reasons, since after all, these features were in what they started with.

Removing features as a way to enforce some kind of religious notion of
programmer discipline (which seems to be what the rest of your message
suggests) is dumb.  I would like to see our shell remain about the size
and speed it is (both of which are best-of-class) but gain something close
to feature parity with ksh.  I think Robert's work is getting us there.

Thor

Re: How to properly daemonize?

[Thor Lancelot Simon]

On Wed, Jul 12, 2017 at 04:12:55PM +0200, Edgar Fu? wrote:
> I know about daemon(3). I'm aware this question is not NetBSD-specific. 
> However, I want to be able to write portable code. Plus there's so much 
> nonsense around on the subject that I want to make sure I'm understanding 
> things correctly in the end.
> 
> Steps to daemonize:

The list in _Advanced Programming in the Unix Environment_ is close to
yours but catches a few other details - like setting the umask to 0.  There
is a (hopefully authorized) copy of the book at
http://poincare.matf.bg.ac.rs/~djenic/apue.pdf - see the beginning of
chapter 13, section 13.3.

Thor

Re: Shipping SSL certificates in the base system

[Thor Lancelot Simon]

On Thu, Jul 06, 2017 at 01:18:44PM +, m...@netbsd.org wrote:
> I don't like such changes because I think netbsd's base is bad at
> handling software.

I disagree.  With NetBSD's base system I get a consistent, fairly
complete traditional Unix system.  I find it comfortable and easy
to use -- unlike, say, the default Debian install where tools like
"ifconfig" are missing unless additional packages are added.

> It bunches a pile of unrelated things in a single
> tarball, garbage collection is done with a 2600 line shell script, and

Better a 2600 line shell script than a half-a-dozen C programs and who knows
which one does what and they all behave differently in edge cases (apt-get,
apt, aptitude, synaptic, wajig...)!  That said, to those who don't come from
traditional Unix environments, perhaps the collection of stuff in "base"
seems arbitrary.  I believe it (plus some modern conveniences like a
web browser) should all be installed by default, a few obsolete things
should perhaps be stripped out, but that a more finely grained
representation of its contents would be a very good thing.

I mean, other very popular systems' base packagers can't even properly
handle file ownership and permissions without truly nasty open-coded
shell scripts burned into every package.  Just by not being stupid, it
seems to me here we end up ahead.

I believe the right solution to the real problem here is, and always has
been, syspkg.  I think it's truly regrettable how we never finished the
work and it basically bitrotted.  It's not clear to me how much effort
would be required to 

> adding just a single file requires me to add an entry to a set list
> that requires a full release build to confirm (can take 8 hours on a
> weak laptop).

You don't need to do a full release build to check that you got the
setlists right.  You just need to do the "sets" step.  This was part
of my standard development workflow (on an anemic laptop, often on
the train with little connectivity) for 6 years.

-- 
  Thor Lancelot Simont...@panix.com

  "We cannot usually in social life pursue a single value or a single moral
   aim, untroubled by the need to compromise with others."  - H.L.A. Hart

Re: float128 in libstdc++

[Thor Lancelot Simon]

On Sat, Jun 24, 2017 at 11:27:47AM +0200, Martin Husemann wrote:
> On Fri, Jun 23, 2017 at 10:27:40PM +0200, Kamil Rytarowski wrote:
> >  3. Disable float128 in libstdc++ for everybody.
> 
> You are missing the obvious one:
> 
>  4. Disable float128 for all architectures that have no hardware support
> for it (this would basically be all 32bit ones plus a few 64bit,
> including x68_64 IIUC) and where the ABI does not require this
> format for long double (for example sparc64).
> 
> Providing this format for anything else just makes no sense to me.
> The availability should not depend on the compiler used.

I am similarly skeptical about whether we should expose a floating
point format not supported by the hardware and which is likely to
cause ill-written or ill-configured applications to run slowly with
software floating point and precision they don't actually need,
instead of quickly with hardware floating point and enough precision
already.

Thor

Re: randomness (crypto?) code example wanted please?

[Thor Lancelot Simon]

On Sun, Jun 25, 2017 at 03:22:07PM +0100, Roy Marples wrote:
> 
> Just like sh, dhcpcd can start very early  sometimwes before enough
> entropy has been generated just to pull blindly from /dev/urandom because it
> can block - or did in the past; it might be fixed now.

No.  Reading from /dev/urandom cannot block, and never could unless the
system on which you observed such behavior had a bug.

-- 
  Thor Lancelot Simont...@panix.com

  "We cannot usually in social life pursue a single value or a single moral
   aim, untroubled by the need to compromise with others."  - H.L.A. Hart

Re: NetBSD on embedded devices

[Thor Lancelot Simon]

On Sun, Jun 04, 2017 at 08:16:16PM +0200, Emmanuel Dreyfus wrote:
> Thor Lancelot Simon <t...@panix.com> wrote:
> 
> > One key idea that may help you is not mine -- it's due to Matt Thomas.  It 
> > is
> > the idea of, making an *additional* bunch of sets files specifying
> > exactly and only the parts of the system you need.  Then install those
> > (embedbase.txt, embedetc.tgz, etc.) rather than base, etc, and so forth.
> > That will give you a simple, maintainable way to build a NetBSD that
> > does what you need, is still a coherent whole rather than a pile of
> > disjoint "system packages", and is much smaller.
> 
> One interesting feature is also to crunchgen(1) binaries to reduce
> memory footprint, which is interesting when booting a kernel with
> ramdisk. It works nicely, but is not documented very much.

It's extremely hard to debug, though, can't crunch some executables
together without nasty hacks, and effectively disables some system
security features.  And it saves less space than it used to now that we
have dynamically linked /.

It's a good tool when you're in too much of a hurry to figure out just
exactly what programs you need to get the job done, but I generally
recommend against.

As an example, late in the development of a product about 10 years ago
I suddenly found that to fix a problem observed in customer beta units
in the field, I needed a very small ramdisk kernel for a single-file
one-shot upgrade.

I managed to make a minimal setlist replacing our already-cut-down
"base" and "etc" with about 30 files, that would fit in a ramdisk of a
couple of MB, crunchgen not required.  This was enough to run a single
carefully written shellscript that fixed the installation problem with
the systems in question.

It took me about a day of experimentation to get it right, but it was
pretty fun really, adjusting the shell script and the setlists for the
right balance of maintainability vs. space.  It wasn't particularly
hard to do.

That said, /rescue is worlds better than busybox and the way the
crunchgen framework can be used to add a program here or there is
pretty cool.  It goes to show how excellent NetBSD is a base for any
embedded product large enough to run _any_ kind of Unix -- much, much
better than any of even the commercial Linux or QNX based starting
points that claim to support this kind of thing.  The only similar
effort I know of really is JunOS; and though JunOS does many things
we don't, they are largely in the realm of support for additional
hardware or networking protocols; I think as a base for a new embedded
system, we're better.

And you can't beat the cross-compilation support for producing a clean,
reproducible system in a simple and elegant way.  Nobody else has that.

Thor

Re: NetBSD on embedded devices

[Thor Lancelot Simon]

On Mon, May 15, 2017 at 02:37:17PM +0300, Jukka Marin wrote:
> 
> I'm wondering what would be the best way of system updates.  I would
> like to have two separate system images, one that is active and running
> and another which can be updated.  At boot time, the system would have
> to check which image to boot from.  (Or maybe I could use chroot or
> some such to select the image to use.. or just mount one or the other..
> or use a virtual machine or.. ;-)

You're describing a fairly basic dual-image embedded setup.  Several
of us have done this with NetBSD over the years but, sad to say, I
think the results belong to each of our former employers.

Here is the best description I can give you of what I did once.  I wish
I had been able to open-source it and put it back in NetBSD, but that
really wasn't in the cards.

You won't be able to use our installer to do this; expect to get very
familiar with fdisk, disklabel, installboot, newfs, and friends.  You
can boot the install CD and pop out to a shell (thank goodness we have
not lobotomized our install CD like others to make this hard) or untar
base and etc into a PXE environment to get yourself going.

One key idea that may help you is not mine -- it's due to Matt Thomas.  It is
the idea of, making an *additional* bunch of sets files specifying
exactly and only the parts of the system you need.  Then install those
(embedbase.txt, embedetc.tgz, etc.) rather than base, etc, and so forth.
That will give you a simple, maintainable way to build a NetBSD that
does what you need, is still a coherent whole rather than a pile of
disjoint "system packages", and is much smaller.

---

Here are some goals:

* Boot quickly
* Maintain multiple complete images to allow quick reversion
* Keep a stable configuration even if system's suddenly powered off
* Working serial console at all stages of boot even with crappy BIOS
* Don't burn out the "disk" by writing it too much

Here's some info on what I did.  It is rather specific to x86 and MBR
system partitioning and would need to be adapted for EFI/GPT or other
firmware and associated label formats.

1) Two complete NetBSD images installed, each described by its own
   partition in the MBR.  Partitioning however also described by a single
   BSD disklabel.

   Once the system's up, the BSD disklabel determines the partitioning.
   But at boot time, BIOS uses the MBR table to figure out which image
   to boot and find the first stage boot loader.  You'll need some
   scripting to ensure these are always in perfect sync.

   Since there's a single BSD disklabel (as opposed to, say, one per
   MBR partition like FreeBSD does it) it's natural and easy for the
   system images in the two MBR partitions to share temp/scratch space,
   access each others' contents for debug or upgrade, etc.  You can even
   boot the system using the boot blocks from image A but the kernel and
   filesystem of image B, if the image is really screwed up.

2) Now think of images "A" and "B", your two system images.  The
   correspondence goes like this.  The first MBR partition (partition 0
   in our fdisk's terminology) is NetBSD wd0a and system image "A" and
   we'll make a directory /A where it's always available when mounted,
   no matter whether image A or B is running.  The second MBR partition
   (NetBSD fdisk #1) is wd0b, system image "B" and is always at /B if
   it is mounted (the inactive image might not be mounted).

3) This brings us to the NetBSD fstab.  A system running from "A" is going
   to have:

/dev/wd0a on /  (ro) <--- there is a symlink to / at /A in this FS
/dev/wd0f on /etc (rw,nodev,nosuid)
tmpfs on /var and /tmp; ptyfs on /dev/pts (rw,nodev,noexec)
/dev/wd0h on /var/crash (rw,noexec,nodev)

*If* we are using the "B" image (upgrading, debugging, etc.)
then we will have /dev/wd0b on /B and /dev/wd0g on /B/etc.  We'll
also have a "swap" partition at /dev/wd0e marked as 'dp' but not
'sw' in the fstab so we can take crash dumps but will never swap.

   If running from image "B", it looks like:

/dev/wd0b on /  <--- there is a symlink to / at /B in this FS
/dev/wd0g on /etc
tmpfs on /var and /tmp; ptyfs on /dev/pts
/dev/wd0h on /var/crash < note the sharing! wd0e is also shared.

   An important note is that your scripting for upgrades, etc. can easily
   find out whether it's running in image "A" or "B" just by looking to
   see whether /A or /B is a symlink to "/", without needing to know disk
   device names etc.
 
4)  If you're following all this you're going to wonder what's with the
separate mount on /etc.  Mounting /etc separately and r/w avoids the
nasty business many vendors' embedded images do with mounting a
memory filesystem and "unpacking" a config database (often basically
just a tar file that's been dd'ed out to a separate disk partition)
into it, etc.

If you're going to mount a separate 

Re: ${LINENO} implementation change for /bin/sh

[Thor Lancelot Simon]

On Mon, May 08, 2017 at 03:12:15PM +, Christos Zoulas wrote:
> In article <25442.1494227...@andromeda.noi.kre.to>,
> Robert Elz  <k...@munnari.oz.au> wrote:
> >In this message ...
> [...]
> >
> >This one I think (though I have not tried it) is implementable, though
> >with more work than the option version, but to me appears a little cleaner
> >and removes both of the questions above - no new option means no need
> >to invent a name for it, and no need to consider its default value.
> >It would mean that the bash, ksh93, /bin/ksh, ... interpretation of how
> >LINENO works in functions embedded in scripts would be adopted by
> >default though.
> >
> >So, what does everyone think about that way?
> 
> meaning that $LINENO is always the physical line number of the script
> regardless if we are in a function or not (which is I guess what ksh/bash do)?

What happens if I pull in a shell function library with "."?  Do I get line
numbers per-file, or does the inclusion cause line numbers to appear to jump
_within_ the file that included, or...?

-- 
  Thor Lancelot Simont...@panix.com

  "We cannot usually in social life pursue a single value or a single moral
   aim, untroubled by the need to compromise with others."  - H.L.A. Hart

Re: Compiling minimal userland?

[Thor Lancelot Simon]

On Fri, Sep 02, 2016 at 03:25:52AM +0200, William Z. wrote:
> Hi,
> 
> I'm currently trying to replace Linux with NetBSD in some of
> my embedded devices.
> 
> I have read "The Guide" and found out how to create a crosscompiler and
> compile a tiny custom kernel. However, I also want to create a very
> minimal userland, but as far as I can tell "The Guide" only explains how
> to compile the full NetBSD userland.
> 
> How can I disable unneeded parts of the userland?

The easiest way is to build everything, but use additional setlists to
produce a sort of "micro-base" and "micro-etc".  Then install these
rather than base and etc.  I have reduced a working
NetBSD userland to well under 64MB in this way, for an embedded
product that shipped thousands of units.  As a test, I reduced it to under
4MB (at which point it was dwarfed by the kernel...) for an upgrade
installer image.  It is a little work to get it going but once done it
is surprisingly easy to maintain.

You may also want to turn off some of the MK options both to shrink
your built binaries and to save time in the build.  There are a number
of options unlikely to be useful for a modern embedded device.

Thor

Re: refine of the GSOC project

[Thor Lancelot Simon]

On Wed, May 11, 2016 at 10:37:58PM -0700, Charles Cui wrote:
> Hi Christos,
> 
>I got the new list post below. Note that the second list is build with
> -D_NETBSD_SOURCE -lpthread -lrt -lm

If you have a proper NetBSD build environment (native or cross compilation)
then -D_NETBSD_SOURCE seems wrong.  If you need it, or see changes in
behavior when using it, I think your compilation environment may still
not be quite right.

Thor

Re: refine of the GSOC project

[Thor Lancelot Simon]

On Mon, May 09, 2016 at 10:58:50AM +0930, Brett Lymn wrote:
> On Mon, May 09, 2016 at 12:53:02AM +, Christos Zoulas wrote:
> > 
> > Heh that's funny. He should then run the build.sh command with the old
> > tooldir name or make a symlink from the old tooldir name to the new one.
> > 
> 
> Why not just "build.sh tools" to regenerate the tools for the new
> kernel version?  Or USETOOLS=NO if it is something quick and dirty...

Why not "build.sh release", followed by installing the kernel, rebooting,
and extracting the already made sets files (all but etc.tgz) using tar with
-p or pax with -p e, followed by postinstall -s etc.tgz fix?  Is there any
reason to do this in a less simple, less sane way?

Thor

Re: refine of the GSOC project

[Thor Lancelot Simon]

   SA_NODEFER
> > | SS_ONSTACK
> > | SS_DISABLE
> > | MINSIGSTKSZ
> >
> > Real-time signals is an area we are missing.
> >
> > | struct sigstack
> > | stack_t
> > | bsd_signal
> > | killpg
> > | sigaltstack_test
> > | sigaltstack
> > | sighold
> > | sigignore
> > | siginterrupt
> > |     sigpause
> > | sigrelse
> > | sigset
> > |
> > | mmap:
> > | POSIX_TYPED_MEM_ALLOCATE
> > | POSIX_TYPED_MEM_ALLOCATE_CONTIG
> > | POSIX_TYPED_MEM_MAP_ALLOCATABLE
> > | struct posix_typed_mem_info
> > | posix_mem_offset
> > | posix_typed_mem_get_info
> > | posix_typed_mem_open
> >
> > I dont know what these are
> >
> > | aio
> > | struct rlimit
> >
> > We have struct rlimit, again we need to check the compilation flags.
> >
> > | cpuset
> > | CPU_ZERO
> > | CPU_SETSIZE
> > | CPU_SET
> > | cpu_set_t
> >
> > This is linux-specific; we have our own and the tests need to be adjusted
> >
> > |
> > | semaphore
> > | _SC_SEM_NSEMS_MAX
> > | features.h
> >
> > christos
> >

-- 
  Thor Lancelot Simont...@panix.com

  "We cannot usually in social life pursue a single value or a single moral
   aim, untroubled by the need to compromise with others."  - H.L.A. Hart

Re: pidfile_lock(3)

[Thor Lancelot Simon]

On Sun, Mar 20, 2016 at 08:53:03PM +, Roy Marples wrote:
> On Sunday 20 March 2016 14:53:46 you wrote:
> > On Sat, 19 Mar 2016 23:29:37 +
> > 
> > Roy Marples  wrote:
> > > pidfile(3) is pretty crap - it just writes to the file without any
> > > locking.
> > 
> > I don't understand why you think any of that matters.  Locks only
> > advisory anyway.  Any noncooperating process can (with sufficient
> > privilege) overwrite the file.  What cooperating process do you imagine
> > would honor a lock on a pidfile?
> 
> So it honors it's own lock to avoid user stupidity of starting the same 
> daemon 
> more than once. This is the primary goal of the function.

That's the wrong way to do it, because it does not work reliably on NFS.
You can take advantage of O_CREAT|O_EXCL or rename() semantics to do the
same thing but in a much more robust way.

Thor