/dev/random is hot garbage
hi, since netbsd won't stop using broken setups like xen (which don't provide randomness) to build packages, why don't we give up on /dev/random entirely?
Re: /dev/random is hot garbage
> since netbsd won't stop using broken setups like xen (which don't > provide randomness) to build packages, why don't we give up on > /dev/random entirely? As in, remove it from everyone's /dev and kernel, or stop using it for package builds, or get rid of it from NetBSD/xen's /dev, or what? I would argue against the first one: just because something is being misused is not, in itself, a reason to get rid of it entirely, at least not when it still has good extant uses. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: /dev/random is hot garbage
On 21.07.2019 16:41, co...@sdf.org wrote:
> hi,
>
> since netbsd won't stop using broken setups like xen (which don't
> provide randomness) to build packages, why don't we give up on
> /dev/random entirely?
>
/dev/random is not defined in POSIX.
http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap10.html
What are the users that really need /dev/random? If there are none we
could symlink /dev/urandom to /dev/random... and maybe add a legacy
fallback /dev/${LETTER}random (strong random).
signature.asc
Description: OpenPGP digital signature
Re: /dev/random is hot garbage
I'm just pissed off with the current status quo that we tolerate everything being broken. AMD CPUs do not provide hardware RNG. NVMe isn't used as an RNG source because it would slow it down too much. netbsd/xen never heard about RNG. Or monotonic time. Why are we using this to build packages?! We allow userland to deplete the entropy pool and then have /dev/random reads block until more randomness is acquired. Which is never, if you use some modern hardware, or old hardware, or embedded hardware, or VMs. Rust reads a byte from /dev/random before deciding it's OK to read /dev/urandom. We are told this is the correct behaviour and that we shouldn't upstream "just use /dev/urandom/sysctl equivalent". Nothing builds, we don't supply binary packages, and nobody cares. "Just get a 128GB RAM 32 core machine like me, and then you can use netbsd"
Re: /dev/random is hot garbage
On 21.07.2019 16:53, Kamil Rytarowski wrote:
> On 21.07.2019 16:41, co...@sdf.org wrote:
>> hi,
>>
>> since netbsd won't stop using broken setups like xen (which don't
>> provide randomness) to build packages, why don't we give up on
>> /dev/random entirely?
>>
>
> /dev/random is not defined in POSIX.
>
> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap10.html
>
> What are the users that really need /dev/random? If there are none we
> could symlink /dev/urandom to /dev/random... and maybe add a legacy
> fallback /dev/${LETTER}random (strong random).
>
For the reference.
It looks like Darwin symlinks /dev/random and /dev/urandom and uses
(used to use?) yarrow pseudorandom number generator algorithm.
signature.asc
Description: OpenPGP digital signature
Re: /dev/random is hot garbage
On Sun, 21 Jul 2019 15:05:53 + co...@sdf.org wrote: > Rust reads a byte from /dev/random before deciding it's OK to read > /dev/urandom. We are told this is the correct behaviour and that we > shouldn't upstream "just use /dev/urandom/sysctl equivalent". Who says this is correct behaviour? It isn't. random(4) has this to say: "Applications should read from /dev/urandom when they need randomly generated data, e.g. key material for cryptography or seeds for simulations. Systems should be engineered to judiciously read at least once from /dev/random at boot before running any services that talk to the internet or otherwise require cryptography, in order to avoid generating keys predictably." My interpretation of the above is that rust is not doing the right thing. It is an application runtime, but behaves like an rc script. Kind regards, -Tobias
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 02:41:57PM +, co...@sdf.org wrote: > hi, > > since netbsd won't stop using broken setups like xen (which don't > provide randomness) to build packages, why don't we give up on > /dev/random entirely? Replacing the /dev/random device node by a symlink to /dev/urandom sounds fine. For binaries it is easy to just use the sysctl instead to get high quality randomness. Are there any shell script like applications that seriously would require something better than /dev/urandom? The other issue is the urban rumour that you may want to pull a real random byte out of /dev/random before using /dev/urandom - maybe we should have a "aggregate" sysctl doing just that (so applications can get a single byte real entropy + as many /dev/urandom ones as they like in a single call)? Martin
Re: /dev/random is hot garbage
I don't think we should change /dev/random. For a very long time, the notion is that the bits from /dev/random really are ok for keys, and there has been a notion that such bits are precious and you should be prepared to wait. If you aren't generating a key, you shouldn't read from /dev/random. So I think rust is wrong and should be fixed. I can see the reason for frustration, but I believe that we should not break things that are sensible because they are abused and cause problems in some environments. It would also be reasonable to have a sysctl to allow /dev/random to return bytes anyway, like urandom would, and to turn this on for our xen builders, as a different workaround. That's easy, and it doesn't break the way things are supposed to be for people that don't ask for it. Also, on the xen build hosts, it would perhaps be good to turn on entropy collection from network and disk. Another approach, harder, is to create a xenrnd(4) pseudodevice and hypervisor call that gets bits from the host's /dev/random and injects them as if from a hardware rng.
Re: /dev/random is hot garbage
> Date: Sun, 21 Jul 2019 17:28:17 +0200 > From: Martin Husemann > > Replacing the /dev/random device node by a symlink to /dev/urandom sounds > fine. For binaries it is easy to just use the sysctl instead to get high > quality randomness. Are there any shell script like applications that > seriously would require something better than /dev/urandom? > > The other issue is the urban rumour that you may want to pull a real random > byte out of /dev/random before using /dev/urandom - maybe we should have > a "aggregate" sysctl doing just that (so applications can get a single byte > real entropy + as many /dev/urandom ones as they like in a single call)? This is the correct way -- that works pretty reliably on almost any platform that has /dev/u?random at all -- for a program to block until the entropy pool has been seeded; there's essentially no other reason ever to read from /dev/random. What may not be clear is _which_ programs need to do this or when, and the farther from the holistic view of system engineering you are, the murkier it gets. * The system view. Someone who is assembling a platform with pre-installed NetBSD to be shipped in a box and deployed needs to ensure, in the system they're shipping, that the entropy pool be seeded by an unpredictable secret _before_ you use any secrets derived from it, e.g. for cryptography. - A system engineer might choose hardware with a hardware RNG. - A system engineer might write an independent seed from their laptop to /var/db/entropy-file on each device (or cloud instance) after flashing it with the standard OS image. - A system engineer might (a) start a daemon in one rc script that reads a seed over a serial port to a Geiger counter with a radiation source, asynchronously; and (b) read a byte from /dev/random in another rc script that has to wait until the seeding daemon has done its job. - A system engineer might flip a coin 256 times, open a shell, and type `echo tththhhththtt... > /dev/random', before starting any applications in a live system. * The application view. Someone who writes an application, like a mail server, which might run in many different systems, needs a way to generate secrets that will be used for cryptography. This is safe only after the entropy pool is seeded -- but the application engineer, who is just writing software, is not assembling the whole system and so can't arrange to set the application up next to a real radiation source and Geiger counter. - An application that has a definite startup phase, like the Postfix master daemon, might reasonably read a single byte from /dev/random at startup, and then use /dev/urandom in all its subprocesses. - An application might reasonably have a command-line argument for a seed file, which is also useful because it facilitates deterministic testing, like gcc's -frandom-seed. - An application might just quietly defer the decision to the system engineer, but the quieter this is, the greater the risk the application will be deployed with a fatal insecurity. * The library view. Someone who writes a library used by many applications, like this Rust vendor/rand library, needs a way to get at secrets from the operating system that will be used to derive other secrets in the library. This, again, is safe only after the entropy pool is seeded, but the library engineer has to make it usable in _many_ applications. A library might read from - /dev/random, - /dev/urandom, - getentropy(), - getrandom(), - sysctl kern.arandom, or something like that. For example, our arc4random library reads from sysctl kern.arandom. Alternatively, a library could accept a parameter -- to be passed by the application -- for a seed, and avoid talking to the operating system at all. Libraries may also be constrained by blocking: it is at least rude, and sometimes a fatal bug or a deadlock, for a library to block when it is expected not to block. It may be especially bad if a library is expected not to block, but blocks _sometimes and only in extremely infrequent circumstances_, like how a POSIX clock skips a beat sometimes but only once every year or two (and simultaneously all over the world, when it does), making it unlikely that the code path will be exercised during tests. Some programs like gpg function more like libraries than like applications in that they are used as subroutines by other programs; the fact that gpg insists on reading every byte of every candidate RSA modulus from /dev/random has led to decades of justifiable frustration with using it as a subroutine. * The OS view. Someone who writes an operating system used by many system engineers that run many applications, using many libraries, needs to provide interfaces for: a way to seed the entropy
Re: /dev/random is hot garbage
> Date: Sun, 21 Jul 2019 11:55:23 -0400 > From: Greg Troxel > > I don't think we should change /dev/random. For a very long time, the > notion is that the bits from /dev/random really are ok for keys, and > there has been a notion that such bits are precious and you should be > prepared to wait. If you aren't generating a key, you shouldn't read > from /dev/random. This notion has been around for a long time, but it doesn't quite match modern cryptography: once you have a 256-bit secret you can have as many secret bits as you want. What reading from /dev/random does that _is_ useful in modern cryptography is that it serves as a barrier to wait for that initial secret. So there's no need to read more than a single byte from it to wait. (One can reject this premise of cryptography, and refuse to use it, and reject TLS/SSH/&c. and everything else, but I'm not interested in getting into those weeds.) > So I think rust is wrong and should be fixed. What Rust's vendor/rand library is doing is sensible _for some goal_: it is waiting for the entropy pool to be seeded before generating keys. Applications may depend on it to do this _at least once_; the trouble arises when applications block on a single byte over and over again. > It would also be reasonable to have a sysctl to allow /dev/random to > return bytes anyway, like urandom would, and to turn this on for our xen > builders, as a different workaround. That's easy, and it doesn't break > the way things are supposed to be for people that don't ask for it. What's the advantage of this over using replacing /dev/random by a symlink to /dev/urandom in the build system? A symlink can be restricted to a chroot, while a sysctl knob would affect the host outside the chroot. The two would presumably require essentially the same privileges to enact.
Re: /dev/random is hot garbage
g...@lexort.com (Greg Troxel) writes: >Another approach, harder, is to create a xenrnd(4) pseudodevice and >hypervisor call that gets bits from the host's /dev/random and injects >them as if from a hardware rng. There is a virtio-rng, implementing that interface would be compatible with other guests. -- -- Michael van Elst Internet: mlel...@serpens.de "A potential Snark may lurk in every tree."
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 04:53:08PM +, Taylor R Campbell wrote: > It has become popular to redefine the traditional semantics of > /dev/random or /dev/urandom so that one or both will block once at > boot until the OS thinks the entropy pool may have been seeded, and > then never block again. > > I don't want to do this because code paths that may block but only > in extreme circumstances, like early at boot on an embedded system, > are likely never to be exercised even during what might otherwise be > extensive testing, and as noted blocking when not expected can have > severe consequences. It currently blocks for literally hours/days. We can't have the OS not function due to this purity.
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 11:55:23AM -0400, Greg Troxel wrote: > Another approach, harder, is to create a xenrnd(4) pseudodevice and > hypervisor call that gets bits from the host's /dev/random and injects > them as if from a hardware rng. > > That requires the ability coordinate "please run this backported patch" to whoever does the package builds. Since we don't let anyone volunteer for tasks and would rather have highly critical things rely on people who stopped having NetBSD time about 5 years ago, that's not going to happen.
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 06:43:04PM +, m...@netbsd.org wrote: > On Sun, Jul 21, 2019 at 11:55:23AM -0400, Greg Troxel wrote: > > Another approach, harder, is to create a xenrnd(4) pseudodevice and > > hypervisor call that gets bits from the host's /dev/random and injects > > them as if from a hardware rng. > > > > > > That requires the ability coordinate "please run this backported patch" > to whoever does the package builds. Since we don't let anyone volunteer > for tasks and would rather have highly critical things rely on people > who stopped having NetBSD time about 5 years ago, that's not going to > happen. no that's not the problem. Lots of nonsense has been written in this thread. /dev/randon actually works as documented and if rust wants /dev/urandom behavior it should use /dev/urandom. Also I'd like to get explained why a compiler needs that much random bits. BTW, while talking about packages availability, when will the bootstrap kit for i386 be available ? -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 08:52:52PM +0200, Manuel Bouyer wrote: > On Sun, Jul 21, 2019 at 06:43:04PM +, m...@netbsd.org wrote: > > On Sun, Jul 21, 2019 at 11:55:23AM -0400, Greg Troxel wrote: > > > Another approach, harder, is to create a xenrnd(4) pseudodevice and > > > hypervisor call that gets bits from the host's /dev/random and injects > > > them as if from a hardware rng. > > > > > > > > > > That requires the ability coordinate "please run this backported patch" > > to whoever does the package builds. Since we don't let anyone volunteer > > for tasks and would rather have highly critical things rely on people > > who stopped having NetBSD time about 5 years ago, that's not going to > > happen. > > no that's not the problem. > Lots of nonsense has been written in this thread. > /dev/randon actually works as documented and if rust wants /dev/urandom > behavior it should use /dev/urandom. Also I'd like to get explained why > a compiler needs that much random bits. > > BTW, while talking about packages availability, when will the bootstrap > kit for i386 be available ? ftp://golden-delicious.urc.uninett.no/pub/rust/rust-std-1.35.0-i686-unknown-netbsd.tar.gz ?
Re: /dev/random is hot garbage
> Date: Sun, 21 Jul 2019 20:52:52 +0200 > From: Manuel Bouyer > > /dev/randon actually works as documented and if rust wants /dev/urandom > behavior it should use /dev/urandom. Also I'd like to get explained why > a compiler needs that much random bits. The difference is that /dev/random may block, and if it blocks, it doesn't wake up until the entropy pool is seeded. In contrast, /dev/urandom never blocks, even if the entropy pool has not yet been seeded. There is no reason in modern cryptography to read more than one byte from /dev/random ever in a single application; once you have done that, or confirmed some other way that the the entropy pool is seeded, you should generate keys from /dev/urandom. What Rust's vendor/rand library seems to guarantee for its callers is that it won't return any data until the entropy pool has been seeded, and then it will return arbitrarily much data without ever blocking again. It does this by reading a single byte from /dev/random, and then generating keys from /dev/urandom. This is _locally_ sensible for a library that may have many users beyond a compiler. But what seems to be happening (although I haven't dived into the build process myself to confirm) is that many subprocesses in the build process are _indepenently_ initializing the Rust vendor/rand library -- reading one byte from /dev/random, which sometimes blocks. In a build chroot, or in a Xen guest, where you aren't handling any secrets (e.g., no sshd except on the local network, no package signing, &c.), you can replace /dev/random by a symlink to /dev/urandom and the build will never block.
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 07:20:08PM +, Taylor R Campbell wrote: > > /dev/randon actually works as documented and if rust wants /dev/urandom > > behavior it should use /dev/urandom. Also I'd like to get explained why > > a compiler needs that much random bits. > > The difference is that /dev/random may block, and if it blocks, it > doesn't wake up until the entropy pool is seeded. In contrast, > /dev/urandom never blocks, even if the entropy pool has not yet been > seeded. > > There is no reason in modern cryptography to read more than one byte > from /dev/random ever in a single application; once you have done > that, or confirmed some other way that the the entropy pool is seeded, > you should generate keys from /dev/urandom. There should be some way to do that without throwing away 8 random bits. > This is _locally_ sensible for a library that may have many users > beyond a compiler. No, not really; it's the system's responsibility. -- David A. Holland dholl...@netbsd.org
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 07:20:08PM +, Taylor R Campbell wrote: > This is _locally_ sensible for a library that may have many users > beyond a compiler. No, it can be sensible behavior to allow *optionally* checking. But it is certainly not sensible default behavior for a library. Joerg
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 06:57:30PM +, m...@netbsd.org wrote: > On Sun, Jul 21, 2019 at 08:52:52PM +0200, Manuel Bouyer wrote: > > On Sun, Jul 21, 2019 at 06:43:04PM +, m...@netbsd.org wrote: > > > On Sun, Jul 21, 2019 at 11:55:23AM -0400, Greg Troxel wrote: > > > > Another approach, harder, is to create a xenrnd(4) pseudodevice and > > > > hypervisor call that gets bits from the host's /dev/random and injects > > > > them as if from a hardware rng. > > > > > > > > > > > > > > That requires the ability coordinate "please run this backported patch" > > > to whoever does the package builds. Since we don't let anyone volunteer > > > for tasks and would rather have highly critical things rely on people > > > who stopped having NetBSD time about 5 years ago, that's not going to > > > happen. > > > > no that's not the problem. > > Lots of nonsense has been written in this thread. > > /dev/randon actually works as documented and if rust wants /dev/urandom > > behavior it should use /dev/urandom. Also I'd like to get explained why > > a compiler needs that much random bits. > > > > BTW, while talking about packages availability, when will the bootstrap > > kit for i386 be available ? > > ftp://golden-delicious.urc.uninett.no/pub/rust/rust-std-1.35.0-i686-unknown-netbsd.tar.gz Why is it not on ftp.NetBSD.org, with other bootstrap kits ? -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 07:20:08PM +, Taylor R Campbell wrote: > > Date: Sun, 21 Jul 2019 20:52:52 +0200 > > From: Manuel Bouyer > > > > /dev/randon actually works as documented and if rust wants /dev/urandom > > behavior it should use /dev/urandom. Also I'd like to get explained why > > a compiler needs that much random bits. > > The difference is that /dev/random may block, and if it blocks, it > doesn't wake up until the entropy pool is seeded. In contrast, > /dev/urandom never blocks, even if the entropy pool has not yet been > seeded. > > There is no reason in modern cryptography to read more than one byte > from /dev/random ever in a single application; once you have done > that, or confirmed some other way that the the entropy pool is seeded, > you should generate keys from /dev/urandom. > > What Rust's vendor/rand library seems to guarantee for its callers is > that it won't return any data until the entropy pool has been seeded, > and then it will return arbitrarily much data without ever blocking > again. It does this by reading a single byte from /dev/random, and > then generating keys from /dev/urandom. > > This is _locally_ sensible for a library that may have many users > beyond a compiler. But what seems to be happening (although I haven't > dived into the build process myself to confirm) is that many > subprocesses in the build process are _indepenently_ initializing the > Rust vendor/rand library -- reading one byte from /dev/random, which > sometimes blocks. I suspect it's the problem. There were several rust processes, all blocked on /dev/random > > In a build chroot, or in a Xen guest, where you aren't handling any > secrets (e.g., no sshd except on the local network, no package > signing, &c.), you can replace /dev/random by a symlink to > /dev/urandom and the build will never block. Actually, I have no idea what the requirements for a full pbulk build are in this area. We'll certainly want to do packages signing at some point. -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: /dev/random is hot garbage
> On Jul 21, 2019, at 3:20 PM, Taylor R Campbell > wrote: > > > [EXTERNAL EMAIL] > >> Date: Sun, 21 Jul 2019 20:52:52 +0200 >> From: Manuel Bouyer >> >> /dev/randon actually works as documented and if rust wants /dev/urandom >> behavior it should use /dev/urandom. Also I'd like to get explained why >> a compiler needs that much random bits. > > The difference is that /dev/random may block, and if it blocks, it > doesn't wake up until the entropy pool is seeded. In contrast, > /dev/urandom never blocks, even if the entropy pool has not yet been > seeded. > > There is no reason in modern cryptography to read more than one byte > from /dev/random ever in a single application; once you have done > that, or confirmed some other way that the the entropy pool is seeded, > you should generate keys from /dev/urandom. The way I see it: /dev/random blocks until it has adequate entropy to deliver cryptographically strong random numbers. Once unblocked it delivers such random numbers. /dev/urandom is equivalent to /dev/random if there is adequate entropy, but it will also deliver random numbers not suitable for cryptography before that time. In addition, the notion of "entropy being consumed" is obsolete (if it was ever valid), so once adequately seeded /dev/random should not block after that. Do we have an implementation that does these things? It's critical to have a good implementation of /dev/random, otherwise you can't run security products. paul
Re: /dev/random is hot garbage
TRC> There is no reason in modern cryptography to read more than one byte TRC> from /dev/random ever in a single application; once you have done TRC> that, or confirmed some other way that the the entropy pool is seeded, TRC> you should generate keys from /dev/urandom. DAH> There should be some way to do that without throwing away 8 random DAH> bits. Isn't that called poll()/select() etc? As far as I understand, it's not about actually reading from /dev/random, but checking whether you could read without blocking, isn't it?
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 08:50:30PM +, paul.kon...@dell.com wrote: > /dev/urandom is equivalent to /dev/random if there is adequate entropy, > but it will also deliver random numbers not suitable for cryptography before > that time. This is somewhat misleading. The problem is that with an unknown entropy state, the system cannot ensure that an attacker couldn't predict the seed used for the /dev/urandom stream. That doesn't mean that the stream itself is bad. It will still pass any statistical test etc. Note that with the option of seeding the CPRNG at boot time, a lot of the distinction is actually moot. Joerg
Re: /dev/random is hot garbage
> "Just get a 128GB RAM 32 core machine like me, and then you can use > netbsd" Hasn't that been NetBSD's stance since 2011-03-30? (Okay, it's an exaggeration, but not all that much of an exaggeration.) /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: /dev/random is hot garbage
> In addition, the notion of "entropy being consumed" is obsolete (if it was e$ Oh, it most certainly was, and is, a valid notion. It may currently appear that the state of the art in PRNGs is good enough that you can stream unlimited amounts of key material once you have enough entropy on hand, but it will not remain so forever. History repeatedly teaches us that "always" and "forever" never are, perhaps especially in cryptography. (It wasn't all that long ago that a Vignière cipher was the ultimate in unbreakable cryptography.) Getting rid of the notion of consumable entropy now will just mean someone needs to reimplement it in the future, when the state of the art once again becomes such that the state-inferrers have the advantage. > Do we have an implementation that does these things? It's critical to have $ Only those that use cryptography, and even then only those so sloppily designed that they (a) have no fallback for systems that don't export a strong random-number interface and (b) trust that interface to perform up to its advertised design specs. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: /dev/random is hot garbage
> On Jul 21, 2019, at 4:55 PM, Edgar Fuß wrote: > > > [EXTERNAL EMAIL] > > TRC> There is no reason in modern cryptography to read more than one byte > TRC> from /dev/random ever in a single application; once you have done > TRC> that, or confirmed some other way that the the entropy pool is seeded, > TRC> you should generate keys from /dev/urandom. > > DAH> There should be some way to do that without throwing away 8 random > DAH> bits. > Isn't that called poll()/select() etc? > As far as I understand, it's not about actually reading from /dev/random, > but checking whether you could read without blocking, isn't it? I don't agree with this reasoning. If /dev/random is implemented right, it won't block later once it unblocks for the first time. Given that, an application that needs a cryptographic random number should simply fetch all the bits it needs from /dev/random. The only reason to read from /dev/urandom is that you want random numbers but they don't need to be strong. If people do these hacks because we still have the "entropy is used up" notion in the code, the answer is to remove that. paul
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 06:33:27PM +, co...@sdf.org wrote: > It currently blocks for literally hours/days. We can't have the OS not > function due to this purity. A rust build blocking (due to rustc or the rust libraries doing stupid things) is very much different from "the OS not function". Martin
Re: /dev/random is hot garbage
Taylor R Campbell writes: >> It would also be reasonable to have a sysctl to allow /dev/random to >> return bytes anyway, like urandom would, and to turn this on for our xen >> builders, as a different workaround. That's easy, and it doesn't break >> the way things are supposed to be for people that don't ask for it. > > What's the advantage of this over using replacing /dev/random by a > symlink to /dev/urandom in the build system? > > A symlink can be restricted to a chroot, while a sysctl knob would > affect the host outside the chroot. The two would presumably require > essentially the same privileges to enact. None, now that I think of it. So let's change that on the xen build host. And, the other issue is that systems need randomness, and we need a way to inject some into xen guests. Enabling some with rndctl works, or at least used to, even if it is theoretically dangerous. But we aren't trying to defend against the dom0.
Re: /dev/random is hot garbage
>> ftp://golden-delicious.urc.uninett.no/pub/rust/rust-std-1.35.0-i686-unknown-netbsd.tar.gz > > Why is it not on ftp.NetBSD.org, with other bootstrap kits? Because copying the bootstrap kits takes forever... I'm in the process of copying them over, but it's a multi-day effort. Regards, - Håvard
Re: /dev/random is hot garbage
On Mon, Jul 22, 2019 at 9:16 AM Havard Eidnes wrote: > > >> ftp://golden-delicious.urc.uninett.no/pub/rust/rust-std-1.35.0-i686-unknown-netbsd.tar.gz > > > > Why is it not on ftp.NetBSD.org, with other bootstrap kits? > > Because copying the bootstrap kits takes forever... > I'm in the process of copying them over, but it's a multi-day > effort. > > Regards, > > - Håvard Because scp is blocked waiting for random bits? :)
Re: /dev/random is hot garbage
>> >> ftp://golden-delicious.urc.uninett.no/pub/rust/rust-std-1.35.0-i686-unknown-netbsd.tar.gz
>> >
>> > Why is it not on ftp.NetBSD.org, with other bootstrap kits?
>>
>> Because copying the bootstrap kits takes forever...
>> I'm in the process of copying them over, but it's a multi-day
>> effort.
>
> Because scp is blocked waiting for random bits? :)
Heh.
I think there's just some bad network in the middle here. No
packet loss, though, but the congestion window nearly never opens
up above 64KB. Then there's the application-window in ssh
itself, but we have the HPN additions on both client and server,
so it *should* be able to scale that higher than 64KB(?)
: {5} ; iperf3 -i 3 -t 30 -c ftp.netbsd.org
Connecting to host ftp.netbsd.org, port 5201
[ 6] local 2001:700:1:0:eeb1:d7ff:fe59:fbaa port 57151 connected to
2001:470:a085:999::21 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 6] 0.00-3.01 sec 1.03 MBytes 2.87 Mbits/sec0 5.50 KBytes
[ 6] 3.01-6.01 sec 678 KBytes 1.85 Mbits/sec0 6.88 KBytes
[ 6] 6.01-9.00 sec 928 KBytes 2.54 Mbits/sec0 13.8 KBytes
[ 6] 9.00-12.02 sec 1.23 MBytes 3.43 Mbits/sec0 37.1 KBytes
[ 6] 12.02-15.01 sec 1.15 MBytes 3.22 Mbits/sec0 22.0 KBytes
[ 6] 15.01-18.01 sec 1.16 MBytes 3.25 Mbits/sec0113 KBytes
[ 6] 18.01-21.01 sec 1.97 MBytes 5.53 Mbits/sec0 75.6 KBytes
[ 6] 21.01-24.01 sec 796 KBytes 2.17 Mbits/sec0 27.5 KBytes
[ 6] 24.01-27.01 sec 1.36 MBytes 3.80 Mbits/sec0 39.9 KBytes
[ 6] 27.01-30.01 sec 1.12 MBytes 3.12 Mbits/sec0 68.8 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 6] 0.00-30.01 sec 11.4 MBytes 3.18 Mbits/sec0 sender
[ 6] 0.00-30.01 sec 10.9 MBytes 3.05 Mbits/sec receiver
iperf Done.
: {6} ; iperf3 -4 -i 3 -t 30 -c ftp.netbsd.org
Connecting to host ftp.netbsd.org, port 5201
[ 6] local 158.38.62.77 port 59576 connected to 199.233.217.201 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 6] 0.00-3.00 sec 261 KBytes 713 Kbits/sec0 11.3 KBytes
[ 6] 3.00-6.01 sec 215 KBytes 586 Kbits/sec0 1.41 KBytes
[ 6] 6.01-9.00 sec 217 KBytes 593 Kbits/sec0 11.3 KBytes
[ 6] 9.00-12.00 sec 178 KBytes 486 Kbits/sec0 28.3 KBytes
[ 6] 12.00-15.01 sec 263 KBytes 718 Kbits/sec0 2.83 KBytes
[ 6] 15.01-18.01 sec 212 KBytes 580 Kbits/sec0 21.2 KBytes
[ 6] 18.01-21.00 sec 270 KBytes 738 Kbits/sec0 7.07 KBytes
[ 6] 21.00-24.02 sec 322 KBytes 877 Kbits/sec0 32.5 KBytes
[ 6] 24.02-27.01 sec 266 KBytes 727 Kbits/sec0 9.90 KBytes
[ 6] 27.01-30.02 sec 170 KBytes 462 Kbits/sec0 5.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 6] 0.00-30.02 sec 2.32 MBytes 648 Kbits/sec0 sender
[ 6] 0.00-30.02 sec 2.21 MBytes 617 Kbits/sec receiver
iperf Done.
: {7} ; traceroute -I ftp.netbsd.org
traceroute to ftp.netbsd.org (199.233.217.201), 64 hops max, 40 byte packets
1 uninett-gsw1 (158.38.62.8) 0.986 ms 0.775 ms 2.415 ms
2 teknobyen-gw4 (128.39.70.8) 0.788 ms 0.790 ms 0.781 ms
3 trd-gw (128.39.230.116) 0.762 ms 29.401 ms 6.203 ms
4 oslo-gw1 (128.39.255.24) 8.750 ms 8.730 ms 8.764 ms
5 se-tug.nordu.net (109.105.102.108) 15.597 ms 15.703 ms 15.563 ms
6 se-fre.nordu.net (109.105.97.1) 15.474 ms 21.950 ms 15.410 ms
7 s-b3-link.telia.net (213.248.97.17) 15.915 ms 17.054 ms 15.955 ms
8 s-bb4-link.telia.net (62.115.119.110) 15.915 ms 15.893 ms 15.799 ms
9 ffm-bb2-link.telia.net (62.115.138.105) 39.084 ms 39.145 ms 39.210 ms
10 ffm-b1-link.telia.net (62.115.137.167) 37.740 ms 37.362 ms 41.580 ms
11 f5networks-ic-341210-ffm-b1.c.telia.net (62.115.169.109) 38.844 ms 38.898
ms 38.871 ms
12 pccw-bb-fra1.sjc1.defense.net (107.162.2.50) 176.183 ms 174.953 ms
189.680 ms
13 107.162.79.2 (107.162.79.2) 173.713 ms 181.094 ms 180.158 ms
14 107.162.16.70 (107.162.16.70) 181.437 ms 191.085 ms 185.156 ms
15 morden.netbsd.org (199.233.217.201) 179.934 ms 194.359 ms 194.508 ms
: {8} ;
: {8} ; traceroute6 -I ftp.netbsd.org
traceroute6 to ftp.netbsd.org (2001:470:a085:999::21) from
2001:700:1:0:eeb1:d7ff:fe59:fbaa, 64 hops max, 16 byte packets
1 uninett-gsw2 0.818 ms 0.793 ms 0.782 ms
2 uninett-gsw1 0.793 ms 0.793 ms 0.761 ms
3 teknobyen-gw4 0.79 ms 0.794 ms 0.789 ms
4 trd-gw 0.784 ms 0.789 ms 0.778 ms
5 oslo-gw1 8.774 ms 9.899 ms 8.611 ms
6 se-tug.nordu.net 15.581 ms 23.129 ms 18.114 ms
7 se-fre.nordu.net 15.516 ms 15.499 ms 15.492 ms
8 dk-o
Re: /dev/random is hot garbage
On Mon, Jul 22, 2019 at 10:08 AM Havard Eidnes wrote:
>
> >> >> ftp://golden-delicious.urc.uninett.no/pub/rust/rust-std-1.35.0-i686-unknown-netbsd.tar.gz
> >> >
> >> > Why is it not on ftp.NetBSD.org, with other bootstrap kits?
> >>
> >> Because copying the bootstrap kits takes forever...
> >> I'm in the process of copying them over, but it's a multi-day
> >> effort.
> >
> > Because scp is blocked waiting for random bits? :)
>
> Heh.
>
> I think there's just some bad network in the middle here. No
> packet loss, though, but the congestion window nearly never opens
> up above 64KB. Then there's the application-window in ssh
> itself, but we have the HPN additions on both client and server,
> so it *should* be able to scale that higher than 64KB(?)
>
>
> : {5} ; iperf3 -i 3 -t 30 -c ftp.netbsd.org
> Connecting to host ftp.netbsd.org, port 5201
> [ 6] local 2001:700:1:0:eeb1:d7ff:fe59:fbaa port 57151 connected to
> 2001:470:a085:999::21 port 5201
> [ ID] Interval Transfer Bandwidth Retr Cwnd
> [ 6] 0.00-3.01 sec 1.03 MBytes 2.87 Mbits/sec0 5.50 KBytes
> [ 6] 3.01-6.01 sec 678 KBytes 1.85 Mbits/sec0 6.88 KBytes
> [ 6] 6.01-9.00 sec 928 KBytes 2.54 Mbits/sec0 13.8 KBytes
> [ 6] 9.00-12.02 sec 1.23 MBytes 3.43 Mbits/sec0 37.1 KBytes
> [ 6] 12.02-15.01 sec 1.15 MBytes 3.22 Mbits/sec0 22.0 KBytes
> [ 6] 15.01-18.01 sec 1.16 MBytes 3.25 Mbits/sec0113 KBytes
> [ 6] 18.01-21.01 sec 1.97 MBytes 5.53 Mbits/sec0 75.6 KBytes
> [ 6] 21.01-24.01 sec 796 KBytes 2.17 Mbits/sec0 27.5 KBytes
> [ 6] 24.01-27.01 sec 1.36 MBytes 3.80 Mbits/sec0 39.9 KBytes
> [ 6] 27.01-30.01 sec 1.12 MBytes 3.12 Mbits/sec0 68.8 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bandwidth Retr
> [ 6] 0.00-30.01 sec 11.4 MBytes 3.18 Mbits/sec0 sender
> [ 6] 0.00-30.01 sec 10.9 MBytes 3.05 Mbits/sec receiver
>
> iperf Done.
> : {6} ; iperf3 -4 -i 3 -t 30 -c ftp.netbsd.org
> Connecting to host ftp.netbsd.org, port 5201
> [ 6] local 158.38.62.77 port 59576 connected to 199.233.217.201 port 5201
> [ ID] Interval Transfer Bandwidth Retr Cwnd
> [ 6] 0.00-3.00 sec 261 KBytes 713 Kbits/sec0 11.3 KBytes
> [ 6] 3.00-6.01 sec 215 KBytes 586 Kbits/sec0 1.41 KBytes
> [ 6] 6.01-9.00 sec 217 KBytes 593 Kbits/sec0 11.3 KBytes
> [ 6] 9.00-12.00 sec 178 KBytes 486 Kbits/sec0 28.3 KBytes
> [ 6] 12.00-15.01 sec 263 KBytes 718 Kbits/sec0 2.83 KBytes
> [ 6] 15.01-18.01 sec 212 KBytes 580 Kbits/sec0 21.2 KBytes
> [ 6] 18.01-21.00 sec 270 KBytes 738 Kbits/sec0 7.07 KBytes
> [ 6] 21.00-24.02 sec 322 KBytes 877 Kbits/sec0 32.5 KBytes
> [ 6] 24.02-27.01 sec 266 KBytes 727 Kbits/sec0 9.90 KBytes
> [ 6] 27.01-30.02 sec 170 KBytes 462 Kbits/sec0 5.66 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval Transfer Bandwidth Retr
> [ 6] 0.00-30.02 sec 2.32 MBytes 648 Kbits/sec0 sender
> [ 6] 0.00-30.02 sec 2.21 MBytes 617 Kbits/sec receiver
>
> iperf Done.
> : {7} ; traceroute -I ftp.netbsd.org
> traceroute to ftp.netbsd.org (199.233.217.201), 64 hops max, 40 byte packets
> 1 uninett-gsw1 (158.38.62.8) 0.986 ms 0.775 ms 2.415 ms
> 2 teknobyen-gw4 (128.39.70.8) 0.788 ms 0.790 ms 0.781 ms
> 3 trd-gw (128.39.230.116) 0.762 ms 29.401 ms 6.203 ms
> 4 oslo-gw1 (128.39.255.24) 8.750 ms 8.730 ms 8.764 ms
> 5 se-tug.nordu.net (109.105.102.108) 15.597 ms 15.703 ms 15.563 ms
> 6 se-fre.nordu.net (109.105.97.1) 15.474 ms 21.950 ms 15.410 ms
> 7 s-b3-link.telia.net (213.248.97.17) 15.915 ms 17.054 ms 15.955 ms
> 8 s-bb4-link.telia.net (62.115.119.110) 15.915 ms 15.893 ms 15.799 ms
> 9 ffm-bb2-link.telia.net (62.115.138.105) 39.084 ms 39.145 ms 39.210 ms
> 10 ffm-b1-link.telia.net (62.115.137.167) 37.740 ms 37.362 ms 41.580 ms
> 11 f5networks-ic-341210-ffm-b1.c.telia.net (62.115.169.109) 38.844 ms
> 38.898 ms 38.871 ms
> 12 pccw-bb-fra1.sjc1.defense.net (107.162.2.50) 176.183 ms 174.953 ms
> 189.680 ms
> 13 107.162.79.2 (107.162.79.2) 173.713 ms 181.094 ms 180.158 ms
> 14 107.162.16.70 (107.162.16.70) 181.437 ms 191.085 ms 185.156 ms
> 15 morden.netbsd.org (199.233.217.201) 179.934 ms 194.359 ms 194.508 ms
> : {8} ;
> : {8} ; traceroute6 -I ftp.netbsd.org
> traceroute6 to ftp.netbsd.org (2001:470:a085:999::21) from
> 2001:700:1:0:eeb1:d7ff:fe59:fbaa, 64 hops max, 16 byte packets
> 1 uninett-gsw2 0.818 ms 0.793 ms 0.782 ms
> 2 uninett-gsw1 0.793 ms 0.793 ms 0.761 ms
> 3 teknobyen-gw4 0.79 ms 0.794 ms 0.789 ms
> 4 trd-gw 0.784 ms 0.789 ms 0.778 ms
> 5 oslo-gw1 8.774 ms 9.899 ms 8.611 ms
> 6 se-tug.nordu.net 15.58
Re: /dev/random is hot garbage
On 22/07/2019 15:08, Havard Eidnes wrote: ftp://golden-delicious.urc.uninett.no/pub/rust/rust-std-1.35.0-i686-unknown-netbsd.tar.gz Why is it not on ftp.NetBSD.org, with other bootstrap kits? Because copying the bootstrap kits takes forever... I'm in the process of copying them over, but it's a multi-day effort. Because scp is blocked waiting for random bits? :) Heh. I think there's just some bad network in the middle here. No packet loss, though, but the congestion window nearly never opens up above 64KB. Then there's the application-window in ssh itself, but we have the HPN additions on both client and server, so it *should* be able to scale that higher than 64KB(?) FWIW I cannot ssh to any NetBSD machine using my HE IPv6 tunnel. $ traceroute6 -I ftp.netbsd.org traceroute6 to ftp.netbsd.org (2001:470:a085:999::21) from 2001:470:690c:1::61, 64 hops max, 16 byte packets 1 gw 9.377 ms 9.48 ms 9.498 ms 2 tunnel391848.tunnel.tserv5.lon1.ipv6.he.net 40.269 ms 69.627 ms 39.12 ms 3 10ge3-16.core1.lon2.he.net 29.34 ms 49.477 ms 29.342 ms 4 100ge13-2.core1.nyc4.he.net 99.374 ms 99.314 ms 100.149 ms 5 100ge8-1.core1.sjc2.he.net 158.68 ms 159.794 ms 159.318 ms 6 10ge4-4.core1.sjc1.he.net 170.059 ms 169.074 ms 159.468 ms 7 e0-50.core4.fmt1.he.net 169.889 ms 168.744 ms 169.616 ms 8 tserv1.fmt1.he.net 159.497 ms 159.266 ms 168.968 ms 9 tunnel426-pt.tunnel.tserv1.fmt.ipv6.he.net 159.609 ms 158.634 ms 159.788 ms 10 ftp.NetBSD.org 159.498 ms 160.065 ms 159.795 ms $ ping6 ftp.netbsd.org PING6(56=40+8+8 bytes) 2001:470:690c:1::61 --> 2001:470:a085:999::21 16 bytes from 2001:470:a085:999::21, icmp_seq=0 hlim=55 time=166.232 ms 16 bytes from 2001:470:a085:999::21, icmp_seq=1 hlim=55 time=169.547 ms ^C --- ftp.netbsd.org ping6 statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/std-dev = 166.232/167.889/169.547/2.344 ms xenity$ ssh -vvv ftp.netbsd.org OpenSSH_8.0 NetBSD_Secure_Shell-20190418-hpn13v14-lpk, OpenSSL 1.1.1b 26 Feb 2019 debug1: Reading configuration data /home/roy/.ssh/config debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 55: Applying options for *.netbsd.org debug2: resolving "ftp.netbsd.org" port 22 debug2: ssh_connect_direct debug1: Connecting to ftp.netbsd.org [2001:470:a085:999::21] port 22. Hangs for a long time before falling back to IPv4 where it works perfectly. Oddly, I can ssh using IPv6 from these NetBSD hosts back to me so I have nfc why this is. So, try forcing scp over IPv4 maybe? Roy
Re: /dev/random is hot garbage
> Is that defense.net your own outgoing protection or is there a route > messed up? I've used them for inline stuff before. Not mine, no. > My traceroute is different to say the least! But your location is also quite different... Regards, - Håvard
Re: /dev/random is hot garbage
On Sun, Jul 21, 2019 at 09:13:48PM +, paul.kon...@dell.com wrote: > > > > On Jul 21, 2019, at 5:03 PM, Joerg Sonnenberger wrote: > > > > > > [EXTERNAL EMAIL] > > > > On Sun, Jul 21, 2019 at 08:50:30PM +, paul.kon...@dell.com wrote: > >> /dev/urandom is equivalent to /dev/random if there is adequate entropy, > >> but it will also deliver random numbers not suitable for cryptography > >> before that time. > > > > This is somewhat misleading. The problem is that with an unknown entropy > > state, the system cannot ensure that an attacker couldn't predict the > > seed used for the /dev/urandom stream. That doesn't mean that the stream > > itself is bad. It will still pass any statistical test etc. > > That's exactly my point. If you're interested in a statistically high > quality pseudo-random bit stream, /dev/urandom is a gread source. But > if you need a cryptographically strong random number, then you can't > safely proceed with an unknown entropy state for the reason you stated, > which translates into "you must use /dev/random". That distinction makes no sense at all to me. /dev/urandom is *always* a cryptographically strong RNG. The only difference here is that without enough entropy during initialisation of the stream, you can brute force the entropy state and see if you get a matching output stream based on that seed. Joerg
Re: /dev/random is hot garbage
> On Jul 22, 2019, at 10:52 AM, Joerg Sonnenberger wrote: > > > [EXTERNAL EMAIL] > > On Sun, Jul 21, 2019 at 09:13:48PM +, paul.kon...@dell.com wrote: >> >> >>> On Jul 21, 2019, at 5:03 PM, Joerg Sonnenberger wrote: >>> >>> >>> [EXTERNAL EMAIL] >>> >>> On Sun, Jul 21, 2019 at 08:50:30PM +, paul.kon...@dell.com wrote: /dev/urandom is equivalent to /dev/random if there is adequate entropy, but it will also deliver random numbers not suitable for cryptography before that time. >>> >>> This is somewhat misleading. The problem is that with an unknown entropy >>> state, the system cannot ensure that an attacker couldn't predict the >>> seed used for the /dev/urandom stream. That doesn't mean that the stream >>> itself is bad. It will still pass any statistical test etc. >> >> That's exactly my point. If you're interested in a statistically high >> quality pseudo-random bit stream, /dev/urandom is a gread source. But >> if you need a cryptographically strong random number, then you can't >> safely proceed with an unknown entropy state for the reason you stated, >> which translates into "you must use /dev/random". > > That distinction makes no sense at all to me. /dev/urandom is *always* a > cryptographically strong RNG. The only difference here is that without > enough entropy during initialisation of the stream, you can brute force > the entropy state and see if you get a matching output stream based on > that seed. I use a different definition of "cryptographically strong". A bit string that's guessable is never, by any useful definition, "cryptographically strong" no matter what the properties of the string extender are. The only useful definition for the term I can see is as a synonym for "suitable for security critical value in cryptographic algorithms". An unseeded /dev/urandom output is not such a value. RFC 1750 is still a useful resource even though it's 25 years old. There is newer work by highly respected cryptographers, too. paul
Re: /dev/random is hot garbage
On 22.07.2019 13:12, Greg Troxel wrote: > Taylor R Campbell writes: > >>> It would also be reasonable to have a sysctl to allow /dev/random to >>> return bytes anyway, like urandom would, and to turn this on for our xen >>> builders, as a different workaround. That's easy, and it doesn't break >>> the way things are supposed to be for people that don't ask for it. >> >> What's the advantage of this over using replacing /dev/random by a >> symlink to /dev/urandom in the build system? >> >> A symlink can be restricted to a chroot, while a sysctl knob would >> affect the host outside the chroot. The two would presumably require >> essentially the same privileges to enact. > > None, now that I think of it. > > So let's change that on the xen build host. > > And, the other issue is that systems need randomness, and we need a way > to inject some into xen guests. Enabling some with rndctl works, or at > least used to, even if it is theoretically dangerous. But we aren't > trying to defend against the dom0. > It looks like we need a paravirt random driver for xen that could solve the rust / random(6) problem. There is already viornd(4) for virtio(4). signature.asc Description: OpenPGP digital signature
Re: /dev/random is hot garbage
> So, try forcing scp over IPv4 maybe? Actually, configuring IPv6 in my case (I didn't have that already) works much better: rust-std-1.32.0-i686-unknown-netbsd.tar.gz 72,048,862 100% 184.72kB/s0:06:20 (xfr#3, to-chk=22/69) rust-std-1.32.0-powerpc-unknown-netbsd.tar.gz 69,624,858 100% 165.15kB/s0:06:51 (xfr#4, to-chk=21/69) rust-std-1.32.0-sparc64-unknown-netbsd.tar.gz 71,612,611 100% 162.35kB/s0:07:10 (xfr#5, to-chk=20/69) Regards, - Håvard
Re: /dev/random is hot garbage
On Mon, Jul 22, 2019 at 07:56:02PM +0200, Havard Eidnes wrote: > > So, try forcing scp over IPv4 maybe? > > Actually, configuring IPv6 in my case (I didn't have that > already) works much better: > > rust-std-1.32.0-i686-unknown-netbsd.tar.gz > 72,048,862 100% 184.72kB/s0:06:20 (xfr#3, to-chk=22/69) > rust-std-1.32.0-powerpc-unknown-netbsd.tar.gz > 69,624,858 100% 165.15kB/s0:06:51 (xfr#4, to-chk=21/69) > rust-std-1.32.0-sparc64-unknown-netbsd.tar.gz > 71,612,611 100% 162.35kB/s0:07:10 (xfr#5, to-chk=20/69) This is also the average speed I got from pkgbuild. -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: /dev/random is hot garbage
> On Sun, Jul 21, 2019 at 04:53:08PM +, Taylor R Campbell wrote: >> It has become popular to redefine the traditional semantics of >> /dev/random or /dev/urandom so that one or both will block once at >> boot until the OS thinks the entropy pool may have been seeded, and >> then never block again. Right, I get the impression this is what FreeBSD does. BTW, you are all of course aware of https://www.2uo.de/myths-about-urandom which provides a good discussion about many of these issues, but perhaps with most of the focus on the myth of /dev/random somehow providing "better" randomness than /dev/urandom (I didn't see this reference mentioned earlier in the discussion, and thought it might be relevant). Now, I'm sure some of the nitty-gritty details of how /dev/random and /dev/urandom is in NetBSD is slightly different from Linux -- perhaps someone more familiar with the finer details may speak up if there are any significant differences. >> I don't want to do this because code paths that may block but only >> in extreme circumstances, like early at boot on an embedded system, >> are likely never to be exercised even during what might otherwise be >> extensive testing, and as noted blocking when not expected can have >> severe consequences. I'm not sure I agree with this argument, especially as long as the silliness with continuing to insist on using /dev/random for "better" random numbers is well and alive, or even as the case with rust's "rnd" library insisting on taking on the role of what should arguably be the system's responsibility (ensuring the system's randomness system has been primed). > It currently blocks for literally hours/days. I agree this is not good. Until Xen is fitted with some device for randomness, I think I'd as system administrator pick the "symlink /dev/random to /dev/urandom" approach, perhaps also turning on net and disk as randomness sources, and also have something in the system startup which ensures /dev/random is properly seeded. Regards, - Håvard
Re: /dev/random is hot garbage
Kamil Rytarowski writes: > On 22.07.2019 13:12, Greg Troxel wrote: >> Taylor R Campbell writes: >> It would also be reasonable to have a sysctl to allow /dev/random to return bytes anyway, like urandom would, and to turn this on for our xen builders, as a different workaround. That's easy, and it doesn't break the way things are supposed to be for people that don't ask for it. >>> >>> What's the advantage of this over using replacing /dev/random by a >>> symlink to /dev/urandom in the build system? >>> >>> A symlink can be restricted to a chroot, while a sysctl knob would >>> affect the host outside the chroot. The two would presumably require >>> essentially the same privileges to enact. >> >> None, now that I think of it. >> >> So let's change that on the xen build host. >> >> And, the other issue is that systems need randomness, and we need a way >> to inject some into xen guests. Enabling some with rndctl works, or at >> least used to, even if it is theoretically dangerous. But we aren't >> trying to defend against the dom0. >> > > It looks like we need a paravirt random driver for xen that could solve > the rust / random(6) problem. > > There is already viornd(4) for virtio(4). rnd and Xen guests is a vexing problem. Lots of things seem to consume bits from the pool until you are often left with none. For me it was Kerberos authentication against a Postgresql DB, but ssh seems to use them and it appears that some are consumed when you use ntpd keys with peers. I built this -> https://anduin.eldar.org/true-rng/ and feed randomness into the Xen guests I have and other systems that I suspect do not produce randomness on their own very well. It is not at all a perfect answer, but appears to work well enough for what I need. For Xen guests, a paravirt driver would seem to be a better answer. -- Brad Spencer - b...@anduin.eldar.org - KC8VKS - http://anduin.eldar.org
Re: /dev/random is hot garbage
On Mon, Jul 22, 2019 at 04:36:41PM +, paul.kon...@dell.com wrote: > > > > On Jul 22, 2019, at 10:52 AM, Joerg Sonnenberger wrote: > > > > > > [EXTERNAL EMAIL] > > > > On Sun, Jul 21, 2019 at 09:13:48PM +, paul.kon...@dell.com wrote: > >> > >> > >>> On Jul 21, 2019, at 5:03 PM, Joerg Sonnenberger wrote: > >>> > >>> > >>> [EXTERNAL EMAIL] > >>> > >>> On Sun, Jul 21, 2019 at 08:50:30PM +, paul.kon...@dell.com wrote: > /dev/urandom is equivalent to /dev/random if there is adequate entropy, > but it will also deliver random numbers not suitable for cryptography > before that time. > >>> > >>> This is somewhat misleading. The problem is that with an unknown entropy > >>> state, the system cannot ensure that an attacker couldn't predict the > >>> seed used for the /dev/urandom stream. That doesn't mean that the stream > >>> itself is bad. It will still pass any statistical test etc. > >> > >> That's exactly my point. If you're interested in a statistically high > >> quality pseudo-random bit stream, /dev/urandom is a gread source. But > >> if you need a cryptographically strong random number, then you can't > >> safely proceed with an unknown entropy state for the reason you stated, > >> which translates into "you must use /dev/random". > > > > That distinction makes no sense at all to me. /dev/urandom is *always* a > > cryptographically strong RNG. The only difference here is that without > > enough entropy during initialisation of the stream, you can brute force > > the entropy state and see if you get a matching output stream based on > > that seed. > > I use a different definition of "cryptographically strong". A bit string > that's guessable is never, by any useful definition, "cryptographically > strong" no matter what the properties of the string extender are. The > only useful definition for the term I can see is as a synonym for > "suitable for security critical value in cryptographic algorithms". > An unseeded /dev/urandom output is not such a value. Again, that's not really a sensible definition. It's always possible to guess the seed of used by the /dev/urandom CPRNG. By definition. That doesn't change the core properties though: there is no sensible way to predict the output of CPRNG without knowing the initial seed and offset. There is no known correlation between variations of the seed. As in: the only thing partial knowledge of the seed gives you is reducing the propability of guessing the right seed. It's a similar situation to why the concept of entropy exhaustion doesn't really make sense. Joerg
Re: /dev/random is hot garbage
> On Jul 22, 2019, at 4:55 PM, Joerg Sonnenberger wrote: > > > [EXTERNAL EMAIL] > > On Mon, Jul 22, 2019 at 04:36:41PM +, paul.kon...@dell.com wrote: >> >> >>> On Jul 22, 2019, at 10:52 AM, Joerg Sonnenberger wrote: >>> >>> >>> [EXTERNAL EMAIL] >>> >>> On Sun, Jul 21, 2019 at 09:13:48PM +, paul.kon...@dell.com wrote: > On Jul 21, 2019, at 5:03 PM, Joerg Sonnenberger wrote: > > > [EXTERNAL EMAIL] > > On Sun, Jul 21, 2019 at 08:50:30PM +, paul.kon...@dell.com wrote: >> /dev/urandom is equivalent to /dev/random if there is adequate entropy, >> but it will also deliver random numbers not suitable for cryptography >> before that time. > > This is somewhat misleading. The problem is that with an unknown entropy > state, the system cannot ensure that an attacker couldn't predict the > seed used for the /dev/urandom stream. That doesn't mean that the stream > itself is bad. It will still pass any statistical test etc. That's exactly my point. If you're interested in a statistically high quality pseudo-random bit stream, /dev/urandom is a gread source. But if you need a cryptographically strong random number, then you can't safely proceed with an unknown entropy state for the reason you stated, which translates into "you must use /dev/random". >>> >>> That distinction makes no sense at all to me. /dev/urandom is *always* a >>> cryptographically strong RNG. The only difference here is that without >>> enough entropy during initialisation of the stream, you can brute force >>> the entropy state and see if you get a matching output stream based on >>> that seed. >> >> I use a different definition of "cryptographically strong". A bit string >> that's guessable is never, by any useful definition, "cryptographically >> strong" no matter what the properties of the string extender are. The >> only useful definition for the term I can see is as a synonym for >> "suitable for security critical value in cryptographic algorithms". >> An unseeded /dev/urandom output is not such a value. > > Again, that's not really a sensible definition. It's always possible to > guess the seed of used by the /dev/urandom CPRNG. By definition. That > doesn't change the core properties though: there is no sensible way to > predict the output of CPRNG without knowing the initial seed and offset. > There is no known correlation between variations of the seed. As in: the > only thing partial knowledge of the seed gives you is reducing the > propability of guessing the right seed. It's a similar situation to why > the concept of entropy exhaustion doesn't really make sense. I guess I didn't state the requirement for "cryptographically strong" clearly enough. A different but equivalent way of stating it: it must not be feasible, given the RNG output up to this point, to predict (with probability better than 50/50 guess) the future output. The /dev/urandom output is a function only of its internal state, for any span of time where no additional entropy is injected. So the requirement translates into: (a) you can't guess the internal state from the output, (b) you can't predict future output from past output without knowing the internal state. Ok, now suppose you have the output from /dev/urandom up to now, and only a few bits of entropy were injected since startup. Is the output strong? No, because the small number of entropy bits means that you can enumerate the possible combinations of entropy bits, construct the corresponding internal state, and generate trials output. The trial output string that matches what you observed tells you the internal state, and consequently the future output. This means that "enough entropy" translates to "so many bits of entropy that it is infeasible to guess the RNG internal state". For an example of how you can create a real world security defect by using an RNG with insufficient entropy in its seed, look up the Debian SSL bug (for example https://www.schneier.com/blog/archives/2008/05/random_number_b.html). paul
Re: /dev/random is hot garbage
On Mon, Jul 22, 2019 at 07:11:34PM +0200, Kamil Rytarowski wrote: > > It looks like we need a paravirt random driver for xen that could solve > the rust / random(6) problem. Or just run on a CPU that has RDRAND / RDSEED available. Our package builders are old; I'd chip in a few bucks to replace them, if that helped. -- Thor Lancelot Simon t...@panix.com "Whether or not there's hope for change is not the question. If you want to be a free person, you don't stand up for human rights because it will work, but because it is right." --Andrei Sakharov
Re: /dev/random is hot garbage
> On Jul 21, 2019, at 5:03 PM, Joerg Sonnenberger wrote: > > > [EXTERNAL EMAIL] > > On Sun, Jul 21, 2019 at 08:50:30PM +, paul.kon...@dell.com wrote: >> /dev/urandom is equivalent to /dev/random if there is adequate entropy, >> but it will also deliver random numbers not suitable for cryptography before >> that time. > > This is somewhat misleading. The problem is that with an unknown entropy > state, the system cannot ensure that an attacker couldn't predict the > seed used for the /dev/urandom stream. That doesn't mean that the stream > itself is bad. It will still pass any statistical test etc. That's exactly my point. If you're interested in a statistically high quality pseudo-random bit stream, /dev/urandom is a gread source. But if you need a cryptographically strong random number, then you can't safely proceed with an unknown entropy state for the reason you stated, which translates into "you must use /dev/random". > Note that with the option of seeding the CPRNG at boot time, a lot of > the distinction is actually moot. Yes, if at boot time you get enough entropy then /dev/random is unblocked. The distinction still matters because an application can't know this, so it should express its requirements by choosing the correct device. paul
