Re: Lost file-system story
On Fri, Dec 9, 2011 at 8:43 PM, Greg A. Woods wrote: > At Fri, 9 Dec 2011 15:50:35 -0500, Donald Allen > wrote: > Subject: Re: Lost file-system story >> >> "does not guarantee to keep a consistent file system structure on the >> disk" is what I expected from NetBSD. From what I've been told in this >> discussion, NetBSD pretty much guarantees that if you use async and >> the system crashes, you *will* lose the filesystem if there's been any >> writing to it for an arbitrarily long period of time, since apparently >> meta-data for async filesystems doesn't get written as a matter of >> course. > > I'm not sure what the difference is. You would be sure if you'd read my posts carefully. The difference is whether the probability of an async-mounted filesystem is near zero or near one. You seem to be quibbling over > minor differences and perhaps one-off experiences. Having a crash almost certainly destroy your filesystem vs. having the filesystem almost certainly survive a crash is not a minor difference. Both OpenBSD and > NetBSD also say that you should not use the "async" flag unless you are > prepared to recreate the file system from scratch if your system > crashes. That means use newfs(8) [and, by implication, something like > restore(8)], not fsck(8), to recover after a crash. You got lucky with > your test on OpenBSD. > > >> And then there's the matter of NetBSD fsck apparently not >> really being designed to cope with the mess left on the disk after >> such a crash. Please correct me if I've misinterpreted what's been >> said here (there have been a few different stories told, so I'm trying >> to compute the mean). > > That's been true of Unix (and many unix-like) filesystems and their > fsck(8) commands since the beginning of Unix. > > fsck(8) is designed to rely on the possible states of on-disk filesystem > metadata because that's now Unix-based filesystems have been guaranteed > to work (barring use of MNT_ASYNC, obviously). > > And that's why by default, and by very strong recommendation, filesystem > metadata for Unix-based filesystems (sans WABPL) should always be > written synchronously to the disk if you ever hope to even try to use > fsck(8). That's simply not true. Have you ever used Linux in all the years that ext2 was the predominant filesystem? ext2 filesystems were routinely mounted async for many years; everything -- data, meta-data -- was written asynchronously with no regard to ordering. And yet, when those systems crashed, fsck generally, not always, but usually, restored the filesystem to working order. Of course, some data could be lost and was, but you rarely suffered the loss of an entire filesystem. That's a fact. > > >> I am not telling the OpenBSD story to rub NetBSD peoples' noses in it. >> I'm simply pointing out that that system appears to be an example of >> ffs doing what I thought it did and what I know ext2 and journal-less >> ext4 do -- do a very good job of putting the world into operating >> order (without offering an impossible guarantee to do so) after a >> crash when async is used, after having been told that ffs and its fsck >> were not designed to do this. > > You seem to be very confused about what MNT_ASYNC is and is not. :-) No, you don't understand what I've said. > > Unix filesystems, including Berkeley Fast File System variant, have > never made any guarantees about the recoverability of an async-mounted > filesystem after a crash. I never thought or asserted otherwise. > > You seem to have inferred some impossible capability based on your > experience with other non-Unix filesystems that have a completely > different internal structure and implementation from the Unix-based > filesystems in NetBSD. Nonsense -- I have inferred no such thing. Instead of referring you to previous posts for a re-read, I'll give you a little summary. I am speaking about probabilities. I completely understand that no filesystem mounted async (or any other way, for that matter), whether Linux or NetBSD or OpenBSD, is GUARANTEED to survive a crash. The probability of surviving a crash for any of them is < 1. But my experience with Linux ext2 over many years has been that the probability of survival is quite high, near 1. When I reported my experience with NetBSD ffs in this thread, I expressed surprise that the filesystem was a total loss, based on what preceded the crash. My surprise was a result of years of Linux experience. I then got some responses -- see the one from Thor Lancelot Simon, for example. In that message, he asserts that, in NetBSD, *nothing* pushes meta-data to the disk for a filesystem mounted async. Others said some contradictory things about that and I'm not sure what the truth is, but if Simon is right, then the probability of crash survival in NetBSD is indeed near zero. Another point that was made was that NetBSD ffs fsck was not designed to put a damaged filesystem back together, at least the kind of damage one might encounter with a
Re: Lost file-system story
On Fri, 9 Dec 2011 22:12:25 -0500 Donald Allen wrote: > Linux systems do periodically write ext2 meta-data to the disk. And > ext2 fsck has always been very good, and has gotten better over the > years, due to the efforts of Ted T'so. I first installed Linux in > 1993, almost 20 years ago, and have been using it continuously ever > since. I have *never* lost an ext2 filesystem and I've never mounted > one sync. I'm not sure if it's the case on Linux with ext2, but by default NetBSD FFS mounts are not sync, nor async; metadata is sync and data blocks are async. In async mode, all data is asyncronously written, including the metadata, and in sync mode everything is written synchronously (the default OpenBSD uses, if I recall). I just wanted to specify this as you mentioned not mounting your ext2 systems in sync mode, but a default NetBSD FFS mount will not be in sync mode either. Other available options with FFS are using soft dependencies (softdep) or WAPBL metadata journalling (log), with which it is possible to have increased performance VS the default mode, without really sacrificing reliability, unlike with the fully async mode. In those modes, metadata is written asynchroneously as well. Sorry if what I said is already obvious to you, -- Matt
Debian OpenSSL desaster (was: Patch: new random pseudodevice)
[I tried to send this as private mail, but get host Sparkle-4.Rodents-Montreal.ORG[216.46.5.7] refused to talk to me: 550-.de's whois server, whois.denic.de, is completely broken, handing 550-out no contact information at all when queried for .de domains in 550 the usual way. Such a domain has no place on a civilized network. I don't know what this is about and why it hinders anybody from accepting my email. I also don't know whether whoever put this in place assumes me to be in a position to influence Denic's behaviour. Or suggests me moving to another country where my employer doesn't reside under the .de domain. Or me using another special email address for communicating with people in his domain.] > I don't recall full details, but I think it was a Linux distro It was the Debian OpenSSL desaster. In essence, they patched OpenSSL's entropy gathering to the point where the PID was the only entropy source being used. So it generated as many different private keys as there were PIDs it could run under, which is 2^16-n, n>1.
Re: Using event counters with network interfaces, is there a reason they're all ifdefed out of mainline use?
On Sat, Dec 10, 2011 at 05:04:41AM -0800, Brian Buhrow wrote: > Hello. I notice that most, if not all, of the network drivers in > NetBSD have interface counters which they use to track things like > collisions, CRC errors, framing errors, etc. It looks like these counters, You have these informations (with less granularity I admit) in netstat -i outtput. -- Manuel Bouyer NetBSD: 26 ans d'experience feront toujours la difference --
Re: Lost file-system story
On Tue, Dec 06, 2011 at 11:58:25AM -0500, Thor Lancelot Simon wrote: > With the filesystem mounted async *nothing* pushes out most > metadata updates, If this is really true, it's a bug and should be fixed. -- David A. Holland dholl...@netbsd.org
Fwd: Lost file-system story
I should have sent this to the mailing list as well as David. Google has fixed something that wasn't broke -- gmail. They've introduced a new UI that I haven't gotten used to yet ... -- Forwarded message -- From: Donald Allen Date: Sun, Dec 11, 2011 at 10:23 AM Subject: Re: Lost file-system story To: David Holland On Sun, Dec 11, 2011 at 8:57 AM, David Holland wrote: > On Tue, Dec 06, 2011 at 11:58:25AM -0500, Thor Lancelot Simon wrote: > > With the filesystem mounted async *nothing* pushes out most > > metadata updates, > > If this is really true, it's a bug and should be fixed. It may very well be true. I just did the following: I brought up my test laptop, running 5.1 GENERIC, with /home mounted async,noatime. I created a new file in my home directory. I should note that when I ZZ'ed out of vi, the disk light flashed momentarily, and I could hear the disk doing something. I did an ls -lt | head and the new file was there. I waited just under a minute (to let syncs happen; this is longer than any of the sysctl vfs.sync.delays, which I assume are in seconds; the man page doesn't say) and then I pulled the plug (no battery in the machine). On restart, I got no fsck errors, but the new file was not in my home directory. I then repeated this test, waiting a little over a minute this time. Same result, the new file was gone (this time I got fsck errors). Then I did the test a third time, but this time I did a sync before pulling the plug. On restart, I still got some fsck errors that were fixed, but the new file was present. This does suggest that the meta-data is not being written, at least within a minute or so of creating a new file. One thing I think we have not discussed much or at all in this thread is that the filesystem surviving a crash and how much data you lose when it does survive are separate issues. The experiments I did yesterday demonstrate that a NetBSD ffs async-mounted filesystem, together with its fsck, *can* survive a crash in bad circumstances -- lots of write activity at the time of the crash. We don't know what the probability of survival is, just that it's > 0. What I did yesterday also does not address loss of data. If Simon is correct and NetBSD is just not writing metadata until sync is explicitly called, then you could have a system up for days or weeks and lose as many as all of the files created in an async filesystem since the last re-boot. We don't know definitively what it's doing yet, but I think I've demonstrated that it's not writing meta-data within one minute windows. I will do some more playing with this, waiting longer and will report what I find. We also know from this morning's tests that a user-called sync does get the meta-data written, reducing the amount of data lost in a crash that the filesystem survives. So those who advocated periodically calling sync in a loop (Christos first suggested this to me in a private email) are correct -- it's necessary if you are going to use async mounting. More later ... /Don > > -- > David A. Holland > dholl...@netbsd.org
Re: Lost file-system story
On Sun, Dec 11, 2011 at 10:25 AM, Donald Allen wrote: > I should have sent this to the mailing list as well as David. Google > has fixed something that wasn't broke -- gmail. They've introduced a > new UI that I haven't gotten used to yet ... > > > -- Forwarded message -- > From: Donald Allen > Date: Sun, Dec 11, 2011 at 10:23 AM > Subject: Re: Lost file-system story > To: David Holland > > > On Sun, Dec 11, 2011 at 8:57 AM, David Holland > wrote: >> On Tue, Dec 06, 2011 at 11:58:25AM -0500, Thor Lancelot Simon wrote: >> > With the filesystem mounted async *nothing* pushes out most >> > metadata updates, >> >> If this is really true, it's a bug and should be fixed. > > It may very well be true. I just did the following: > > I brought up my test laptop, running 5.1 GENERIC, with /home mounted > async,noatime. I created a new file in my home directory. I should > note that when I ZZ'ed out of vi, the disk light flashed momentarily, > and I could hear the disk doing something. I did an ls -lt | head and > the new file was there. I waited just under a minute (to let syncs > happen; this is longer than any of the sysctl vfs.sync.delays, which I > assume are in seconds; the man page doesn't say) and then I pulled the > plug (no battery in the machine). On restart, I got no fsck errors, > but the new file was not in my home directory. I then repeated this > test, waiting a little over a minute this time. Same result, the new > file was gone (this time I got fsck errors). Then I did the test a > third time, but this time I did a sync before pulling the plug. On > restart, I still got some fsck errors that were fixed, but the new > file was present. > > This does suggest that the meta-data is not being written, at least > within a minute or so of creating a new file. > > One thing I think we have not discussed much or at all in this thread > is that the filesystem surviving a crash and how much data you lose > when it does survive are separate issues. The experiments I did > yesterday demonstrate that a NetBSD ffs async-mounted filesystem, > together with its fsck, *can* survive a crash in bad circumstances -- > lots of write activity at the time of the crash. We don't know what > the probability of survival is, just that it's > 0. > > What I did yesterday also does not address loss of data. If Simon is > correct and NetBSD is just not writing metadata until sync is > explicitly called, then you could have a system up for days or weeks > and lose as many as all of the files created in an async filesystem > since the last re-boot. We don't know definitively what it's doing > yet, but I think I've demonstrated that it's not writing meta-data > within one minute windows. I will do some more playing with this, > waiting longer and will report what I find. We also know from this > morning's tests that a user-called sync does get the meta-data > written, reducing the amount of data lost in a crash that the > filesystem survives. So those who advocated periodically calling sync > in a loop (Christos first suggested this to me in a private email) are > correct -- it's necessary if you are going to use async mounting. I repeated the test without the sync, but waited 15 minutes after creating the new file before killing the power. When the system came up, I got fsck errors that were fixed, and the new file I created 15 minutes before pulling the plug was not present. Whether this is intentional or a bug, I agree with David Holland -- it's wrong and should be fixed. /Don > > More later ... > > /Don > > >> >> -- >> David A. Holland >> dholl...@netbsd.org
Re: Debian OpenSSL desaster (was: Patch: new random pseudodevice)
> [I tried to send this as private mail, but get
> host Sparkle-4.Rodents-Montreal.ORG[216.46.5.7] refused to talk to me:
> 550-.de's whois server, whois.denic.de, is completely broken, [...]
I wrote up a point-by-point reply to this, but then realized, this is
tech-kern, not tech-broken-network-governance. So I'll confine myself
to saying my respnse is at
{ftp,http}://ftp.rodents-montreal.org/mouse/ccTLD-thoughts.txt for
anyone interested. (Actually, will be at; as I send this mail, I'm
still writing it - the draft is available at
.../ccTLD-thoughts-draft.txt and I'll move it when I'm done.)
As for the content...
>> I don't recall full details, but I think it was a Linux distro
> It was the Debian OpenSSL desaster. In essence, they patched
> OpenSSL's entropy gathering to the point where the PID was the only
> entropy source being used.
Ah. Yeah, that'll do it. Thanks for the correction; I'm not surprised
I got some of the details wrong - but the actual incident works just as
well for the argument I was making with it.
/~\ The ASCII Mouse
\ / Ribbon Campaign
X Against HTMLmo...@rodents-montreal.org
/ \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Lost file-system story
On Sun, Dec 11, 2011 at 10:50:29AM -0500, Donald Allen wrote: > I repeated the test without the sync, but waited 15 minutes after > creating the new file before killing the power. When the system came > up, I got fsck errors that were fixed, and the new file I created 15 > minutes before pulling the plug was not present. Whether this is > intentional or a bug, I agree with David Holland -- it's wrong and > should be fixed. I disagree. It is exactly why I use FFS with -o async -- to get a disk backed storage, that doesn't waste resources, if everything fits into memory, but falls gracefully otherwise. Joerg
Re: Lost file-system story
On Sun, Dec 11, 2011 at 11:04 AM, Joerg Sonnenberger wrote: > On Sun, Dec 11, 2011 at 10:50:29AM -0500, Donald Allen wrote: >> I repeated the test without the sync, but waited 15 minutes after >> creating the new file before killing the power. When the system came >> up, I got fsck errors that were fixed, and the new file I created 15 >> minutes before pulling the plug was not present. Whether this is >> intentional or a bug, I agree with David Holland -- it's wrong and >> should be fixed. > > I disagree. It is exactly why I use FFS with -o async -- to get a disk > backed storage, that doesn't waste resources, if everything fits into > memory, but falls gracefully otherwise. Certainly a valid requirement, but we haven't talked about what the fix should be. I think it should have an adjustable sync frequency, so that the user can turn a knob from "I want to lose as little as possible" to "I want maximum performance". If I get my wish, you can use the latter, which should set the frequency to zero. /Don > > Joerg
Re: Lost file-system story
On Sun, Dec 11, 2011 at 11:32:51AM -0500, Donald Allen wrote: > On Sun, Dec 11, 2011 at 11:04 AM, Joerg Sonnenberger > wrote: > > On Sun, Dec 11, 2011 at 10:50:29AM -0500, Donald Allen wrote: > >> I repeated the test without the sync, but waited 15 minutes after > >> creating the new file before killing the power. When the system came > >> up, I got fsck errors that were fixed, and the new file I created 15 > >> minutes before pulling the plug was not present. Whether this is > >> intentional or a bug, I agree with David Holland -- it's wrong and > >> should be fixed. > > > > I disagree. It is exactly why I use FFS with -o async -- to get a disk > > backed storage, that doesn't waste resources, if everything fits into > > memory, but falls gracefully otherwise. > > Certainly a valid requirement, but we haven't talked about what the > fix should be. I think it should have an adjustable sync frequency, so > that the user can turn a knob from "I want to lose as little as > possible" to "I want maximum performance". If I get my wish, you can > use the latter, which should set the frequency to zero. I don't see the point. Out of order meta updates can fry the file system at any point. Really, just don't use them if you can't recreate the file system freely. As has been mentioned elsewhere in the thread, the default mount option is *not* async. Joerg
Re: Lost file-system story
On Sun, Dec 11, 2011 at 11:44 AM, Joerg Sonnenberger wrote: > On Sun, Dec 11, 2011 at 11:32:51AM -0500, Donald Allen wrote: >> On Sun, Dec 11, 2011 at 11:04 AM, Joerg Sonnenberger >> wrote: >> > On Sun, Dec 11, 2011 at 10:50:29AM -0500, Donald Allen wrote: >> >> I repeated the test without the sync, but waited 15 minutes after >> >> creating the new file before killing the power. When the system came >> >> up, I got fsck errors that were fixed, and the new file I created 15 >> >> minutes before pulling the plug was not present. Whether this is >> >> intentional or a bug, I agree with David Holland -- it's wrong and >> >> should be fixed. >> > >> > I disagree. It is exactly why I use FFS with -o async -- to get a disk >> > backed storage, that doesn't waste resources, if everything fits into >> > memory, but falls gracefully otherwise. >> >> Certainly a valid requirement, but we haven't talked about what the >> fix should be. I think it should have an adjustable sync frequency, so >> that the user can turn a knob from "I want to lose as little as >> possible" to "I want maximum performance". If I get my wish, you can >> use the latter, which should set the frequency to zero. > > I don't see the point. Out of order meta updates can fry the file system > at any point. Really, just don't use them if you can't recreate the > file system freely. As has been mentioned elsewhere in the thread, the > default mount option is *not* async. Yes, they *can* destroy the filesystem, but in Linux ext2, they rarely do (see what I've said about this in previous messages in this thread), and I've started, in a small way, to build a case for NetBSD ffs and its fsck also having a reasonable probability of surviving a crash (what really matters is the joint probability of crashing -- very low in the case of Linux over the years -- *and* losing the filesystem on restart). As for the knob, it probably doesn't make sense to mount a filesystem async and then set the knob to sync every 50 milliseconds. One isn't going to get much of a performance benefit in return for incurring the risk of async mounting (I would guess that the risk goes down as the sync frequency goes up, but doesn't go to zero). If safety is one's orientation, it would probably be better to mount default, sync, or softdep, or use the new journaling option. But sync'ing every 5 minutes or 10 minutes might well give one the performance benefit that brought async to consideration in the first place, while likely limiting lost work to a 5- or 10-minute window. I say "likely", because I emphasize again, for the umpteenth time in this discussion, that I completely understand that async incurs the risk of losing the whole filesystem. But if NetBSD/ffs/fsck turns out to exhibit the same behavior as Linux/ext2 has exhibited for years, the joint probability of crashing and incurring that loss is extremely low. And if it happens, I can and will deal with that. As an example, the machine I'm typing this on is running 5.1 with an /etc/fstab that looks like this: # NetBSD /etc/fstab # See /usr/share/examples/fstab/ for more examples. /dev/wd0a / ffs rw,noatime 1 1 /dev/wd0b noneswapsw,dp 0 0 /dev/wd0e /usrffs rw,noatime 1 2 /dev/wd0f /varffs rw,noatime 1 2 /dev/wd0g /home ffs rw,noatime,async1 2 /dev/wd0b /tmpmfs rw,-s=205632 kernfs /kern kernfs rw ptyfs /dev/ptsptyfs rw procfs /proc procfs rw /dev/cd0a /cdrom cd9660 ro,noauto So everything has the default mounting+noatime except /home, which is noatime,async. I routinely rsync my home directory among my many machines, so I've got N very up-to-date backups. If I lose /home, not that big a deal. But if the system crashes and the filesystem is recovered, I'd like to have the option to make it a smaller deal still, and be able to define a maximum-loss window, something smaller than the min(time since last normal reboot, time since last rsync). /Don > > Joerg
Re: Debian OpenSSL desaster (was: Patch: new random pseudodevice)
On Sun, Dec 11, 2011 at 12:35:07PM +0100, Edgar Fu? wrote: > [I tried to send this as private mail, but get > > host Sparkle-4.Rodents-Montreal.ORG[216.46.5.7] refused to talk to me: > 550-.de's whois server, whois.denic.de, is completely broken, handing > 550-out no contact information at all when queried for .de domains in > 550 the usual way. Such a domain has no place on a civilized network. > > I don't know what this is about and why it hinders anybody from > accepting my email. > > I also don't know whether whoever put this in place assumes me to > be in a position to influence Denic's behaviour. Or suggests me > moving to another country where my employer doesn't reside under > the .de domain. Or me using another special email address for > communicating with people in his domain.] The short answer is that Mouse likes tilting at windmills. :-) -- David A. Holland dholl...@netbsd.org
Re: Using event counters with network interfaces, is there a reason they're all ifdefed out of mainline use?
hello. I'm interested in seeing detailed errors like CRC, frame, underrun and overrun error counts. Further investigation reveals that drivers that came later in the game, i.e. nfe(4) and age(4) don't have latent support for these counters, but it looks fairly easily added. While I understand performance considerations are necessary, the counters I'm proposing to add should be fairly low frequency events, since we already count bytes and packets in and out, and if the performance of the event counters infrastructure were really a problem, we should be taking a hit because many of the interrupt counters are using it to keep their statistics. While I don't aspire to be like Linux, Linux does report these kinds of errors, as do most modern networking devices. I think we should hav the capability of tracking such errors, and I think we can for little effort and almost no performance loss. Since there's been some nervousness expressed to me about performance hits, I propose to do one driver at a time, do some testing of mine, then commit those changes to the tree. We can either set the COUNT_INTERFACE_EVENTS option to true at that time, or have a self selected group do some early testing. However, it is my wish to have this higher granularity of error tracking turned on by default in future releases of NetBSD. -thanks -Brian On Dec 11, 1:18pm, Manuel Bouyer wrote: } Subject: Re: Using event counters with network interfaces, is there a reas } On Sat, Dec 10, 2011 at 05:04:41AM -0800, Brian Buhrow wrote: } > Hello. I notice that most, if not all, of the network drivers in } > NetBSD have interface counters which they use to track things like } > collisions, CRC errors, framing errors, etc. It looks like these counters, } } You have these informations (with less granularity I admit) in } netstat -i outtput. } } -- } Manuel Bouyer } NetBSD: 26 ans d'experience feront toujours la difference } -- >-- End of excerpt from Manuel Bouyer
ccTLD filtering (was: Debian OpenSSL desaster)
> So I'll confine myself to saying my respnse is at
> {ftp,http}://ftp.rodents-montreal.org/mouse/ccTLD-thoughts.txt
http://www.math.uni-bonn.de/people/ef/ccTLD-blocking-thoughts.txt
In short: I don't see what's wrong with Denic's whois, its my job to run
math.uni-bonn.de (which I do on NetBSD servers), so I can't choose my domain.
Feel free to reply be email, I both accept and read your mail.
Re: Debian OpenSSL desaster (was: Patch: new random pseudodevice)
>> [...] > The short answer is that Mouse likes tilting at windmills. :-) Eh. I think that is at least a little of a misstatement. I don't do such things because I enjoy doing them. Quite the opposite. I do them because I must. I'm not entirely sure what I mean by that. It's difficult to explain, even to myself. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
Re: Debian OpenSSL desaster (was: Patch: new random pseudodevice)
On Sun, Dec 11, 2011 at 02:52:28PM -0500, Mouse wrote: > >> [...] > > The short answer is that Mouse likes tilting at windmills. :-) > > Eh. I think that is at least a little of a misstatement. I don't do > such things because I enjoy doing them. Quite the opposite. > > I do them because I must. > > I'm not entirely sure what I mean by that. It's difficult to explain, > even to myself. I think you're strengthening my description... -- David A. Holland dholl...@netbsd.org
Re: Patch: new random pseudodevice
On Fri, Dec 09, 2011 at 02:41:25PM -0500, Paul Koning wrote: > ... That's essentially what old time Ethernet chips like Lance did IIRC The lance's CSMACD backoff was deterministic, if you were really unlucky two systems could collide packets for ever! (On a network with only 2 hosts.) On a separate note, I've sometimes wondered how well RC4 would work as an entropy pool. Just cycle it in key-gen mode for any byte that might have any randomness in it. When you want data, cycle it a few times, take some bytes hash them (to remove any RC4-ness), and cycle it a few more times. Probably the same idea as Fortuna is using. Adding a small amount of randomness should impove things significantly - and you don't actual care if it is random. David -- David Laight: da...@l8s.co.uk
Re: Lost file-system story
>> More later ... I installed OpenBSD 5.0 on the same machine, similar setup (all filesystems noatime except /tmp and /home, which are both async,noatime). I repeated my experiment -- wrote a new file in my home directory, waited a few minutes, and killed the power. On reboot, there were complaints from the fscks, async and not, all fixed. The system came up without a manual fsck and the new file was present in my directory. So meta-data for async filesystems is being written within a window of a handful of minutes with OpenBSD. /Don
Re: Patch: new random pseudodevice
On Dec 11, 2011, at 5:18 PM, David Laight wrote: > On Fri, Dec 09, 2011 at 02:41:25PM -0500, Paul Koning wrote: >> ... That's essentially what old time Ethernet chips like Lance did > > IIRC The lance's CSMACD backoff was deterministic, if you were > really unlucky two systems could collide packets for ever! > (On a network with only 2 hosts.) You're thinking of the capture effect. I think that had to do with deferral, not collision backoff. The collision RNG in Lance is definitely NOT deterministic, that was an absolute requirement. It used what was referred to as "flaky oscillators". > > On a separate note, I've sometimes wondered how well RC4 would work as > an entropy pool. Just cycle it in key-gen mode for any byte that might > have any randomness in it. > When you want data, cycle it a few times, take some bytes hash them > (to remove any RC4-ness), and cycle it a few more times. > Probably the same idea as Fortuna is using. > Adding a small amount of randomness should impove things significantly > - and you don't actual care if it is random. That's a pretty good description of the basic idea of a cryptographic PRNG. paul
Re: Lost file-system story
On Fri 09 Dec 2011 at 17:40:29 -0500, Donald Allen wrote: > If I can find the time, I'll do that. Even a little shell script would do: #!/bin/sh while sleep 30; do sync; done -Olaf. -- ___ Olaf 'Rhialto' Seibert -- There's no point being grown-up if you \X/ rhialto/at/xs4all.nl-- can't be childish sometimes. -The 4th Doctor
Re: Lost file-system story
On Sun, Dec 11, 2011 at 3:21 PM, Donald Allen wrote: >>> More later ... > > I installed OpenBSD 5.0 on the same machine, similar setup (all > filesystems noatime except /tmp and /home, which are both > async,noatime). I repeated my experiment -- wrote a new file in my > home directory, waited a few minutes, and killed the power. On reboot, > there were complaints from the fscks, async and not, all fixed. The > system came up without a manual fsck and the new file was present in > my directory. So meta-data for async filesystems is being written > within a window of a handful of minutes with OpenBSD. I haven't read every single word you've said about this subject, so I apologize if I'm missing something. I assume you're using async because you want better performance and you have some tolerance for data loss, otherwise this wouldn't even be a discussion I think. We're just talking about probabilities of data loss then, correct? For some people (I suspect, a few that have already answered), this isn't something they are willing to discuss, even though we all know it's impossible to get to "1" as you said. But you can get really close these days. If solving your problem depends on sync frequency, I don't see why this shouldn't be managed by some knob to twiddle. Given that the crash scenario doesn't get worse depending on where the knob is or if the crash happens while the knob is working. If it does, it's pointless. Why haven't other solutions been discussed? NetBSD supports ext2. And raid. And all kinds of other stuff. Why not use it? Andy
Re: Patch: new random pseudodevice
On 9 Dec, 2011, at 17:38 , Mouse wrote: > A PRNG cannot create information; its output contains no more > information than its input. If it is keyed with 128 bits, say, it > cannot produce more than 2^128 different output keystreams, even if > those keystreams are far longer, and thus appear to contain far more > information, than that. Unless you're using it for a purpose that > doesn't need more than 128 bits of information content, it is > unsuitable. (And if you don't need more than that, any good > cryptographic hash function will do just as well.) I agree with many of the things you are arguing, and I do only know enough about this to be dangerous, but I'm pretty sure the assertion you seem to keep making that if you use 128 bits of information to key a CSPRNG, or a cryptographic hash, or pretty much any other cryptographic device, then taking 128 bits of output from the device is fine but the 129th bit is somehow suspect, is misunderstanding how these things work and is not correct. If you key a CSPRNG with 128 truly random bits then you have 2^128 possible input states. For 128 bits of output from the CSPRNG to have the same amount of information as the input would require that the 128 bits of output have the same number of possible states, i.e. 2^128. For this to be true, however, would require that there be a one-to-one correspondence between each of the possible 2^128 inputs and each of the possible 2^128 outputs, and if that were true then the function would be trivially invertible since it is a linear transformation. Because cryptographic-anything needs to protect its input it can't work like this. The output of a CSPRNG (or cryptographic hash, for that matter) has less entropy per bit than its input because this is necessary to keep from trivially revealing its input. That is, if you take 128 bits of output from the 128-bit-salted CSPRNG some of the 2^128 possible 128 bit codes will be reliably missing from that output. Any particular 128 bits of output from the CSPRNG is less random than the 128 bits you put in. The argument that the 128 bits are indistinguishable from random instead relies on a computational expense argument: knowing that codes are missing does an attacker no good unless he knows which codes are missing, and if the only way to determine which codes are missing is by trying all 2^N possible inputs to see what is *not* missing then, for sufficiently large N, the computation may be infeasible. I think, then, that if you are comfortable taking 128 bits of output from a CSPRNG then there's no reason not to be comfortable taking a 129th (and 130th, and 131st and so on). The state of the 129th is only infinitesimally less unpredictable than the 128th bit since its state depends on input information that was not revealed in the first 128 bits of output (since the first 128 bits of output contain less than the 128 bits of "information" the CSPRNG was keyed with). If you are uncomfortable with the 129th bit, however, then you must also be uncomfortable with the 128th (and 127th and 126th) since they are only a little bit better than the 129th. There is no sharp cutoff at 128 bits. I assume there might be some sequence of less than 128 bits which would be as random as the 128 input bits, but good luck figuring out what that length would be. > This is why I'm hammering on the information-theoretic considerations: > they are fundamental, not subject to change with advances in > cryptanalysis. The security of bits drawn from an properly-designed > entropy pool depends on much weaker assumptions than the security of > bits produced by a PRNG (when the PRNG seed is smaller than the number > of bits produced). You keep saying the bit in parentheses, but I'm pretty sure that's wrong. A cryptographic function where an 'N' bit output was as random as the 'N' bit input would not be a cryptographic function, so if you are assuming that N->N bit transformations through cryptographic functions don't lose information you are about as reliant on those "weaker assumptions" as the code you are criticizing here, I think. Dennis Ferguson
Re: Lost file-system story
On Sun, Dec 11, 2011 at 05:04:23PM +0100, Joerg Sonnenberger wrote: > > I repeated the test without the sync, but waited 15 minutes after > > creating the new file before killing the power. When the system came > > up, I got fsck errors that were fixed, and the new file I created 15 > > minutes before pulling the plug was not present. Whether this is > > intentional or a bug, I agree with David Holland -- it's wrong and > > should be fixed. > > I disagree. It is exactly why I use FFS with -o async -- to get a disk > backed storage, that doesn't waste resources, if everything fits into > memory, but falls gracefully otherwise. That's as may be, but it's still wrong. The syncer should be writing out the metadata buffers as well as file data. (For your purpose, you'd want it to be writing out neither, btw.) Note the result from OpenBSD; we probably broke it with the UBC merge and never noticed. Don't we have at least one filesystem that doesn't support UBC? What happens to it? -- David A. Holland dholl...@netbsd.org
Re: Patch: new random pseudodevice
> [...] I'm pretty sure the assertion you seem to keep making that if > you use 128 bits of information to key a CSPRNG, or a cryptographic > hash, or pretty much any other cryptographic device, then taking 128 > bits of output from the device is fine but the 129th bit is somehow > suspect, I don't think that's quite what I've been saying. I've been saying that the output contains at most 128 bits of information, no matter how many bits it's spread among. This information is not necessarily all present in the first 128 bits of output, and, as you point out, if it is then it is a weakness in the CS part of the PRNG. > The argument that the 128 bits are indistinguishable from random > instead relies on a computational expense argument: Right. That's why I consider it a weak argument - attacks just get better. Once upon a time, a Vigenère cipher was computationally secure, in that nobody knew how to break it. > knowing that codes are missing does an attacker no good unless he > knows which codes are missing, and if the only way to determine which > codes are missing is by trying all 2^N possible inputs to see what is > *not* missing then, for sufficiently large N, the computation may be > infeasible. I see no reason to believe the first part. Cryptography is littered with cases where something general was known to be true but specifics were unknown and it was used as a point of attack nevertheless. *If* the only way to determine which codes are missing is exhaustive search, yes, it's infeasible. I see no particular reason to think that's so. Indeed, it is almost certainly not so; we just don't know of any significantly better way at present. >> The security of bits drawn from an properly-designed entropy pool >> depends on much weaker assumptions than the security of bits >> produced by a PRNG (when the PRNG seed is smaller than the number of >> bits produced). > You keep saying the bit in parentheses, but I'm pretty sure that's > wrong. Assuming, in both cases, that the input contains as much information as its bit count allows: I'm saying that a PRNG - or any other determinstic computation - that generates more bits of output than it's fed as input cannot output more information than it was given; the output cannot contain as much information as its bit count seems to imply. I am not saying that a PRNG, even a CSPRNG, that _isn't_ drawn on for more bits than it's keyed with _will_ contain as much information in its output as the bit count allows. I think the latter is what you are saying is wrong. (If so, I agree.) > A cryptographic function where an 'N' bit output was as random as the > 'N' bit input would not be a cryptographic function, What is a "cryptographic function", here? It's not a phrase I recognize as a technical term. /~\ The ASCII Mouse \ / Ribbon Campaign X Against HTMLmo...@rodents-montreal.org / \ Email! 7D C8 61 52 5D E7 2D 39 4E F1 31 3E E8 B3 27 4B
