Missing ZFS info in 11.2 and 11.1 release notes
Hi everyone, Major changes to ZFS weren't documented in the 11.1 and 11.2 release notes. Can they be added retrospectively? Is there an easier way to find out what has changed than reading the SVN commits? Examples include significant changes to ZIL write behaviour in both 11.1 and 11.2 and the newly added ability to remove vdevs from a pool in 11.2. I'm hoping someone else has a neat list of changes on hand and that I don't have to read every commit (for there are many, some arcane). - Stephen. ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Data corruption in cd9660 on FreeBSD 4.11?
I haven't finished all the suggested tests, but since I'm taking so long to do so, I thought I should send what I have so far. On Saturday, 25th June 2005, Peter Jeremy wrote: On Fri, 2005-Jun-24 22:31:06 +1000, Stephen McKay wrote: I'm experiencing data corruption when reading CDs and DVDs on FreeBSD 4.11. ... So, can anyone suggest any more tests I could try? Or is there a kind of hardware fault that could cause this substitution of whole blocks read from CDs without causing any other problems? You might like to post the relevant sections of a verbose boot - the ATA and CD probes. I've appended it to this messages, so that the flow is not ruined. Note that I am not currently using ATAPI-CAM for my tests. I am using /dev/acd0a and /dev/acd1a to mount the CDs in the DVD-ROM and DVD-R respectively. Also the non-ATA66 cable thing is true; it is a plain ATA33 cable. Are you running the CD/DVD drives in PIO or UDMA modes? I normally run both DVD drives at UDMA33. My test runs normally fail every 2nd or 3rd run. I've seen it do 5 OK runs in a row once though, so I don't yet have a very good test. I tested with PIO4 and ran 12 consecutive test runs without error. It was a little slower at 150 seconds per run instead of the normal 135, possibly because 75% to 80% of the cpu was dedicated to interrupt handling (doing pio, I assume). It seems that either DMA or ATA interrupts (or maybe both) are required to cause the problem. Also, I tried some tests with the noclusterr mount option on the CD. The test ran much slower (approx 232 seconds instead of 135) but I also saw no failures (with only 6 test runs though as I was pressed for time). The noclusterr option is interesting because it defeats read clustering resulting in the ATA driver issuing only 2K reads instead of up to 64K at a time. I assume that the 64K reads would require scatter-gather DMA, so maybe this is relevant to the problem. Oddly, I noticed that a fixed value of 65534 is found in atapi-all.c as a request size limit. No, not 65536 = 2^16, but 2 bytes less. Puzzling. Have you tried anything other than ISO9660 filesystems on a physical CD? I have not tried anything but cd9660 file systems on CDs and DVDs. I will see if I can build a UFS file system to test with, when I get a chance. What happens if you just dd the CD-ROM? When I dd the CD-ROM it seems to work correctly. I have done this only infrequently however, so I may just be lucky to not have had a failure. I've now done 6 consecutive dd reads of my test CD-ROM in UDMA33 mode with no errors. It only takes 125 seconds, so it's a bit faster than comparing directory trees. Only 6 tests isn't many, so I'll do more later, this time with other system activity. What happens if you use a vnode mount (see vnconfig(8)) of an ISO filesystem sitting in a UFS filesystem? I'll test this when I get a chance. Anything unusual in your kernel config file? Nothing too unusual. I'm running a uni-processor kernel with HTT disabled. I skimmed through my config and this is the only interesting thing: HZ=500 I don't think that's too dangerous. On the other hand, it does increase the rate of interrupts, and if there's a race somewhere, it may make it worse. Have you tried building a kernel with WITNESS and/or DIAGNOSTIC? I'm now running with INVARIANTS, INVARIANT_SUPPORT, and DIAGNOSTIC on 4.11. No change in the failure rate and no significant slowdown either. Any chance of you repeating the tests with a 5.x system? Maybe on a spare small partition or using a 5.4-RELEASE disk1 as a live filesystem. I was experimenting with current in late April, so I installed that drive for testing. So far, I have not been able to reproduce the failure on April's current though I've only had time for a quick run of 6 repetitions. Current takes the same time (135 seconds, on average) to read and compare the CD. That seems good, considering all the debugging is still enabled. I'm pretty sure that ATA MK III is in this kernel. Sadly, it panics immediately if I run atacontrol mode 1 so I'm just assuming it is running in DMA mode by the speed of it. (And I have hw.ata.atapi_dma=1 in /boot/loader.conf). That's where I'm up to so far in stress testing. Right now I'm trying to understand some unusual looking code in ata_dmasetupd_cb() in 4.11's ata-dma.c. The attached comment is A maximum segment size was specified for bus_dma_tag_create, but some busdma code does not seem to honor this, so fix up if needed. The fix-up code seems to be gone in current, so it looks suspicious to me. When I work out what it does, I'll report back. Stephen. -- Verbose boot of 4.11-p10 (the ata related parts, at least): atapci0: Intel ICH5 ATA100 controller port 0xfc00-0xfc0f,0-0x3,0-0x7,0-0x3,0-0x7 irq 0 at device 31.1 on pci0 ata0: iobase=0x01f0 altiobase=0x03f6 bmaddr=0xfc00 ata0: mask=03 ostat0=50 ostat2=00 ata0-master: ATAPI 00 00 ata0
VFS_BIO_DEBUG and 4.11
Hi-diddly-ho! Is VFS_BIO_DEBUG still supposed to work in 4.11? I'm trying to debug a data corruption problem that could be a bug in the cd9660 file system and thought that enabling VFS_BIO_DEBUG might help. Instead it complains a lot about directories and character devices being VMIO'd nowadays, then panics with biodone: zero vnode ref count before it even finishes booting. I have reason to believe this was a useful flag back in 4.4 (because I saw a kernel config from Matt Dillon that included it), but have not found any evidence of use more recent than that. So, is it obsolete now? Or is it just only a little bit broken? I don't (yet) understand the invariants it is trying to enforce, so perhaps none of them apply any more. Stephen. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Data corruption in cd9660 on FreeBSD 4.11?
Hi! I'm experiencing data corruption when reading CDs and DVDs on FreeBSD 4.11. My best theory so far is that cd9660 or perhaps the VFS layer is mishandling 2048 byte buffers (since they are smaller than one virtual memory page), occasionally writing them to the wrong location in RAM. Read on for why I think so. First up, I don't think this is the usual hardware problem since the machine has done huge numbers of buildworlds (in 4.x and -current) without any of the telltale signs (eg bus errors and segmentation violations). There are no error messages in /var/log/messages. Also, it moonlights as a games machine and plays Doom 3, Battlefield 1942, Neverwinter Nights and so forth like a champ. Memory, cpu, video, disk, networking are all just fine 100% of the time. The hardware is an ASUS P4P800 mobo (including onboard Marvell Yukon gigabit ethernet) with a P4 2.8GHz cpu, 1GB RAM, Maxtor 120GB disk, Pioneer 103S DVD-ROM, LiteOn SOHW-1673S DVD burner in an Antec Sonata case. Now that I have a DVD burner, I make backups of my main machines (over NFS) but have found that they often don't verify as 100% correct. The symptom is that, for some files, an entire 2048 DVD sector is replaced with different (non-zero) data. This occurs both when reading with the Pioneer DVD-ROM and when reading with the LiteOn burner (though I don't test with the Pioneer much as it is slower). I emphasise that all burns have been 100% correct (ie the burning process worked and this can be verified by reading on, say, my iBook), so all of the hardware seems to be operating correctly (and swiftly, I might add). The problem is that reading the iso9660 file system is not safe. After some experimenting, I've found that the problem also occurs when reading CDs, and I built a test CD (of photos of a recent wedding) and in testing I read this CD over and over. I compare the CD with the original files (via NFS) using diff. When diff finds a difference, I save copies of the differing files before they can be flushed from the cache. I have calculated checksums for all 2048 blocks on the CD, so I can know if any given block of 2048 bytes came from the CD and if so which file it came from. In all cases so far, the 2048 byte error has been a block from another file, not a random corruption. I am starting to believe that, under high load, the cd9660 file system code tells the ata driver to put a 2K block in the wrong spot in memory, leaving some old junk in the gap in the file being read, and blasting some other 2K block of memory. It may not be cd9660 code per se that is wrong, but a bug in the complex buffer handling code (getblk, getnewbuf, allocbuf, etc). Why do I believe it is writing to the wrong memory, rather than any number of other flaws? In two runs (out of many), unusual things occurred that are consistent with memory being overwritten, rather than, say, a 2K block just not being read at all: In one, an innocent sshd core-dumped (which is something that has never happened except when running my cd9660 tests), and in another, a previously OK cached NFS file became corrupted. Explaining that last case further: I had been running a test script that would mount the CD, compare files, unmount the CD, and repeat. This meant that the NFS copy of the files was read over and over and hence became memory resident (there being enough space in 1GB of RAM for one copy of the files, plus my normal programs). Several tests passed without fault (hence all the NFS files were cached and correct), when suddenly there were multiple corruptions; call them file A and file B. File A was the usual corruption where a 2K block of another file was unexpectedly present in the copy read from the CD, but in file B it was the NFS file that was wrong. In fact it contained the missing block from file A! In short, the fully memory resident NFS file B had been corrupted by reading file A from the CD. It's been pretty interesting hunting this problem, but now I'm sort of stuck. I believe that some 2K reads from DVDs and CDs end up in the wrong place in RAM, but I can't find where this happens in the code (it's pretty hard to work out just by reading it), and I can't rule out the possibility that there's a hardware error here that I've just never run across before. So, can anyone suggest any more tests I could try? Or is there a kind of hardware fault that could cause this substitution of whole blocks read from CDs without causing any other problems? And does anyone know of any commits made anywhere in the 5 years since 4.x split off from 5.x that may be relevant? Yep. 5 years. I have started looking, but there's a fair bit of stuff in there... Stephen. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: nvi dying with Resource temporarily unavailable [SOLVED]
This is resurrecting an old thread, but I'd like the answer to be found in searches, so here goes: On Monday, 25th August 2003, Stephen McKay wrote: On Saturday, 9th August 2003, Doug White wrote: On Fri, 8 Aug 2003, Stephen McKay wrote: Stephen McKay wrote: Since I upgraded to FreeBSD 4.8 (from 4.5) I've noticed occasional failures of nvi. It will suddenly die as a key is pressed, emitting: Error: input: Resource temporarily unavailable We went round and round on irc about this a few weeks back. We pinned it down to a bad error check in nvi. Unfortunately the fix was non-obvious. There's a read() that needs to check for EAGAIN and loop back around on the read. If someone wants to take a crack at this, the offending read() is at common/cl_read.c line 266. You almost had me convinced until I got this extraordinary result: $ cat cat: stdin: Resource temporarily unavailable $ Never seen the like in all my born days. I'm running zsh 3.1.9 on FreeBSD 4.8-RELEASE. /bin/cat is a simple program. If it isn't working properly, there's a fault in zsh or the kernel. I have not upgraded zsh since July 2000. That makes it a bug in the kernel. What else could it be? At a stretch perhaps a bug in libc. Nothing else comes to mind. The fault lies in libc_r. I don't yet know how to fix libc_r, or even if it will ever be fixed, but I have installed a protective mechanism on my system that I am willing to live with. One of the things that libc_r does is a trick to allow I/O in one thread to not block other threads: it sets all file descriptors to nonblocking, including 0, 1 and 2. Descriptors 0, 1 and 2 normally refer to your tty (usually a pty nowadays) unless you go to the trouble of redirecting them. A further twist is that these descriptors are not the result of reopening your tty, but come from dup(), and hence share the underlying file flags with all other processes in that session, including your shell and any processes it starts before or after the one that uses libc_r. Setting nonblocking mode on a shared descriptor like this affects *all* processes using it. In other words, your shell, nvi, cat, and indeed all other programs on that tty now have a nonblocking descriptor for it. Having stdin or stdout suddenly become non-blocking causes many programs to fail mysteriously. In short, just running a program linked against libc_r in the background can cause other programs to fail. This is clearly unacceptable. It has taken me quite some years to track this down and it has almost made me lose faith in FreeBSD. (Why would anyone use an OS that fails randomly?) It's especially illuminating (from a programming point of view) that the root cause is in a subsystem I've never used and hence never examined all those times I went looking for the problem. It's the other programmers that have been using libc_r in more and more programs (some which I use without even knowing it) that has caused this slow degradation of my FreeBSD experience. How many other bugs like this are hidden in the ever increasing complexity of FreeBSD (or indeed any other software)? Unexpected interactions are everywhere and we should work hard to minimise them! OK, enough of the rambling philosophy: How can this be prevented? As described in this posting: http://lists.freebsd.org/mailman/htdig/freebsd-hackers/2005-January/009742.html I have added code to my 4.11 kernel to prevent background processes from setting O_NONBLOCK on ttys. I've been running with this for over 3 months and in that time have had no unexpected nvi exits or other weirdness. I believe this is a cure. I also believe that no process can reasonably expect to set O_NONBLOCK on its tty when in the background and hence I think this should be added to -current. But the side effect of the cure is that you cannot start a threaded program in the background without redirecting stdin, stdout and stderr elsewhere. I accept this as a cost of fixing the problem. You may not be so generous. If so, perhaps you can think of a way of fixing libc_r directly. Personally, I'd be happy enough to prevent the damage (by banning background O_NONBLOCK on ttys) while waiting for libc_r to die a natural death as the other threading libraries in 5.x and 6.x take over. Stephen. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Should I merge fix for PR#64091 (NFS data corruption)?
On Monday, 19th April 2004, Stephen McKay wrote: On Sunday, 18th April 2004, Eugene Grosbein wrote: Should one patch (and reboot in case of NFS compiled in a kernel) both of a client and a server? It is a client-side patch. The server could be anything, though I've only tested with FreeBSD. By the way, Eugen, how do I get on your whitelist? Your challenge response spam protection forgot to do the challenge bit! Stephen. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: nvi dying with Resource temporarily unavailable
On Thursday, 7th August 2003, Chuck Swiger wrote: Stephen McKay wrote: Since I upgraded to FreeBSD 4.8 (from 4.5) I've noticed occasional failures of nvi. It will suddenly die as a key is pressed, emitting: Error: input: Resource temporarily unavailable and you are staring at your command prompt. I googled for this message and found that people here and there have been complaining of this since about FreeBSD 4.2, but that absolutely none of them have received a reply. Are you using bash as your shell, by any chance? Zsh, actually. Long live zsh! :-) I suppose zsh could be interfering with the pty, but this failure happens in the middle of an editing session, not at the start. I assumed some sort of race condition, but couldn't (easily) find anything in the nvi code. Stephen. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: nvi dying with Resource temporarily unavailable
On Thursday, 7th August 2003, Ruslan Ermilov wrote: On Thu, Aug 07, 2003 at 10:30:22PM +1000, Stephen McKay wrote: Since I upgraded to FreeBSD 4.8 (from 4.5) I've noticed occasional failures of nvi. It will suddenly die as a key is pressed, emitting: Error: input: Resource temporarily unavailable and you are staring at your command prompt. Any ideas? stty sane Eh? Which tty flag (out of the many affected by stty sane) do you believe causes my problem? Why would my pty be insane in the first place? Should nvi be defensive and handle this case? Stephen. ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Strangeness with 4.0-S
Alan Edmonds wrote: Bill Paul wrote: (I'm starting to regret putting them in there in the first place. Here I was thinking I was being nice to people by letting them know what the driver is doing, and everybody panics the moment they see them.) Maybe prefix the warning with "info" or "warning" and/or some message like "Don't panic!" Also, would it help to see "increasing TX threshold to %d" in the message? I appreciate Bill's intent here to inform the public, and I usually code the same way. But I've come to the conclusion that it just scares people, and isn't beneficial after the shake out in -current. Still, I have an alternative suggestion to just ripping out the messages. I believe the most useful option would be to make the default to always store and forward, and allow an option (config file option, or some ifconfig command, sysctl, or some such) to enable the start-before-you've-got-the-data method. I don't think any normal user would see the speed difference. No one would see those messages soon after every boot. And best of all, we wouldn't see the connection hang for several seconds each time that message rolled by. It annoyed me so much I hacked my copy to not do any fancy stuff, and just go store and forward. I still get 10MB/s ftp between boxes. I think it's a clear winner. Stephen. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
Re: installworld : problem 1 (softupdates)
On Monday, 23rd August 1999, Sheldon Hearn wrote: On Fri, 20 Aug 1999 14:54:41 MST, "David O'Brien" wrote: Softupdates doesn't buy you anything on reads -- just writes. So unless you have it turned on for / because /tmp is on /, you aren't getting anything out of softupdates than frustration. Does softupdates score you something over MFS for /tmp ? Possibly. Because MFS pretends to be a real disk you often have 2 copies of any block in memory, one for the MFS "disk", and one for the "in core" copy. You wouldn't have that problem with softupdates. But, softupdates uses memory to keep track of the before and after versions of things, so it's hard to tell if you would win. Try it. :-) By the way, I use MFS for /tmp and don't use softupdates for root. Stephen. To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-stable" in the body of the message
