Summary: 12TB GEOM stripe, newfs, then fsck: cannot alloc 768053748 bytes for blockmap
Thanks a lot to Mr. Zbyslaw. Being too lazy myself to fiddle with the soft limit via the shell or bootloader, I did the following in the kernel build config file: options MAXDSIZ=(1024UL*1024*1024) options DFLDSIZ=(1024UL*1024*1024) This way, fsck works just fine. To address other people's suggestions: I haven't tried to force the fsck under the old limit. A local friend has suggested to increase the block size to newfs, or something along those lines - essentially to decrease the FS overhead and the size of the blockmap. I haven't tried that but I guess it sounds reasonable - may make sense on machines where you just can't get enough RAM or are not willing to grant it all to a single process. Thanks to everyone who replied :-) Frank Rysanek On 8 Jun 2005 at 16:01, Alex Zbyslaw wrote: cannot alloc 768053748 bytes for blockmap Do you have MAXDSIZ set in your kernel? And what about any limits? e.g. I have options MAXDSIZ=(1024UL*1024*1024) and limit memoryuse unlimited ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: Summary: 12TB GEOM stripe, newfs, then fsck: cannot alloc 768053748 bytes for blockmap
On 9 Jun 2005 at 9:46, Alex Zbyslaw wrote: A local friend has suggested to increase the block size It ought also make sense if you are serving up *large* files (didn't you say video/audio?). ... I don't have anything on your scale (23Gb of document database pales into insignificance against 12Tb :-) ) I myself have never met that sort of data volume, either. I actually just work for a reseller / assembly shop. We needed to test these three units before shipping them. FreeBSD 5 is the only free UNIX known to me that handles 2TB out of the box on a 32bit x86 PC. (I didn't bother to install a 64bit OS on the dual Nocona box used for testing.) While I was at it, I thought I could give GEOM a try. Never used it before. Took me 5 minutes to find the example in the manpage. Then it took two commands in the shell and the block device was up. Unbelievable. Me being a FreeBSD illiterate. I seem to recall that the boxes will be used for medium-term video archival at their final destination, separately. Thanks again for your help :-) Frank Rysanek P.S.: It makes you wonder. A disk's transfer rate grows roughly with the square root of the disk's capacity (== storage density), if other conditions are unchanged (RPM, number of heads etc.) It used to take three minutes to read my first 100MB hard drive. It takes 40 minutes to over 1h to read the current desktop drives - alone, at their respective maximum sustained rate. The RAID controllers have a lower total throughput than the sum of their drives - maybe 150 MBps per unit of 16 drives. Using three separate SCSI busses for the three RAID units, maybe I could crank them up to 400 MBps of sustained transfer rate. That's about 8 hours just to read the whole 12TB thing. Think of sipping a swimming pool with a straw. ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
12TB GEOM stripe, newfs, then fsck: cannot alloc 768053748 bytes for blockmap
Dear everyone, this is not an important issue, but I'd like to ask anyway, just in case the solution was obvious: I've bundled three 4TB RAID boxes using GEOM::stripe into a single 12TB volume. I didn't partition it, I just created UFS on it using newfs (a dangerously dedicated volume). This is not a production setup, I'm just testing the hardware. I thought fsck would be a plausible benchmarking tool. -bash-2.05b# fsck_ufs /dev/stripe/data ** /dev/stripe/data cannot alloc 768053748 bytes for blockmap * FILE SYSTEM MARKED DIRTY * After a forced mount, the 12TB volume works fine. It just can't be fsck'ed. The machine is a dual Xeon/Nocona + i7520, with 2 GB of RAM, running FreeBSD 5.4-RELEASE with an SMP kernel. Any ideas are welcome :-) Frank Rysanek ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: can't get the Linux aaccli to work under 4.8-RELEASE, 4.9-RC1, 5.1-RELEASE
Yup, it works on 4.8, as well as on 4.9-RC1 and 5.1. Only on 5.1, AAC_COMPAT_LINUX doesn't exist anymore. But COMPAT_LINUX still seems to be necessary. I guess I'll post a summary to USENET news. Frank Rysanek options AAC_COMPAT_LINUX _and_ options COMPAT_LINUX ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 4.8, ASR2120, SMP, degraded RAID1/mirror = storage failure
Dear Mr. Long, during the last week or so, I've been trying to get more information about the problem in 4.8-RELEASE that this thread has been about. Yesterday, just when I thought that maybe I had some interesting data worth sending to you, I noticed that 4.9-RC1 was out. So I tested for the symptoms in that and I have to admit that IT WORKS IN 4.9-RC1 ! Wonderful! I.e., the machine does boot from a rebuilding array and array degradation at runtime doesn't make the machine hang because of storage failure. All of that with SMP, APIC_IO and HT enabled. No need to include the irrelevant ISP driver (see the attachments for an explanation of this comment). In the newsgroups, I've noticed other people complaing about various aac-based hardware under FreeBSD. I am aware that there have been changes to dev/aac/* between 4.8 and 4.9. I'm not sure whether you have managed to squash the bug, or if the remedy was incidental, and whether or not the bug was in the aac drivers or elsewhere in the system (APIC handling? DMA mapping?). Therefore, just in case you were interested, the information I gathered in 4.8 is attached to this message. Hmm. Now that this is solved, I'd like to focus on the defunct aaccli. I guess I'd better start another thread related to that. Thanks for the great job that you're doing in the FreeBSD team. And, thanks for your patience with me. Frank Rysanek The following problem description applies exclusively to 4.8-RELEASE. I have managed to carry out further research related to the topic of this e-mail thread, in the original 4.8-RELEASE. I have found some more deterministic symptoms of the problem, one other dependency apart from SMP+APIC_IO, but I'm stuck again. A detailed explanation follows. PROBLEM SUMMARY --- With the GENERIC kernel, the ASR2120 (driver AAC) works fine under all circumstances. With SMP+APIC_IO enabled or with device isp disabled, the controller and the driver work fine as long as the array volumes are fine. When an array becomes degraded at runtime, or when booting off a degraded (especially rebuilding) array, the system crashes miserably. The bottom line is, that my ASR2120 is not fault tolerant in SMP. DISCOVERED CFG DEPENDENCIES --- The ASR2120 controller and the aac driver WORK FINE under these conditions: 1) SMP+APIC_IO are disabled (a UP-only kernel) 2) 'device isp' is _enabled_ (though its PCI probe doesn't find anything) And no, I don't have any Qlogic chips in my machine. SYMPTOMS 1) the zero-padded FIB AKA unknown command from controller upon runtime disk fault or when booting from a rebuilding array. This is the original symptom. 2) During the kernel boot sequence, still with interrupts disabled, when aac_startup() probes for containers, it finds none - as a result of the controller being stuck as per symptom 3). 3) let's focus on booting: during aac_init(), when the controller is notified of a ready mailbox for the first time, the controller pukes - the drive LED's start flashing red and symptom 2) follows. Once interrupts are enabled, symptom 3) follows. I have discovered the precise moment when the controller pukes by inserting debug messages and DELAY(10 s) statements at various points in the code of aac_attach(), aac_init(), aac_sync_command() etc... Obviously the red flashing doesn't occur in the workable setup - the controller keeps rebuilding the array(s) merrily throughout FreeBSD boot. 4) with the working setup (see the CFG DEPENDENCIES section above), the MMIO assumes a different config than with any defunct setup. The physical address of the FIBs (or whatever it is) is different, I don't know why. See the two log snippets below. See also the attached tarball for more complete boot logs. The following are some variable dumps, logged by instrumentation that I have inserted into aac.c. This is the workable setup (UP 'device isp' enabled): FRR: generic attach - aac_attach() called FRR: Disabling interrupts. FRR:aac_init(): initing controller FRR: -- Init structure contents: -- FRR:aac_common is at e198 FRR:ac_init is ate1981000 FRR:InitStructRevision = 3 FRR:MiniPortRevision = 1 FRR:AdapterFibsPhysicalAddress = 1c000 FRR:AdapterFibsVirtualAddress = e198 FRR:AdapterFibsSize = 1000 FRR:AdapterFibAlign = 200 FRR:PrintfBufferAddress = 1f184 FRR:PrintfBufferSize = 100 FRR:HostPhysMemPages = 3fccf FRR:HostElapsedSeconds = 0 FRR: setting the outbound doorbell register to all one's. FRR:aac_common_busaddr = 1c000 FRR:ac_init offset = 1000 FRR: aac_sync_command() called. FRR: - populating the mailbox... FRR:aac_rx_set_mailbox() called. FRR:btag: 1 FRR:handle: dd96e000 FRR:command: 5 FRR:arg0: 1d000 FRR:arg1: 0 FRR:arg2: 0 FRR:arg3: 0 FRR: - clearing
can't get the Linux aaccli to work under 4.8-RELEASE, 4.9-RC1, 5.1-RELEASE
Dear Mr. Long,
I'm not able to make the Linux aaccli work under any recent FreeBSD.
I've obtained a report from Buki that it worked for him under some STABLE
snapshot but it doesn't work for me...
I am aware that the old binary release of aaccli for FreeBSD 4.4 from the
Adaptec web is dangerous. For mere looking it works just fine though, both
in 4.8 and in 4.9-RC1. All I had to do was use /dev/MAKEDEV to create a
device node (/dev/aac0) which is not there by default.
I haven't tried any recovery operations with it (rip out a drive, remove
it from the array using aaccli, plug it back and assign it as a hot spare
to invoke a rebuild).
Back to the Linux aaccli:
I'm trying to use the linux binary shipped on the Adaptec bootable CD
with the ASR2120 controllers. I'm not unpacking it out of an RPM, as you
have described in a recent e-mail/posting to someone.
I do have the Linux emulation installed - I just asked for that during
the standard install procedure and it seems to work. I also have
AAC_COMPAT_LINUX enabled in the 4.x versions.
The aaccli from the CD complains about an incorrect ABI version of the
libncurses.so.5 - it's wrong, the so.5 is really missing, this issue
can be solved by exporting a LD_LIBRARY_PATH pointing to
/cdrom/bootcd/usr/lib in the current shell, or by copying the library from
the CD into /compat/linux/usr/lib.
The real problem is elsewhere. The SENDFIB IOCTL still doesn't get
through. This probably applies to the other aac IOCTL's, too - it's
just that SENDFIB is the first thing that the aaccli does when
open aac0 is issued, so this is the only error that I obtain.
CLI open aac0
Command Error: The driver could not execute the requested IOCTL SENDFIB,
22=invalid argument.
linux: 'ioctl' fd=4, cmd=0x2008 (' ',8) not implemented
The IOCTL code seems to match. Both the FreeBSD and the
Linux aac driver source files define essentially the same codes:
FSACTL_SENDFIB CTL_CODE == 0x0004 | (2050 2)
== 0x0004 | (0x0802 2)
== 0x00042008
The trouble is that aac_linux_ioctl() never gets called...
The Linux emulator layer hijacks the IOCTL, generates the
kernel error message and never passes the IOCTL to the
aac driver. Perhaps the SYSINIT hook in aac_linux.c
fails for some reason?
The brandelf util doesn't make any difference, either. I understand that
it helps to get the Linux version of libraries to load. I've even tried
copying both aaccli and the libncurses.so.5 from the CD to a directory
under /usr (mounted RW), to no avail.
I've also tried stracing aaccli to see if it opens /dev/aac0 at all.
I donwloaded and compiled the shiny new strace 4.5. I got garbage
instead of file names in the records of open() calls. My immediate
impression was OK, a bug in strace - so I downloaded strace 4.4 that had
worked for me on FreeBSD 4.8 in the past. This time I got core dumps from
strace. Then I tried stracing some other proggies and voila, the file
names in open() were reported just fine.
Hahaa, the Linux binaries are using different syscalls. No need to report
a bug to the authors of strace.
So I first tried stracing a shell and run aaccli from that - to
encapsulate aaccli with the emulator in a shell. That way, it seemed as if
I ran nothing from the shell...
The next thing I did, I tried downloading a strace binary from Linux
and run it on FreeBSD. The emulator kicked in all right - and complained
that ptrace() was not implemented :-(
Nevertheless, I believe that aaccli does indeed open /dev/aac0.
If it failed to open the device node, it would complain at that point
already. The ioctl is clearly there:
ioctl(address, 0x4, 0x42008) = -1 (errno -22)
Could it be opening a different file? Hmm.
I'm pretty much stuck right there, relying on the dated FreeBSD aaccli
binary for the moment. I'm somewhat reluctant to start studying the IOCTL
passing path in the Linux emulator source.
Any ideas are welcome.
Frank Rysanek
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: can't get the Linux aaccli to work under 4.8-RELEASE, 4.9-RC1, 5.1-RELEASE
I'm not able to make the Linux aaccli work under any recent FreeBSD.
heck, seems like I got it, you need _both_
options AAC_COMPAT_LINUX
_and_
options COMPAT_LINUX
because the linking of aac_linux.o (or whatever it's called) into the
kernel binary is dependent on COMPAT_LINUX - see /usr/src/sys/conf/files.
I guess removing the compat_linux tag from /usr/src/sys/conf/files would
work too.
As a side effect (really the main effect) of COMPAT_LINUX, the linux ABI
apparently becomes statically linked to the kernel, as opposed to modular,
which is the default with the stock install.
I had to find out the hard way. Being a newbie, I use instrumentation
wherever I should really use the kernel debugger. This time, I put several
watches at interesting points in the code - to see where the aac driver
thinks the ioctl handler entry point is and to see if the Linux ABI init
routine finds it in its input chain of ioctl handlers.
compat/linux/linux_ioctl.c:
===
linux_ioctl_register_handler()
{
...
printf(FRR: registering linux ioctl handler at 0x%x\n,
(unsigned int) h-func);
...
}
dev/aac/aac_linux.c:
/* removed the 'static' keyword */
/*static struct linux_ioctl_handler aac_linux_handler =
{aac_linux_ioctl,*/
struct linux_ioctl_handler aac_linux_handler = {aac_linux_ioctl,
...
dev/aac/aac_pci.c:
==
#include machine/../linux/linux.h
extern struct linux_ioctl_handler aac_linux_handler;
aac_pci_attach()
{
...
printf(FRR: the aac linux ioctl handler is at 0x%x\n,
(unsigned int) aac_linux_handler.func);
...
}
Now the linker yelled upon linking - it complained about
unknown symbol aac_linux_handler.
That's where it started to dawn on me that I should fumble
in the Makefiles and config data.
I got it working a few minutes later.
At least, it works in 4.8. I have yet to test it in different
FreeBSD versions. I have to leave that for tomorrow.
Frank Rysanek
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-questions
To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: FreeBSD 4.8, ASR2120, SMP, degraded RAID1/mirror = storage failure
Dear Mr. Long, thanks a lot for taking the time to respond - especially given that you're on vaccations and that it's almost 2 a.m. your time. I apologize for having used vague formulations in my past mail. Also, perhaps I have made up wrong meanings for some vocabulary occuring in the driver's code. Specifically: 2. What is a zero-padded FIB? I concede that the AIF handling in the driver is sub-par and needs to be revisited, so I'd like to know what you are seeing. I was referring to this: aac_dequeue_fib: called aac0: aac_host_command: FIB @ 0xe1984000 aac0: XferState 0 aac0: Command 0 aac0: StructType0 aac0: Flags 0x0 aac0: Size 0 aac0: SenderSize0 aac0: SenderAddress 0x0 aac0: RcvrAddress 0x0 aac0: SenderData0x0 aac0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 aac0:00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 aac0: unknown command from controller The size is a zero, the data dump at the end contains all zeroes. That's why I called it a zero-padded FIB. This only occurs when an unhandled array failure arrives - when the machine is about to hang upon runtime array degradation or at boot from a degraded array. Which normally only happens with SMP+APIC_IO enabled. Not with a UP kernel. All the other FIB listings that I've seen contain some non-zero data and claim non-zero length... In my last message, I have attached a tarball with some logs. To see what I'm talking about, please take a look at this: - runtime array degradation - compare the two logs: - SMP, unrecoverable failure: logs/DEBUG_CAM_AAC_L2/SMP-2_disk_failed (line 23) - UP, system keeps going just fine: logs/DEBUG_CAM_AAC_L2/NOSMP-2_disk_failed (line 76) - boot from a degraded array - compare the two logs: - SMP, unrecoverable failure logs/DEBUG_AAC_L4/SMP-3_boot_with_degraded_array_failed (line 296) - UP, system boots just fine: logs/DEBUG_AAC_L4/NOSMP-3_boot_with_degraded_array_OK (line 273) 3. The split and corrupted messages on the console were likely due to kernel printfs happening from different contexts at the same time. The printf facility has no serializing ability, unfortunately. OK. 4. I'm unclear on what you mean by there being a problem in the asynchronous handling of device printfs and host command fibs. I'd be very interested in more information on this. I didn't mean to say that there was the cause of my problem in that area. I meant to say that I have a problem understanding what's going on. I'm not a skilled coder, I have a hard time understanding how the driver's code works. I am able to see where a function is called with some arguments and returns with a result. However, when a SCSI command is issued to the controller, at the driver level the request/response doesn't happen within a single function. One function queues the command to the controller via the MMIO (?) region and the response from the controller eventually comes back within an interrupt, invoking an interrupt handler. The response may be a valid SCSI response to the SCSI command, or a something went wrong **Monitor** event. I am vaguely aware that the SCSI controller can reorder commands in the queue or process them out of order. Combine this with the unserialized logging and I'm lost :) Sorry. If there's something specific I should check for, please let me know. Thanks for being patient with my hasty descriptions :) Frank Rysanek ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
FreeBSD 4.8, ASR2120, SMP, degraded RAID1/mirror = storage failure
)
{
// process it
}
else
{
break; // go to sleep again
}
}
}
}
aac_dequeue_fib()
{
if (ci != pi) // consumer/producer indices
{
// there are some FIBs in the queue
// !!! at the same time, the FIB is zero-padded !!!
}
else return(ENOENT);
}
Another symptom is that, upon array degradation, the controller
seems to reset the RAID-private SCSI bus (I hope that's what
the **Monitor** message says).
The trouble is that both the aac_host_command() wakeup with the
zero-padded FIB and the monitor messages appear in asynchronous
context (in a separate kthread or in an interrupt) and I'm not
as skilled as to say which previous action of the driver is
the immediate cause.
More on the behavior of the disk LEDs:
These LEDs on my server case are controlled by the SCA/SAF-TE
chip (GEM318).
- When the array is degraded but operating normally, the dead
disk's LED is dark and the live disk's led flashes green,
indicating normal storage transfers.
- When a degraded array is rebuilding, the two disk LEDs dance
in shades of green to orange (both the green and red
pads flashing).
- When the whole controller or the RAID-private SCSI channel
is being reset, both the two LEDs shine a steady red.
- When the machine fails at boot with a rebuilding array,
the LEDs often turn red for a few seconds (reset?) and then
one of them remains red and the other one starts dancing
green/orange... and the reset may come back a few times
before the machine locks up entirely or the BSD manages
to do an auto-reboot. Or the LED's just stay red and the
machine hangs.
- When the machine boots and runs fine (i.e., with a UP kernel
under normal conditions), the disk LED's never go red, except
for a cold reset of the whole PC. When the array is rebuilding,
the LED's keep dancing merrily between green and orange
throughout the boot process.
I guess this would indicate that it's not just the BSD driver
getting messed up - the controller probably also gets
seriously confused. Is that a chicken-vs.-egg style puzzle?
As a side note: it seems interesting to me that, regardless
of whethere debugging and SMP is on or off in any particular
combination, the kernel always rushes through to
Waiting 15 seconds for the SCSI devices to settle
and _immediately_ reports the RAID containers.
Only then it waits those fifteen seconds before proceeding
to detect the regular SCSI devices.
Attached is my kernel config file and a listing of
`lspci -lv`
I can't think of anything else to tell you at the moment.
Ask me if you need further help - perhaps I can modify the
debugging flags and try again, add some more instrumentation
hooks here and there to focus on particular points in the
code etc.
Any ideas are welcome.
Sorry about wasting your time by sending such an eloquent
explanation.
Thanks for the great job that you're doing.
Frank Rysanek
[EMAIL PROTECTED]:0:0: class=0x06 card=0x chip=0x00141166 rev=0x31
hdr=0x00
vendor = 'Reliance Computer Corp./ServerWorks'
device = 'CNB20-HE Host Bridge'
class= bridge
subclass = HOST-PCI
[EMAIL PROTECTED]:0:1: class=0x06 card=0x chip=0x00141166 rev=0x00
hdr=0x00
vendor = 'Reliance Computer Corp./ServerWorks'
device = 'CNB20-HE Host Bridge'
class= bridge
subclass = HOST-PCI
[EMAIL PROTECTED]:0:2: class=0x06 card=0x chip=0x00151166 rev=0x00
hdr=0x00
vendor = 'Reliance Computer Corp./ServerWorks'
device = 'CMIC-GC Hostbridge and MCH'
class= bridge
subclass = HOST-PCI
[EMAIL PROTECTED]:2:0: class=0x03 card=0x80041002 chip=0x47521002 rev=0x27
hdr=0x00
vendor = 'ATI Technologies'
device = 'Rage XL PCI'
class= display
subclass = VGA
[EMAIL PROTECTED]:15:0: class=0x060100 card=0x02011166 chip=0x02011166 rev=0x93
hdr=0x00
vendor = 'Reliance Computer Corp./ServerWorks'
device = 'CSB5 PCI to ISA Bridge'
class= bridge
subclass = PCI-ISA
[EMAIL PROTECTED]:15:1: class=0x01018a card=0x02121166 chip=0x02121166 rev=0x93
hdr=0x00
vendor = 'Reliance Computer Corp./ServerWorks'
device = 'CSB5 PCI EIDE Controller'
class= mass storage
subclass = ATA
[EMAIL PROTECTED]:15:2: class=0x0c0310 card=0x02201166 chip=0x02201166 rev=0x05
hdr=0x00
vendor = 'Reliance Computer Corp./ServerWorks'
device = 'OSB4 OpenHCI Compliant USB Controller'
class= serial bus
subclass = USB
[EMAIL PROTECTED]:15:3: class=0x06 card=0x02301166 chip=0x02251166 rev=0x00
hdr=0x00
vendor = 'Reliance Computer Corp./ServerWorks'
device = 'CSB5 PCI Bridge'
class= bridge
subclass = HOST-PCI
[EMAIL PROTECTED]:17:0: class=0x06 card=0x chip=0x01011166 rev=0x03
hdr=0x00
vendor = 'Reliance Computer Corp./ServerWorks'
device = 'CIOB-X2'
class= bridge
subclass = HOST-PCI
[EMAIL PROTECTED]:17:2: class=0x06 card=0x chip
