Re: Pegasos i8042 broken again
Gabriel Paubert writes: > > Ok, I got fed up about it. The patch referred above is obviously wrong since > it leaves interrupts at 0 when a device_type or name of 8042 is found, > so what about the following? Looks like the workaround I was using for a while. In the original report I said I wasn't sending my kernel workaround patch because of the previous disagreements about whether the kernel should work around this type of bug. (In fact the current difficulty is the result of changes being made without considering the special case that was created by my first workaround... what a mess.) I also said I wasn't comfortable hacking the Forth-based part of the boot sequence because I didn't know the language. As it turned out, learning Forth was much easier than getting any guidance from the kernel people on how to proceed with a workaround, so I wrote this: === CUT HERE === " /isa/8042" find-device : open true ; : close ; : decode-unit ( addr len -- phys ) 1 <> if abort" invalid unit address" then c@ dup ascii 0 = if drop 0 exit then ascii 1 = if 1 exit then abort" invalid unit address" ; : encode-unit ( phys -- addr len ) dup 0 = if " 0" exit then 1 = if " 1" exit then abort" invalid unit address" ; 1 encode-int 3 encode-int encode+ d# 12 encode-int encode+ 3 encode-int encode+ " interrupts" property 0 0 " 0" " /isa/8042" begin-package " keyboard" device-name " keyboard" device-type " pnpPNP,303" encode-string " compatible" property 0 encode-int " reg" property end-package 0 0 " 1" " /isa/8042" begin-package " mouse" device-name " mouse" device-type " pnpPNP,f03" encode-string " compatible" property 1 encode-int " reg" property end-package === CUT HERE === Along with the previous device tree patch (pegasos-dts-20071018), this should present the kernel with a properly filled-out 8042 device-tree node, preventing the need for any more patching the next time the kernel changes its mind about how to initialize the keyboard driver. -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: Pegasos OHCI bug (was Re: PROBLEM: memory corrupting bug,
Segher Boessenkool writes:
>
> > Now I'm just trying to find the more correct way of doing it, without
> > hardcoded addresses. That'll be something like this:
> >
> > search the device tree for OHCI nodes
> > for each OHCI node
> > get assigned-addresses
> > map-in
> > set HCR
> > wait for acknowledgement
> > map-out
> Sounds like it should work, yes.
>
I have a mostly-finished patch to do the above. I'll include it below, but
first a few words about why it's only mostly finished.
The other Pegasos workarounds are in fixup_device_tree_chrp, and I don't see
anything like an "if(machine_is_pegasos)" around them. What keeps them from
being erroneously run on other CHRP-type machines? I made this patch mainly
by copying pieces of other functions from prom_init.c, but couldn't find the
"test for Pegasos before running a Pegasos workaround" piece.
Another issue is, since the firmware doesn't give me a "compatible" property
with the details of the controller, I just have to assume that it's
little-endian. I'm not sure if that's clean, since the real ohci driver
supports both endiannesses, with at least 3 different Kconfig options(!) to
choose between them.
Then there's the volatile which I guess is supposed to be replaced by
something else, but I don't know what the something else is. I believe this
usage is extremely close to what volatile was meant for.
Finally, when I updated to a more recent upstream kernel to test the patch, I
found that an intervening commit (3df7169e73fc1d71a39cffeacc969f6840cdf52b,
OHCI: work around for nVidia shutdown problem) has had a major effect,
on the appearance of my bug.
Before that change, the window in which the bug could strike was from the end
of prom_init (when the kernel believes that devices are quiescent) to the
initialization of the ohci-hcd driver (which actually quietens the device, or
at least directs its scribbling to a properly allocated page). After the
change, the window ends at some point early in the PCI bus setup. That's a
window so small that with a new kernel, I can't provoke a symptom even if I
try.
Mostly-finished patch:
diff --git a/arch/powerpc/kernel/prom_init.c b/arch/powerpc/kernel/prom_init.c
index 941ff4d..a14f21b 100644
--- a/arch/powerpc/kernel/prom_init.c
+++ b/arch/powerpc/kernel/prom_init.c
@@ -2237,6 +2237,81 @@ static void __init fixup_device_tree_chrp(void)
}
}
}
+
+/*
+ * Pegasos firmware doesn't quiesce OHCI controllers, so do it manually
+ */
+static void __init pegasos_quiesce(void)
+{
+ phandle node, parent_node;
+ ihandle parent_ih;
+ int rc;
+ char type[16], *path;
+ u32 prop[5], map_size;
+ prom_arg_t ohci_virt;
+
+ for (node = 0; prom_next_node(&node); ) {
+ memset(type, 0, sizeof(type));
+ prom_getprop(node, "device_type", type, sizeof(type));
+ if (strcmp(type, RELOC("usb")) != 0)
+ continue;
+
+ /* Parent should be a PCI bus (so class-code makes sense).
+ class-code should be 0x0C0310 */
+ parent_node = call_prom("parent", 1, 1, node);
+ if (!parent_node)
+ continue;
+ rc = prom_getprop(node, "class-code", prop, sizeof(u32));
+ if (rc != sizeof(u32) || prop[0] != 0x0c0310)
+ continue;
+
+ rc = prom_getprop(node, "assigned-addresses",
+ prop, 5*sizeof(u32));
+ if (rc != 5*sizeof(u32))
+ continue;
+
+ /* Open the parent and call map-in */
+
+ /* It seems OF doesn't null-terminate the path :-( */
+ path = RELOC(prom_scratch);
+ memset(path, 0, PROM_SCRATCH_SIZE);
+
+ if (call_prom("package-to-path", 3, 1, parent_node,
+ path, PROM_SCRATCH_SIZE-1) == PROM_ERROR)
+ continue;
+ parent_ih = call_prom("open", 1, 1, path);
+
+ /* Get the OHCI node's pathname, for printing later */
+ memset(path, 0, PROM_SCRATCH_SIZE);
+ call_prom("package-to-path", 3, 1, node,
+ path, PROM_SCRATCH_SIZE-1);
+
+ map_size = prop[4];
+ if (call_prom_ret("call-method", 6, 2, &ohci_virt,
+ ADDR("map-in"), parent_ih,
+ map_size, prop[0], prop[1], prop[2]) == 0) {
+ prom_printf("resetting OHCI device %s...", path);
+
+ /* Set HostControllerReset (==1) in HcCommandStatus,
+* located at offset 8 in the register area. The <<24
+* is because the CPU is big-endian and the device is
+* little-endian. */
+ *(volatile u32 *)(ohci_virt + 8) |= (1<<24);
+
+
Re: Pegasos OHCI bug (was Re: PROBLEM: memory corrupting bug,
Segher Boessenkool writes: > > > So is it wrong to leave the host controller enabled when the OS is booted? > > Yes. Or, rather, there should be some way for the client to turn off > all dma and interrupt activity; if the client closes the ihandles in > "/chosen", and perhaps calls "quiesce", that should be enough. Sounds good to me, I only wish someone had written down what "quiesce" means. > > > > Almost all of my devices are under that PCI node. What will I prove by > > disabling them? > > You should put it after "load", and before "go". > > It should give you a working system; it's a sledgehammer workaround. I can do it a little more gracefully than that. This works to deactivate the problem devices manually: 1 lbflip 8000 8 + rl! 1 lbflip 80001000 8 + rl! where 8000 and 80001000 have been obtained from /p...@8000/u...@5/assigned-addresses and /p...@8000/u...@5,1/assigned-addresses; 8 is the offset of the HcCommandStatus register; and the 1 bit is HostControllerReset (HCR). Now I'm just trying to find the more correct way of doing it, without hardcoded addresses. That'll be something like this: search the device tree for OHCI nodes for each OHCI node get assigned-addresses map-in set HCR wait for acknowledgement map-out which can be done any time before the quiesce call, since that marks the point where the kernel assumes that there are no devices writing to memory. Sound good? -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: Pegasos OHCI bug (was Re: PROBLEM: memory corrupting bug,
Segher Boessenkool writes: > > >> 1) Figure out what exactly is going on; > > > > I thought we were past that. > > We are not. > > > The startup sequence leaves the device in a > > bad > > state (writing 1000 times per second to memory that the kernel believes is > > not in use), so it needs to be given a reset command before the kernel > > tries > > to use that memory. > > The question now is what causes the firmware to do that, and then > what is the best way to stop it from doing that. As far as I can tell, it turns on the host controller during the global probe, which is not wrong because USB devices could theoretically be used for booting, or for console display. Then it never turns off the host controller because someone forgot to put in the code to turn it off. It's not easy to figure out exactly where that should have been done. Turning off the host controller too soon would rule out booting from USB, but leaving it running while the OS is starting up has caused a major problem. So is it wrong to leave the host controller enabled when the OS is booted? If not, then the error must be in the communication of which memory addresses are in use by OF. I've got a node /mem...@0 whose "available" property looks like this: 0040 00584000 0007c000 0092a1d8 4e28 00a2f000 005d1000 0180 0e3fd000 0fbffab4 054c >From that list, it looks to me like OF is telling the kernel that it should not attempt to use any address above 0xfbffab4+0x54c == 0xfc0. The addresses being written to by the OHCI controller are 0xfc5c080 and 0xfc61080. If the kernel is staying within the "available" list, there won't be a problem. Later, when the kernel decides it's done using OF, what's supposed to happen? It closes stdin, but that doesn't help here since the offending device is a bus node, not an input node. It looks to me like the kernel makes the assumption that all devices other than stdin and stdout will have been deactivated already when the kernel starts, and that this assumption has been violated. Who is wrong, from the perspective of the OF standard, the assumer or the violator? Then there's the "quiesce" call, which I don't understand at all since it's not mentioned in any of the specification documents I've been able to find. It's been mentioned as an Apple-only thing. Seems like it would be a good name for a "make all the devices stop puking on the RAM" function. Since the OF spec doesn't include this function, they must not have thought it was necessary. > > /p...@8000/u...@5/assigned-addresses > > 02002810 8000 1000 > > Lovely, incorrect data (it should start with 82002810, i.e., > not relocatable -- it is already an assigned address!). Now you see how I have trouble relating the docs to the reality... > > This means: 32-bit MMIO address space for bus 0 dev 5 fn 0, > first BAR; assigned to address 8000; size is 1000. But "address 8000" is a physical address (I think), so do I need to do a map-in on it before using it? > > You could try a boot script like this: > > > dev /pci > 0 04 DO 0 i config-w! -100 +LOOP > device-end > > > which should disable all PCI devices on all busses, on that Almost all of my devices are under that PCI node. What will I prove by disabling them? -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: Pegasos OHCI bug (was Re: PROBLEM: memory corrupting bug,
Segher Boessenkool writes: > > >> > |1. How do I locate all usb nodes in the device tree? > >> > | > >> > |2. How do I know if a particular usb node is OHCI? > > You look for compatible "usb-ohci". There is no "compatible" there. I can probably use class-code since the parent is a PCI bus. > > But this doesn't help you. You do not know yet if the > problem happens for all usb-ohci; for example, it could be > that you have the console output device on usb; or as another > example, it could be that this firmware leaves all pci devices > in some active state. > > So as I see it you have only two options: > > 1) Figure out what exactly is going on; I thought we were past that. The startup sequence leaves the device in a bad state (writing 1000 times per second to memory that the kernel believes is not in use), so it needs to be given a reset command before the kernel tries to use that memory. > > The big question that I'm still stumbling over is how to access the device > > registers. The "reg" property looks like this: > > You should look at "assigned-addresses", not "reg". Well, > you first need to look at "reg" to figure out what entry > in "assigned-addresses" to use. The properties look like this: /p...@8000/u...@5/assigned-addresses 02002810 8000 1000 /p...@8000/u...@5/reg 2800 02002810 1000 I'm not sure how I'm supposed to know which entry from "reg" is the right one. I've been guessing that it's the second one, since that one matches the only entry in "assigned-addresses". It's supposed to go the other direction? -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: Pegasos OHCI bug (was Re: PROBLEM: memory corrupting bug,
Olaf Hering writes:
>
> On Wed, Oct 27, pac...@kosh.dhis.org wrote:
>
> > |1. How do I locate all usb nodes in the device tree?
> > |
> > |2. How do I know if a particular usb node is OHCI?
>
> In the installed system, run 'lspci | grep -i usb', this gives the pci
> bus numbers. Then run 'find /sys -name devspec', and look or the bus
Once the system is running, I have no problem figuring it out. What I meant
was how do I write some code to identify OHCI devices correctly, from within
the limited environment of the Forth interpreter, which will work in the
general case.
I already know that /p...@8000/u...@5 and /p...@8000/u...@5,1 are the
problem nodes on my machine. And I've learned enough about OF to do a full
recursive device tree search to find the USB nodes, so the first question is
answered.
But the UHCI and OHCI nodes look very much alike in the OF properties. "name"
is just "usb" and there's no "compatible".
The big question that I'm still stumbling over is how to access the device
registers. The "reg" property looks like this:
phys size
-- -
2800
02002810 1000
so I take the second group of 5 words, which should be the device registers,
and try to map it to a virtual address. The members are unpacked on the stack
like this:
02002810 1000
which looks like this stack diagram from OF spec:
map-in ( phys.lo ... phys.hi size -- virt )
and the method call goes like this:
" map-in" $call-parent
The result: "invalid pointer". But I notice it only popped 4 items. I think
maybe the "size" for map-in is not the same as the "size" found in the reg
property. Maybe #size-cells applies in one place but not the other. Thanks
for not documenting that! Try again:
02002810 1000 " map-in" $call-parent
This one doesn't complain, but leaves me a 0 on the stack as its answer. The
OHCI registers have been mapped to virtual address 0? Doesn't seem likely.
--
Alan Curry
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
Pegasos OHCI bug (was Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55)
Benjamin Herrenschmidt writes: > > Ok so you'll have to make up a "workaround" in prom_init that looks for > OHCI's in the device-tree and disable them. > > Check if the OHCI node has some existing f-code words you can use for > that with "dev /path-to-ohci words" in OF for example. If not, you may > need to use the low level register accessors. Use OF client interface > "interpret" to run forth code from C. I responded with a long list of reasons that I'm not qualified to do that work myself: |Here are the major problems: | |1. How do I locate all usb nodes in the device tree? | |2. How do I know if a particular usb node is OHCI? | |3. Knowing that a node is OHCI, how do I know where its control registers |are? I'm sure this is calculated from the "reg" property but I don't see how. | |4. Knowing where the control registers are, how do I access them? Do I need |to request a virt-to-phys mapping or can I assume that it's already mapped, |or that the "rl!" command will do the right thing with a physical address? | |5. Which control register should I use to tell the OHCI to be quiet? Just do |a general reset, or is there something that specifically turns off the |counter that's been causing the trouble? Since then, the silence has been deafening. My assumption now is that this is not ever getting fixed. I'm certainly not able to fix it. I'm not a even kernel programmer! I got far enough to diagnose the cause just with the "add more printk's and boot it again" technique. Hundreds of reboots trying to figure it out. I was a conscientious bug-reporter, I thought. I could pull the PCI card and be done with it. I never used those USB ports anyway. But after all the suffering I went through to find this bug... the crashing e2fsck's and consequent filesystem corruption... I hate the idea of surrendering to it. There are possibly other affected users who I'd be abandoning to suffer similarly in the future. For the last week I've studied OpenFirmware as hard as I can. I read the spec cover to cover. And the USB annex, and the PCI annex. But I'm still lost in all the different address formats. I took my best guess on how to handle this problem, and ran with it, ending up with a 97-line Forth script, and that was just to get a virtual address, not to actually do anything with it, and it used a hardcoded device path. But it didn't work, all I got was an "invalid pointer" error. I made another guess at something that wasn't documented anywhere (the fact that this stuff is insufficiently documented is the one thing I can state with complete confidence!) and out came a successful translation to a virtual address: 0. If I'm the only one fighting this bug, the bug wins. -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55
Benjamin Herrenschmidt writes: > > On Wed, 2010-10-20 at 13:33 -0500, pac...@kosh.dhis.org wrote: > > > Just try :-) "quiesce" is something that afaik only apple ever > > > implemented anyways. It uses hooks inside their OF to shut down all > > > drivers that do bus master (among other HW sanitization tasks). > > > > I booted a version with a prom_close_stdout after the last prom_debug. It > > didn't have any effect. That 1000Hz clock was still ticking. > > Ok so you'll have to make up a "workaround" in prom_init that looks for > OHCI's in the device-tree and disable them. I'm a long way from understanding how to do that. > > Check if the OHCI node has some existing f-code words you can use for > that with "dev /path-to-ohci words" in OF for example. If not, you may Nothing there but open close decode-unit encode-unit > need to use the low level register accessors. Use OF client interface > "interpret" to run forth code from C. Here are the major problems: 1. How do I locate all usb nodes in the device tree? 2. How do I know if a particular usb node is OHCI? 3. Knowing that a node is OHCI, how do I know where its control registers are? I'm sure this is calculated from the "reg" property but I don't see how. 4. Knowing where the control registers are, how do I access them? Do I need to request a virt-to-phys mapping or can I assume that it's already mapped, or that the "rl!" command will do the right thing with a physical address? 5. Which control register should I use to tell the OHCI to be quiet? Just do a general reset, or is there something that specifically turns off the counter that's been causing the trouble? -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55
Benjamin Herrenschmidt writes: > > On Tue, 2010-10-19 at 22:23 -0500, pac...@kosh.dhis.org wrote: > > The diff fragment above applied inside prom_close_stdin, but there are > > some > > prom_printf calls after prom_close_stdin. Calling prom_printf after > > closing > > stdout sounds like it could be bad. If I moved it down below all the > > prom_printf's, it would be after the "quiesce" call. Would that be > > acceptable > > (or even interesting as an experiment)? Does a close need a quiesce > > after it? > > Just try :-) "quiesce" is something that afaik only apple ever > implemented anyways. It uses hooks inside their OF to shut down all > drivers that do bus master (among other HW sanitization tasks). I booted a version with a prom_close_stdout after the last prom_debug. It didn't have any effect. That 1000Hz clock was still ticking. -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55
Benjamin Herrenschmidt writes:
>
> On Tue, 2010-10-19 at 22:47 +0200, Segher Boessenkool wrote:
> >
> > It looks like it is the frame counter in an USB OHCI HCCA.
> > 16-bit, 1kHz update, offset x'80 in a page.
> >
> > So either the kernel forgot to call quiesce on it, or the firmware
> > doesn't implement that, or the firmware messed up some other way.
>
> I vote for the FW being on crack. Wouldn't be the first time with
> Pegasos.
>
> It's an OHCI or an UHCI in there ?
There's one of each... UHCI on the motherboard, OHCI on a card in a PCI
expansion slot. They shipped the ODW with the extra controller on an
expansion card since the on-board UHCI doesn't do USB2.0.
And that OHCI controller does appear to be the culprit. The 2 affected
addresses tick at 1000Hz until ohci-hcd is modprobe'd, then they stop.
I think the mm people can consider this closed. 6dda9d55 didn't do anything
but expose a problem which has been here all along. Will drop them from Cc
list in any further messages.
>
> Can you try in prom_init.c changing the prom_close_stdin() function to
> also close "stdout" ?
>
> if (prom_getprop(_prom->chosen, "stdin", &val, sizeof(val)) > 0)
> call_prom("close", 1, 0, val);
> +if (prom_getprop(_prom->chosen, "stdout", &val, sizeof(val)) > 0)
> + call_prom("close", 1, 0, val);
>
> See if that makes a difference ?
Huge difference. With no stdout to print to, the kernel seems to freeze up.
Or at least it loses the console. The last message it prints is "Device tree
struct 0x00933000 -> 0x00957000" then there's just nothing. I waited a while
for the console to come on but it didn't.
The diff fragment above applied inside prom_close_stdin, but there are some
prom_printf calls after prom_close_stdin. Calling prom_printf after closing
stdout sounds like it could be bad. If I moved it down below all the
prom_printf's, it would be after the "quiesce" call. Would that be acceptable
(or even interesting as an experiment)? Does a close need a quiesce after it?
--
Alan Curry
___
Linuxppc-dev mailing list
Linuxppc-dev@lists.ozlabs.org
https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55
Benjamin Herrenschmidt writes: > > > > I thought of that, but as far as I can tell, this CPU doesn't have DABR. > > AFAIK, the 7447 is just a derivative of the 7450 design which -does- > have a DABR ... Unless it's broken :-) Hmm. gdb resorts to single-stepping when I set a watchpoint while debugging some userspace program, which I assumed was caused by lack of hardware watchpoint support. But that's not important right now. I made a new discovery. During a test boot while looking at the usual symptom of a corrupted page cache, I run md5sum /sbin/e2fsck twice and got 2 different results, neither one of them correct. The third time, yet another different result. A few dozen more times, a few dozen more unique results. I had somehow managed to get a usable interactive shell while corruption was ongoing. So then I ran dd if=/dev/mem bs=4 count=1 skip=$((0xfc5c080/4)) | od -t x4 a few times very fast, plucking the first affected word directly out of memory by its physical address. The result: The low 16 bits are always zero as before. The high 16 bits are a counter, being incremented at about 1000Hz (as close as I could measure with a crude shell script. 1024Hz would also be within the margin of error). And it's little-endian. While I was watching this happen, there were only 5 or 6 userspace processes running, and 3 of them were shells. So I doubt that anything in userspace was doing it. It went on for a few minutes before I exited the interactive shell and allowed the boot to continue, while keeping an extra shell running on tty2 to continue making observations. It stopped incrementing almost immediately. So what type of driver, firmware, or hardware bug puts a 16-bit 1000Hz timer in memory, and does it in little-endian instead of the CPU's native byte order? And why does it stop doing it some time during the early init scripts, shortly after the root filesystem fsck? I have not yet attempted to repeat the experiment. If it is repeatable, I'll probe more deeply into those init scripts later. I'm looking hard at /etc/rcS.d/S11hwclock.sh -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55
Benjamin Herrenschmidt writes: > > You can do something fun... like a timer interrupt that peeks at those > physical addresses from the linear mapping for example, and try to find > out "when" they get set to the wrong value (you should observe the load > from disk, then the corruption, unless they end up being loaded > incorrectly (ie. dma coherency problem ?) ... I'm headed toward something like that. Maybe not a timer, maybe a "check it every time the kernel is entered". But first I have to work out exactly when the disk load completes so I know when to start checking. > > >From there, you might be able to close onto the culprit a bit more, for > example, try using the DABR register to set data access breakpoints > shortly before the corruption spot. AFAIK, On those old 32-bit CPUs, you > can set whether you want it to break on a real or a virtual address. I thought of that, but as far as I can tell, this CPU doesn't have DABR. /proc/cpuinfo processor : 0 cpu : 7447/7457 clock : 999.90MHz revision: 1.1 (pvr 8002 0101) bogomips: 66.66 timebase: platform: CHRP model : Pegasos2 machine : CHRP Pegasos2 Memory : 512 MB My next thought was: right after the correct value appears in memory, unmap the page from the kernel and let it Oops when it tries to write there. Then I found out that the kernel is using BATs instead of page tables for its own view of memory. Booting with "nobats" completely changes the memory usage pattern (probably because it's allocating a lot of pages to hold PTEs that it didn't need before) > > You can also sprinkle tests for the page content through the code if > that doesn't work to try to "close in" on the culprit (for example if > it's a case of stray DMA, like a network driver bug or such). No network drivers are loaded when this happens. -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55
Mel Gorman writes: > > A bit but I still don't know why it would cause corruption. Maybe this is > still > a caching issue but the difference in timing between list_add and > list_add_tail > is enough to hide the bug. It's also possible there are some registers > ioremapped after the memmap array and reading them is causing some > problem. I've been doing a lot more tests and I'm sure that 6dda9d55 is not really responsible. It just happens to provoke the bug in my particular setup. Whatever it is, it's very sensitive to small changes. At the end of free_all_bootmem, the free list for order 9 has 4 entries. Which one is at the head of the list depends on whether 6dda9d55 is applied or not. If page number 130048 is at the head of the list, it gets used fairly soon, and everything's fine. The alternative is that page number 64512 is at the head of the list, so it gets used fairly soon, and corruption occurs. > > Andrew, what is the right thing to do here? We could flail around looking > for explanations as to why the bug causes a user buffer corruption but never > get an answer or do we go with this patch, preferably before 2.6.36 releases? I've been flailing around quite a bit. Here's my latest result: Since I can view the corruption with md5sum /sbin/e2fsck, I know it's in a clean cached page. So I made an extra copy of /sbin/e2fsck, which won't be loaded into memory during boot. So now after the corruption happens, I can cmp -l /sbin/e2fsck good-e2fsck for a quick look at the changed bytes. Much easier than provoking a segfault under gdb. Then I got really creative and wrote a cmp replacement which mmaps the files and reports the physical addresses from /proc/self/pagemap of the pages that don't match. And the consistent result is that physical pages 64604 and 64609 (both in the range of the order=9 64512) have wrong contents. And the corruption is always a single word 128 bytes after the start of the page. Physical addresses 0x0fc5c080 and 0x0fc61080 are hit every time. The values of the corrupted words, observed in 5 consecutive boots, were: at 0fc5c080 at 0fc61080 --- --- c354 9251 565c 2359 c85b 9758 d15f 9e5c d95b a858 The low 16 bits are all 0 and the upper 16 bits seem randomly distributed. But look at the differences: c354 - 9251 = 3103 565c - 2359 = 3303 c85b - 9758 = 3103 d15f - 9e5c = 3303 d95b - a858 = 3103 This means something... but I don't know what. In a completely different method of investigation, I went back a few stable kernels, got 2.6.33.7 and applied 6dda9d55 to it, thinking that if 6dda9d55 only reveals a pre-existing bug, I could bisect it using 6dda9d55 as a bug-revealing assistant. The bug appeared when running 2.6.33.7 with 6dda9d55 applied. That was discouraging. >This patch fixes the problem by ensuring we are not reading a possibly >invalid location of memory. It's not clear why the read causes >corruption but one way or the other it is a buggy read. At least that part of the explanation is wrong. Where's the buggy read? The action taken by the 6dda9d55 version of __free_one_page looks perfectly legitimate to me. Page numbers: [129024 ] [130048 ] order=10 [129024 129536] [130048 130560] order=9 130048 is being freed. 130560 is not free. 129024 (the higher_buddy) is already free at order=10. So 130048 is being pushed to the tail of the free list, on the speculation that 130560 might soon be free and then the whole thing will form an order=11 free page, the only problem being that order=11 is too high so that later merge will never happen. It's not useful, and maybe not conceptually valid to say that 129024 is the buddy of 130048, but it is an existing page, and the only way it wouldn't be is if the total memory size was not a multiple of 1<<(MAX_ORDER-1) pages -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: PROBLEM: memory corrupting bug, bisected to 6dda9d55
Mel Gorman writes: > > On Mon, Oct 11, 2010 at 02:00:39PM -0700, Andrew Morton wrote: > > > > It's corruption of user memory, which is unusual. I'd be wondering if > > there was a pre-existing bug which 6dda9d55bf545013597 has exposed - > > previously the corruption was hitting something harmless. Something > > like a missed CPU cache writeback or invalidate operation. > > > > This seems somewhat plausible although it's hard to tell for sure. But > lets say we had the following situation in memory > > [][ ] > INITRDmemmap array I don't use initrd, so this isn't exactly what happened here. But it could be close. Let me throw out some more information and see if it triggers any ideas. First, I tried a new test after seeing the corruption happen: # md5sum /sbin/e2fsck ; echo 1 > /proc/sys/vm/drop_caches ; md5sum /sbin/e2fsck And got 2 different answers. The second answer was the correct one. Since applying the suggested patch which changed MAX_ORDER-1 to MAX_ORDER-2, I've been trying to isolate exactly when the corruption happens. Since I don't know much about kernel code, my main method is stuffing the area full of printk's. First I duplicated the affected function __free_one_page, since it's inlined at 2 different places, so I could apply the patch to just one of them. This proved that the problem is happening when called from free_one_page. The patch which fixes (or at least covers up) the bug will only matter when order==MAX_ORDER-2, otherwise everything is the same. So I added a lot of printk's to show what's happening when order==MAX_ORDER-2. I found that, very repeatably, 126 such instances occur during boot, and 61 of them pass the page_is_buddy(higher_page, higher_buddy, order + 1) test, causing them to call list_add_tail. Next, since the bug appears when this code decides to call list_add_tail, I made my own wrapper for list_add_tail, which allowed me to force some of the calls to do list_add instead. Eventually I found that of the 61 calls, the last one makes the difference. Allowing the first 60 calls to go through to list_add_tail, and switching the last one to list_add, the symptom goes away. dump_stack() for that last call gave me a backtrace like this: [c0303e80] [c0008124] show_stack+0x4c/0x144 (unreliable) [c0303ec0] [c0068a84] free_one_page+0x28c/0x5b0 [c0303f20] [c0069588] __free_pages_ok+0xf8/0x120 [c0303f40] [c02d28c8] free_all_bootmem_core+0xf0/0x1f8 [c0303f70] [c02d29fc] free_all_bootmem+0x2c/0x6c [c0303f90] [c02cc7dc] mem_init+0x70/0x2ac [c0303fc0] [c02c66a4] start_kernel+0x150/0x27c [c0303ff0] [3438] 0x3438 And this might be interesting: the PFN of the page being added in that critical 61st call is 130048, which exactly matches the number of available pages: free_area_init_node: node 0, pgdat c02fee6c, node_mem_map c033 DMA zone: 1024 pages used for memmap DMA zone: 0 pages reserved DMA zone: 130048 pages, LIFO batch:31 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 130048 Suspicious? If 130048 is added to the head of the order==MAX_ORDER-2 free list, there's no symptom. Add it to the tail, and the corruption appears. That's all I know so far. -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: Pegasos i8042 broken again
Benjamin Herrenschmidt writes: > > Those things really suck. They absolutely refuse to fix their FW for > reasons I never quite managed to figure out. The last time around, they did release a firmware patch (pegasos-dts-20071018) to fix up the device tree enough to satisfy the kernel. Now that the kernel has become dissatisfied again, maybe another patch will appear. > > At this stage, I'd say the best is to add yet another pegasos workaround > in prom_init that adds the missing compatible property. This one would be more complex than the other fixes in prom_init. It's not just the compatible property that's missing. The 8042 node in the device tree has no children. -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Pegasos i8042 broken again
Pegasos has no keyboard again. I blame commit 540c6c392f01887dcc96bef0a41e63e6c1334f01, which tries to find i8042 IRQs in the device-tree but doesn't fall back to the old hardcoded 1 and 12 in all failure cases. Specifically, the case where the device-tree contains nothing matching pnpPNP,303 or pnpPNP,f03 doesn't seem to be handled well. It sort of falls through to the old code, but leaves the IRQs set to 0. The last time something like this happened, I submitted a patch: http://lists.ozlabs.org/pipermail/linuxppc-dev/2007-July/039988.html which got committed, but afterward I was scolded for working around a bug instead of fixing it in nvramrc. This time I just won't send my workaround patch, at least until it's decided that the kernel should be made to understand the device-tree as is. If it's decided instead that the firmware should be patched... well I just don't feel comfortable inventing my own patch for nvramrc, since it's written in a language I don't know and presumably could brick the machine if I get it wrong. Also I'm not even sure what the kernel is expecting to find there. -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
Re: "event-scan failed" logflood
Benjamin Herrenschmidt writes: > > Well, first it should be called once per second, not 60 times per > second, so something is wrong there... Actually I think it was happening a lot more than 60 times per second, and klogd was losing most of the messages because they came too fast. When running the new kernel, vmstat shows 17 context switches per second (and CPU usage never going below 5%) even when everything should be idle. My next experiment was to hack rtas_init to return 0 unconditionally. Now everything still runs fine, and the context switches per second are back down in the 2-digit range. > > Then, it wouldn't surprise me if the Pegasos RTAS didn't implement > the event scan properly. Maybe we failed silently before that ? I > doubt it's necessary there anyways. If I'm reading things correctly, rtas-event-scan-rate is the key to the frequency of attempts. And rtasd.c didn't expect this: $ od -t x1 /proc/device-tree/rtas/rtas-event-scan-rate 000 00 00 00 00 004 -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
"event-scan failed" logflood
I upgraded the kernel on my Pegasos from 2.6.32 to 2.6.33 and now it sends the message "event-scan failed" to the kernel log about 60 times per second as long as it's running. The message comes from arch/powerpc/kernel/rtasd.c but I don't know what's going on in there so I can't say much more about it. How may I help find the cause of this problem? (Or alternatively: what is the correct way to disable this "event-scan"? I suspect it's not really necessary since it didn't exist in the previous kernel version, and the system is running perfectly aside from the error message flood.) -- Alan Curry ___ Linuxppc-dev mailing list Linuxppc-dev@lists.ozlabs.org https://lists.ozlabs.org/listinfo/linuxppc-dev
