Re: panic: Memory modified after free
Thanks again for looking at this problem Doug White wrote: > On Thu, 23 Oct 2003, othermark wrote: > Onboard fiber? What kind of system is this? They're wired to the board. I'd probably break the connector if I remove it. This box has custom hardware attached, I don't expect any of the drivers to attach (with exception of the std onboard ethernet) because of this. I do want -current to come up so I can begin driver twiddling. >> > That or perhaps you have bad memory. Do you have ECC RAM in the >> > system? I found some and turned on bios ecc logging. Same panic, no ECC errors corrections. > I suspect the actual last user is irrelevant; its a leaking pointer > reference somewhere and the memory allocator is handing the memory block > it points to back out to some innocent bystander who triggers the panic. > > Have you emailed the em driver maintainer yet? Based on my later replies - October 16th boots fine, and October 17th snapshot b0rks on this panic, I'm not convinced the em driver is at fault. I will recompile w/o em in the kernel to test this theory. -- othermark atkin901 at nospam dot yahoo dot com (!wired)?(coffee++):(wired); ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: panic: Memory modified after free
On Thu, 23 Oct 2003, othermark wrote: > these are fibre 1000 base sx connections. They don't attach correctly in > the 5.0-release kernel as well (with the exact same error), but it does > continue to boot correctly. These are hardwired into the bus, and I'm > unable to disable them. :( Onboard fiber? What kind of system is this? > > That or perhaps you have bad memory. Do you have ECC RAM in the system? > > I'm not positive, so I'm going to say no, but I'm also fairly sure that > the memory is good. I ran make buildworld on 5.0 successfully w/o any > problems. Slow bios memcheck at startup is good. That memcheck is useless, sadly. You might track down a copy of memtest86 and run it on your system just to be sure. Its a much more intensive diagnostic. > this seems similar to: > http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/53566 > > except the last user is of memory is different. I suspect the actual last user is irrelevant; its a leaking pointer reference somewhere and the memory allocator is handing the memory block it points to back out to some innocent bystander who triggers the panic. > I think the next step is to move up to a 5.1-release kernel and see if > it boots as well as the 5.0-release does, or provides a more interesting > panic. Have you emailed the em driver maintainer yet? -- Doug White| FreeBSD: The Power to Serve [EMAIL PROTECTED] | www.FreeBSD.org ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: panic: Memory modified after free
Hi, thanks for taking a gander at my problem. The original panic can be reviewed here: http://article.gmane.org/gmane.os.freebsd.current/31913 now to answer your query... Doug Rabson wrote: > On Thu, 2003-10-23 at 22:45, othermark wrote: >> I wrote: >> > I will try seeing how far I can go up the list of snapshots until I >> > encounter the first boot -s panic. >> >> Well I walked up the available snapshots and the first panic occurs with >> the snapshot from the 17th of October. Reviewing the commit logs between >> the 16th and the 17th I note the following commits are the most >> 'interesting.' as related to this panic.. This is just a cursory look >> at the logs, I haven't gotten into compiling and fingering an exact >> commit yet (which takes loads of time). >> >> dfr 2003/10/16 02:16:28 PDT >> >> FreeBSD src repository >> >> Modified files: >> sys/sys bus.h kobj.h param.h >> sys/kern subr_bus.c subr_kobj.c >> Log: >> * Add multiple inheritance to kobj. > > I haven't had any other reports of breakage related to this. Is it > possible that you are using a kernel module which you have not re-built > after this date (e.g. nvidia.ko)? I'm not loading any modules with the single user boot 'boot -s'. (kldstat shows no modules, just 'kernel'). In fact I only downloaded the 'kernel' file for each snapshot off current.freebsd.org, placed it in it's own directory under /boot and referenced it explicitly at the boot prompt. Beginning at the oct 17th snapshot, I got the same panic as referenced in my original post to the list. Does anyone else have a box with several legacy isa pnp cards or embedded devices that can try to boot up -current from after the 17th? -- othermark atkin901 at nospam dot yahoo dot com (!wired)?(coffee++):(wired); ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: panic: Memory modified after free
On Thu, 2003-10-23 at 22:45, othermark wrote: > I wrote: > > I will try seeing how far I can go up the list of snapshots until I > > encounter the first boot -s panic. > > Well I walked up the available snapshots and the first panic occurs with > the snapshot from the 17th of October. Reviewing the commit logs between > the 16th and the 17th I note the following commits are the most > 'interesting.' as related to this panic.. This is just a cursory look > at the logs, I haven't gotten into compiling and fingering an exact commit > yet (which takes loads of time). > > dfr 2003/10/16 02:16:28 PDT > > FreeBSD src repository > > Modified files: > sys/sys bus.h kobj.h param.h > sys/kern subr_bus.c subr_kobj.c > Log: > * Add multiple inheritance to kobj. I haven't had any other reports of breakage related to this. Is it possible that you are using a kernel module which you have not re-built after this date (e.g. nvidia.ko)? ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: panic: Memory modified after free
I wrote: > I will try seeing how far I can go up the list of snapshots until I > encounter the first boot -s panic. Well I walked up the available snapshots and the first panic occurs with the snapshot from the 17th of October. Reviewing the commit logs between the 16th and the 17th I note the following commits are the most 'interesting.' as related to this panic.. This is just a cursory look at the logs, I haven't gotten into compiling and fingering an exact commit yet (which takes loads of time). dfr 2003/10/16 02:16:28 PDT FreeBSD src repository Modified files: sys/sys bus.h kobj.h param.h sys/kern subr_bus.c subr_kobj.c Log: * Add multiple inheritance to kobj. ... dfr 2003/10/16 02:18:36 PDT FreeBSD src repository Modified files: sys/i386/isa isa_compat.c Log: Add a workaround for the fact that the priv field was removed ... bde 2003/10/16 03:44:24 PDT FreeBSD src repository Modified files: sys/i386/isa apic_vector.s Log: Don't forget to load %es with the kernel data segment selector in Xcpustop(). ... -- othermark atkin901 at nospam dot yahoo dot com (!wired)?(coffee++):(wired); ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: panic: Memory modified after free
apologies for repling to my own post, but it seemed the best way to continue the thread. othermark wrote: > I think the next step is to move up to a 5.1-release kernel and see if > it boots as well as the 5.0-release does, or provides a more interesting > panic. I tried a 5.1-RELEASE kernel and booted successfully. To take it a step further, I tried an ftp'd kernel from current.freebsd.org 5.1-CURRENT-20031009-JPSNAP and I was able to bootstrap the box into single user mode. If I bootstrap the box into multiuser (snapshot kernel + 5.0 userland) I get the following panic: Memory modified after free 0xc4987800(2044) val=c4986800 @ 0xc4987950 panic: Most recently used by bus Debugger("panic") Stopped at Debugger+0x54: xchgl %ebx,in_Debugger.0 db> where Debugger(c083db04,c08ffbc0,c0855049,d782662c,100) at Debugger+0x54 panic(c0855049,c081f6e0,7fc,c4986800,c4987950) at panic+0xd5 mtrash_ctor(c4987800,800,0,583,c4987800) at mtrash_ctor+0x67 uma_zalloc_arg(c103ae40,0,2,c08feb04,0) at uma_zalloc_arg+0x1ce malloc(800,c0899c40,2,a7c,c0843763) at malloc+0xd3 allocbuf(cec0ef88,800,c0843763,a31,4000) at allocbuf+0x202 getblk(c49d136c,0,0,800,0) at getblk+0x4d6 breadn(c49d136c,0,0,800,0) at breadn+0x52 bread(c49d136c,0,0,800,0) at bread+0x4c ffs_blkatoff(c49d136c,0,0,0,d7826888) at ffs_blkatoff+0xcf ufs_lookup(d7826948,d7826984,c0685211,d7826948,d7826bec) at ufs_lookup+0x393 ufs_vnoperate(d7826948,d7826bec,d7826c00,c0844f5d,c1d05390) at ufs_vnoperat +0x18 vfs_cache_lookup(d78269c8,d78269e4,c068a2b2,d78269c8,20002) at vfs_cache_lookup+0x301 ufs_vnoperate(d78269c8,20002,c1d05390,c062d9a0,c1d05390) at ufs_vnoperat +0x18 lookup(d7826bd8,0,c0844896,a6,c1d05390) at lookup+0x302 namei(d7826bd8,0,c09091e0,3,c1d05390) at namei+0x24e vn_open_cred(d7826bd8,d7826cd8,0,c1cfbe00,9) at vn_open_cred+0x251 vn_open(d7826bd8,d7826cd8,0,9,c083b124) at vn_open+0x30 kern_open(c1d05390,bfbfefb0,0,a,0) at kern_open+0x140 open(c1d05390,d7826d10,c08590bb,3ec,3) at open+0x30 syscall(2f,2f,2f,bfbfefaf,bfbfdde4) at syscall+0x273 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (5, FreeBSD ELF32, open), eip = 0x280b6973, esp = 0xbfbfdd3c, ebp = 0xbfbfe218 --- db> I will try seeing how far I can go up the list of snapshots until I encounter the first boot -s panic. -- othermark atkin901 at nospam dot yahoo dot com (!wired)?(coffee++):(wired); ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: panic: Memory modified after free
Thanks for looking at this. I'm still scratching my head on it. Doug White wrote: > On Mon, 20 Oct 2003, othermark wrote: >> I have a strange panic during the isa pnp code that does not occur with a >> 5.0-release kernel. ... > Can you pull out or disable the gig-e card? Its having trouble > initializing, and I'm wondering if its doing something bad in the process. these are fibre 1000 base sx connections. They don't attach correctly in the 5.0-release kernel as well (with the exact same error), but it does continue to boot correctly. These are hardwired into the bus, and I'm unable to disable them. :( > That or perhaps you have bad memory. Do you have ECC RAM in the system? I'm not positive, so I'm going to say no, but I'm also fairly sure that the memory is good. I ran make buildworld on 5.0 successfully w/o any problems. Slow bios memcheck at startup is good. > Here is the panic again: > >> Memory modified after free 0xc4758800(2044) val=c4756800 @ 0xc47589dc >> panic: Most recently used by bus-sc this seems similar to: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/53566 except the last user is of memory is different. Speaking of memory 5.0 - release: real memory = 536870912 (512 MB) Physical memory chunk(s): ! 0x1000 - 0x0009efff, 647168 bytes (158 pages) ! 0x0064f000 - 0x1fff7fff, 530223104 bytes (129449 pages) ! avail memory = 515031040 (491 MB) -current: real memory = 536870912 (512 MB) Physical memory chunk(s): ! 0x1000 - 0x0009efff, 647168 bytes (158 pages) ! 0x0010 - 0x003f, 3145728 bytes (768 pages) ! 0x00c26000 - 0x1f6d9fff, 514539520 bytes (125620 pages) ! avail memory = 511942656 (488 MB) I think the next step is to move up to a 5.1-release kernel and see if it boots as well as the 5.0-release does, or provides a more interesting panic. -- othermark atkin901 at nospam dot yahoo dot com (!wired)?(coffee++):(wired); ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: panic: Memory modified after free
On Mon, 20 Oct 2003, othermark wrote: > I have a strange panic during the isa pnp code that does not occur with a > 5.0-release kernel. I have tried enabling and disabling acpi. it does > not effect this panic one way or another. This is a kernel from -current > 10/20 (today). I'm not sure how to get this to boot with no way to disable > pnp probing (pnpbios(4)). Can you pull out or disable the gig-e card? Its having trouble initializing, and I'm wondering if its doing something bad in the process. That or perhaps you have bad memory. Do you have ECC RAM in the system? Here is the failed em attach: > em0: mem > 0xfeae-0xf > eaf irq 5 at device 0.0 on pci1 > em0: [MPSAFE] > em0: Hardware Initialization Failedem0: Unable to initialize the hardware > device_probe_and_attach: em0 attach returned 5 The other em failing (intel motherboard?): > em0: mem > 0xfebe-0xf > ebf irq 9 at device 1.0 on pci2 > em0: [MPSAFE] > em0: Hardware Initialization Failedem0: Unable to initialize the hardware > device_probe_and_attach: em0 attach returned 5 Here is the panic again: > Memory modified after free 0xc4758800(2044) val=c4756800 @ 0xc47589dc > panic: Most recently used by bus-sc > > Debugger("panic") > Stopped at Debugger+0x54: xchgl %ebx,in_Debugger.0 > db> where > Debugger(c083c6e1,c08fe300,c0853cc0,c0c21b4c,100) at Debugger+0x54 > panic(c0853cc0,c083dd01,7fc,c4756800,c47589dc) at panic+0xd5 > mtrash_ctor(c4758800,800,0,583,c4758800) at mtrash_ctor+0x67 > uma_zalloc_arg(c103ae40,0,1,2c21bbc,c0891040) at uma_zalloc_arg+0x1ce > malloc(7ec,c0891040,1,c473dc80,c478f000) at malloc+0xd3 > isa_add_config(c4765b00,c478d280,0,c478f000,c478f000) at isa_add_config+0x33 > pnp_parse_resources(c478d280,c478e30e,19,0,c478e302) at pnp_parse_resource > +0x3b8 > pnpbios_identify(c08d0db4,c4765b00,c0863280,c085d008,c08caab0) at > pnpbios_identify+0x43f > bus_generic_probe(c4765b00,c0c21d5c,c064f78e,c1cfd180,c474904c) at > bus_generic_probe+0x62 > isa_probe_children(c4765b00,c08570dd,0,c0c21d98,c0610455) at > isa_probe_children+0x14 > configure(0,c1e000,c1ec00,c1e000,0) at configure+0x4b > mi_startup() at mi_startup+0xb5 > begin() at begin+0x2c > db> > > > -- Doug White| FreeBSD: The Power to Serve [EMAIL PROTECTED] | www.FreeBSD.org ___ [EMAIL PROTECTED] mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-current To unsubscribe, send any mail to "[EMAIL PROTECTED]"
Re: Panic: memory modified after free
On Fri, 20 Dec 2002, Gavin Atkinson wrote: > > Running 5.0-RC as of yesterday on i386. background fsck was in progress, > > but other than that system was idle. Logged in as root on the console, had > > cd'd to a ports directory. (note that it panics almost instantly when > > using the console, but lasts upto 10 minutes when in use over ssh) > > Running "make deinstall" triggered this panic: > > > > Memory modified after free 0xc1891c00(1020) > > panic: Most recently used by none > > Update: I re-cvsupped (to 19 Dec 14:00 GMT) , and recompiled world and > kernel. I can no longer cause the panic. I then (out of interest) dropped > back to the old kernel that was panicing (18 Dec 12:00 GMT), but run with > the new world, and could not recreate the panic. I therefore believe that > one of the userland binaries that I replaced was tickling the bug, and now > I have replaced that binary, the problem no longer occurs. There were several bugs fixed between those dates, most importantly this one. It is unlikely the userland changes that day could mask a bug like that. mckusick2002/12/18 11:50:28 PST Modified files: sys/ufs/ffs ffs_snapshot.c Log: Fix corruption introduced in previous delta. Reported by:Aurelien Nephtali <[EMAIL PROTECTED]> Sponsored by: DARPA & NAI Labs. Revision ChangesPath 1.57 +12 -4 src/sys/ufs/ffs/ffs_snapshot.c To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
Re: Panic: memory modified after free
On Thu, 19 Dec 2002, Gavin Atkinson wrote: > Running 5.0-RC as of yesterday on i386. background fsck was in progress, > but other than that system was idle. Logged in as root on the console, had > cd'd to a ports directory. (note that it panics almost instantly when > using the console, but lasts upto 10 minutes when in use over ssh) > Running "make deinstall" triggered this panic: > > Memory modified after free 0xc1891c00(1020) > panic: Most recently used by none >[snip backtrace] > The machine seems perfectly stable in single user mode. It also seems > pretty stable at the moment with linux emulation, usbd, sendmail, ipv6, > nfs server and moused enables commented out of rc.conf. I will try to add > one at a time tonight to determine which is at fault. Update: I re-cvsupped (to 19 Dec 14:00 GMT) , and recompiled world and kernel. I can no longer cause the panic. I then (out of interest) dropped back to the old kernel that was panicing (18 Dec 12:00 GMT), but run with the new world, and could not recreate the panic. I therefore believe that one of the userland binaries that I replaced was tickling the bug, and now I have replaced that binary, the problem no longer occurs. So, unless anyone can think of a better reason for this, I suspect there is a kernel use-after-free bug laying dormant. Gavin To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message