Re: panic: Memory modified after free

2003-10-24 Thread othermark
Thanks again for looking at this problem

Doug White wrote:
> On Thu, 23 Oct 2003, othermark wrote:
> Onboard fiber? What kind of system is this?

They're wired to the board.  I'd probably break the connector if I remove
it.  This box has custom hardware attached, I don't expect any of the
drivers to attach (with exception of the std onboard ethernet) because
of this.  I do want -current to come up so I can begin driver twiddling.
 
>> > That or perhaps you have bad memory.  Do you have ECC RAM in the
>> > system?

I found some and turned on bios ecc logging.  Same panic, no ECC errors
corrections.

> I suspect the actual last user is irrelevant; its a leaking pointer
> reference somewhere and the memory allocator is handing the memory block
> it points to back out to some innocent bystander who triggers the panic.
>
> Have you emailed the em driver maintainer yet?

Based on my later replies - October 16th boots fine, and October 17th
snapshot b0rks on this panic, I'm not convinced the em driver is at fault.
I will recompile w/o em in the kernel to test this theory.

-- 
othermark
atkin901 at nospam dot yahoo dot com
(!wired)?(coffee++):(wired);

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: panic: Memory modified after free

2003-10-24 Thread Doug White
On Thu, 23 Oct 2003, othermark wrote:

> these are fibre 1000 base sx connections.  They don't attach correctly in
> the 5.0-release kernel as well (with the exact same error), but it does
> continue to boot correctly.  These are hardwired into the bus, and I'm
> unable to disable them. :(

Onboard fiber? What kind of system is this?

> > That or perhaps you have bad memory.  Do you have ECC RAM in the system?
>
> I'm not positive, so I'm going to say no, but I'm also fairly sure that
> the memory is good.  I ran make buildworld on 5.0 successfully w/o any
> problems.  Slow bios memcheck at startup is good.

That memcheck is useless, sadly.  You might track down a copy of memtest86
and run it on your system just to be sure. Its a much more intensive
diagnostic.

> this seems similar to:
> http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/53566
>
> except the last user is of memory is different.

I suspect the actual last user is irrelevant; its a leaking pointer
reference somewhere and the memory allocator is handing the memory block
it points to back out to some innocent bystander who triggers the panic.

> I think the next step is to move up to a 5.1-release kernel and see if
> it boots as well as the 5.0-release does, or provides a more interesting
> panic.

Have you emailed the em driver maintainer yet?

-- 
Doug White|  FreeBSD: The Power to Serve
[EMAIL PROTECTED]  |  www.FreeBSD.org
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: panic: Memory modified after free

2003-10-24 Thread othermark
Hi, thanks for taking a gander at my problem.  The original panic
can be reviewed here:
http://article.gmane.org/gmane.os.freebsd.current/31913

now to answer your query...

Doug Rabson wrote:
> On Thu, 2003-10-23 at 22:45, othermark wrote:
>> I wrote:
>> > I will try seeing how far I can go up the list of snapshots until I
>> > encounter the first boot -s panic.
>> 
>> Well I walked up the available snapshots and the first panic occurs with
>> the snapshot from the 17th of October.  Reviewing the commit logs between
>> the 16th and the 17th I note the following commits are the most
>> 'interesting.' as related to this panic..   This is just a cursory look
>> at the logs, I haven't gotten into compiling and fingering an exact
>> commit yet (which takes loads of time).
>> 
>> dfr 2003/10/16 02:16:28 PDT
>> 
>>   FreeBSD src repository
>> 
>>   Modified files:
>> sys/sys  bus.h kobj.h param.h
>> sys/kern subr_bus.c subr_kobj.c
>>   Log:
>>   * Add multiple inheritance to kobj.
> 
> I haven't had any other reports of breakage related to this. Is it
> possible that you are using a kernel module which you have not re-built
> after this date (e.g. nvidia.ko)?

I'm not loading any modules with the single user boot 'boot -s'. (kldstat
shows no modules, just 'kernel'). In fact I only downloaded the 'kernel'
file for each snapshot off current.freebsd.org, placed it in it's own
directory under /boot and referenced it explicitly at the boot prompt. 
Beginning at the oct 17th snapshot, I got the same panic as referenced in
my original post to the list.

Does anyone else have a box with several legacy isa pnp cards or embedded
devices that can try to boot up -current from after the 17th?  

-- 
othermark
atkin901 at nospam dot yahoo dot com
(!wired)?(coffee++):(wired);

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: panic: Memory modified after free

2003-10-24 Thread Doug Rabson
On Thu, 2003-10-23 at 22:45, othermark wrote:
> I wrote:
> > I will try seeing how far I can go up the list of snapshots until I
> > encounter the first boot -s panic.
> 
> Well I walked up the available snapshots and the first panic occurs with
> the snapshot from the 17th of October.  Reviewing the commit logs between
> the 16th and the 17th I note the following commits are the most
> 'interesting.' as related to this panic..   This is just a cursory look
> at the logs, I haven't gotten into compiling and fingering an exact commit
> yet (which takes loads of time).
> 
> dfr 2003/10/16 02:16:28 PDT
> 
>   FreeBSD src repository
> 
>   Modified files:
> sys/sys  bus.h kobj.h param.h 
> sys/kern subr_bus.c subr_kobj.c 
>   Log:
>   * Add multiple inheritance to kobj.

I haven't had any other reports of breakage related to this. Is it
possible that you are using a kernel module which you have not re-built
after this date (e.g. nvidia.ko)?


___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: panic: Memory modified after free

2003-10-23 Thread othermark
I wrote:
> I will try seeing how far I can go up the list of snapshots until I
> encounter the first boot -s panic.

Well I walked up the available snapshots and the first panic occurs with
the snapshot from the 17th of October.  Reviewing the commit logs between
the 16th and the 17th I note the following commits are the most
'interesting.' as related to this panic..   This is just a cursory look
at the logs, I haven't gotten into compiling and fingering an exact commit
yet (which takes loads of time).

dfr 2003/10/16 02:16:28 PDT

  FreeBSD src repository

  Modified files:
sys/sys  bus.h kobj.h param.h 
sys/kern subr_bus.c subr_kobj.c 
  Log:
  * Add multiple inheritance to kobj.
...
dfr 2003/10/16 02:18:36 PDT

  FreeBSD src repository

  Modified files:
sys/i386/isa isa_compat.c 
  Log:
  Add a workaround for the fact that the priv field was removed
...
bde 2003/10/16 03:44:24 PDT

  FreeBSD src repository

  Modified files:
sys/i386/isa apic_vector.s 
  Log:
  Don't forget to load %es with the kernel data segment selector in
  Xcpustop().
...


-- 
othermark
atkin901 at nospam dot yahoo dot com
(!wired)?(coffee++):(wired);

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: panic: Memory modified after free

2003-10-23 Thread othermark
apologies for repling to my own post, but it seemed the best way to continue
the thread.

othermark wrote:
> I think the next step is to move up to a 5.1-release kernel and see if
> it boots as well as the 5.0-release does, or provides a more interesting
> panic.

I tried a 5.1-RELEASE kernel and booted successfully.  To take it a step
further, I tried an ftp'd kernel from current.freebsd.org 

5.1-CURRENT-20031009-JPSNAP

and I was able to bootstrap the box into single user mode.

If I bootstrap the box into multiuser (snapshot kernel + 5.0 userland) I get
the following panic:

Memory modified after free 0xc4987800(2044) val=c4986800 @ 0xc4987950
panic: Most recently used by bus
Debugger("panic")
Stopped at  Debugger+0x54:  xchgl   %ebx,in_Debugger.0
db> where
Debugger(c083db04,c08ffbc0,c0855049,d782662c,100) at Debugger+0x54
panic(c0855049,c081f6e0,7fc,c4986800,c4987950) at panic+0xd5
mtrash_ctor(c4987800,800,0,583,c4987800) at mtrash_ctor+0x67
uma_zalloc_arg(c103ae40,0,2,c08feb04,0) at uma_zalloc_arg+0x1ce
malloc(800,c0899c40,2,a7c,c0843763) at malloc+0xd3
allocbuf(cec0ef88,800,c0843763,a31,4000) at allocbuf+0x202
getblk(c49d136c,0,0,800,0) at getblk+0x4d6
breadn(c49d136c,0,0,800,0) at breadn+0x52
bread(c49d136c,0,0,800,0) at bread+0x4c
ffs_blkatoff(c49d136c,0,0,0,d7826888) at ffs_blkatoff+0xcf
ufs_lookup(d7826948,d7826984,c0685211,d7826948,d7826bec) at ufs_lookup+0x393
ufs_vnoperate(d7826948,d7826bec,d7826c00,c0844f5d,c1d05390) at ufs_vnoperat
+0x18
vfs_cache_lookup(d78269c8,d78269e4,c068a2b2,d78269c8,20002) at
vfs_cache_lookup+0x301
ufs_vnoperate(d78269c8,20002,c1d05390,c062d9a0,c1d05390) at ufs_vnoperat
+0x18
lookup(d7826bd8,0,c0844896,a6,c1d05390) at lookup+0x302
namei(d7826bd8,0,c09091e0,3,c1d05390) at namei+0x24e
vn_open_cred(d7826bd8,d7826cd8,0,c1cfbe00,9) at vn_open_cred+0x251
vn_open(d7826bd8,d7826cd8,0,9,c083b124) at vn_open+0x30
kern_open(c1d05390,bfbfefb0,0,a,0) at kern_open+0x140
open(c1d05390,d7826d10,c08590bb,3ec,3) at open+0x30
syscall(2f,2f,2f,bfbfefaf,bfbfdde4) at syscall+0x273
Xint0x80_syscall() at Xint0x80_syscall+0x1d
--- syscall (5, FreeBSD ELF32, open), eip = 0x280b6973, esp = 0xbfbfdd3c,
ebp = 0xbfbfe218 ---
db>

I will try seeing how far I can go up the list of snapshots until I
encounter the first boot -s panic.

-- 
othermark
atkin901 at nospam dot yahoo dot com
(!wired)?(coffee++):(wired);

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: panic: Memory modified after free

2003-10-23 Thread othermark
Thanks for looking at this.  I'm still scratching my head on it.

Doug White wrote:
> On Mon, 20 Oct 2003, othermark wrote:
>> I have a strange panic during the isa pnp code that does not occur with a
>> 5.0-release kernel.

...

> Can you pull out or disable the gig-e card?  Its having trouble
> initializing, and I'm wondering if its doing something bad in the process.

these are fibre 1000 base sx connections.  They don't attach correctly in 
the 5.0-release kernel as well (with the exact same error), but it does
continue to boot correctly.  These are hardwired into the bus, and I'm
unable to disable them. :(

> That or perhaps you have bad memory.  Do you have ECC RAM in the system?

I'm not positive, so I'm going to say no, but I'm also fairly sure that 
the memory is good.  I ran make buildworld on 5.0 successfully w/o any
problems.  Slow bios memcheck at startup is good.

> Here is the panic again:
> 
>> Memory modified after free 0xc4758800(2044) val=c4756800 @ 0xc47589dc
>> panic: Most recently used by bus-sc

this seems similar to:
http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/53566

except the last user is of memory is different.

Speaking of memory

5.0 - release:

  real memory  = 536870912 (512 MB)
  Physical memory chunk(s):
! 0x1000 - 0x0009efff, 647168 bytes (158 pages)
! 0x0064f000 - 0x1fff7fff, 530223104 bytes (129449 pages)
! avail memory = 515031040 (491 MB)

-current:
  real memory  = 536870912 (512 MB)
  Physical memory chunk(s):
! 0x1000 - 0x0009efff, 647168 bytes (158 pages)
! 0x0010 - 0x003f, 3145728 bytes (768 pages)
! 0x00c26000 - 0x1f6d9fff, 514539520 bytes (125620 pages)
! avail memory = 511942656 (488 MB)

I think the next step is to move up to a 5.1-release kernel and see if
it boots as well as the 5.0-release does, or provides a more interesting
panic.

-- 
othermark
atkin901 at nospam dot yahoo dot com
(!wired)?(coffee++):(wired);

___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: panic: Memory modified after free

2003-10-22 Thread Doug White
On Mon, 20 Oct 2003, othermark wrote:

> I have a strange panic during the isa pnp code that does not occur with a
> 5.0-release kernel.  I have tried enabling and disabling acpi.  it does
> not effect this panic one way or another.  This is a kernel from -current
> 10/20 (today).  I'm not sure how to get this to boot with no way to disable
> pnp probing (pnpbios(4)).

Can you pull out or disable the gig-e card?  Its having trouble
initializing, and I'm wondering if its doing something bad in the process.

That or perhaps you have bad memory.  Do you have ECC RAM in the system?

Here is the failed em attach:

> em0:  mem
> 0xfeae-0xf
> eaf irq 5 at device 0.0 on pci1
> em0: [MPSAFE]
> em0: Hardware Initialization Failedem0: Unable to initialize the hardware
> device_probe_and_attach: em0 attach returned 5

The other em failing (intel motherboard?):

> em0:  mem
> 0xfebe-0xf
> ebf irq 9 at device 1.0 on pci2
> em0: [MPSAFE]
> em0: Hardware Initialization Failedem0: Unable to initialize the hardware
> device_probe_and_attach: em0 attach returned 5

Here is the panic again:

> Memory modified after free 0xc4758800(2044) val=c4756800 @ 0xc47589dc
> panic: Most recently used by bus-sc
>
> Debugger("panic")
> Stopped at  Debugger+0x54:  xchgl   %ebx,in_Debugger.0
> db> where
> Debugger(c083c6e1,c08fe300,c0853cc0,c0c21b4c,100) at Debugger+0x54
> panic(c0853cc0,c083dd01,7fc,c4756800,c47589dc) at panic+0xd5
> mtrash_ctor(c4758800,800,0,583,c4758800) at mtrash_ctor+0x67
> uma_zalloc_arg(c103ae40,0,1,2c21bbc,c0891040) at uma_zalloc_arg+0x1ce
> malloc(7ec,c0891040,1,c473dc80,c478f000) at malloc+0xd3
> isa_add_config(c4765b00,c478d280,0,c478f000,c478f000) at isa_add_config+0x33
> pnp_parse_resources(c478d280,c478e30e,19,0,c478e302) at pnp_parse_resource
> +0x3b8
> pnpbios_identify(c08d0db4,c4765b00,c0863280,c085d008,c08caab0) at
> pnpbios_identify+0x43f
> bus_generic_probe(c4765b00,c0c21d5c,c064f78e,c1cfd180,c474904c) at
> bus_generic_probe+0x62
> isa_probe_children(c4765b00,c08570dd,0,c0c21d98,c0610455) at
> isa_probe_children+0x14
> configure(0,c1e000,c1ec00,c1e000,0) at configure+0x4b
> mi_startup() at mi_startup+0xb5
> begin() at begin+0x2c
> db>
>
>
>

-- 
Doug White|  FreeBSD: The Power to Serve
[EMAIL PROTECTED]  |  www.FreeBSD.org
___
[EMAIL PROTECTED] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "[EMAIL PROTECTED]"


Re: Panic: memory modified after free

2002-12-19 Thread Nate Lawson
On Fri, 20 Dec 2002, Gavin Atkinson wrote:
> > Running 5.0-RC as of yesterday on i386. background fsck was in progress,
> > but other than that system was idle. Logged in as root on the console, had
> > cd'd to a ports directory. (note that it panics almost instantly when
> > using the console, but lasts upto 10 minutes when in use over ssh)
> > Running "make deinstall" triggered this panic:
> >
> > Memory modified after free 0xc1891c00(1020)
> > panic: Most recently used by none
> 
> Update: I re-cvsupped (to 19 Dec 14:00 GMT) , and recompiled world and
> kernel. I can no longer cause the panic. I then (out of interest) dropped
> back to the old kernel that was panicing (18 Dec 12:00 GMT), but run with
> the new world, and could not recreate the panic. I therefore believe that
> one of the userland binaries that I replaced was tickling the bug, and now
> I have replaced that binary, the problem no longer occurs.

There were several bugs fixed between those dates, most importantly this
one.  It is unlikely the userland changes that day could mask a bug like
that.

mckusick2002/12/18 11:50:28 PST

  Modified files:
sys/ufs/ffs  ffs_snapshot.c 
  Log:
  Fix corruption introduced in previous delta.
  
  Reported by:Aurelien Nephtali <[EMAIL PROTECTED]>
Sponsored by:   DARPA & NAI Labs.
  
  Revision  ChangesPath
  1.57  +12 -4 src/sys/ufs/ffs/ffs_snapshot.c




To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message



Re: Panic: memory modified after free

2002-12-19 Thread Gavin Atkinson
On Thu, 19 Dec 2002, Gavin Atkinson wrote:

> Running 5.0-RC as of yesterday on i386. background fsck was in progress,
> but other than that system was idle. Logged in as root on the console, had
> cd'd to a ports directory. (note that it panics almost instantly when
> using the console, but lasts upto 10 minutes when in use over ssh)
> Running "make deinstall" triggered this panic:
>
> Memory modified after free 0xc1891c00(1020)
> panic: Most recently used by none
>[snip backtrace]
> The machine seems perfectly stable in single user mode. It also seems
> pretty stable at the moment with linux emulation, usbd, sendmail, ipv6,
> nfs server and moused enables commented out of rc.conf. I will try to add
> one at a time tonight to determine which is at fault.

Update: I re-cvsupped (to 19 Dec 14:00 GMT) , and recompiled world and
kernel. I can no longer cause the panic. I then (out of interest) dropped
back to the old kernel that was panicing (18 Dec 12:00 GMT), but run with
the new world, and could not recreate the panic. I therefore believe that
one of the userland binaries that I replaced was tickling the bug, and now
I have replaced that binary, the problem no longer occurs.

So, unless anyone can think of a better reason for this, I suspect there
is a kernel use-after-free bug laying dormant.

Gavin

To Unsubscribe: send mail to [EMAIL PROTECTED]
with "unsubscribe freebsd-current" in the body of the message