...looks like a problem with cleaning up an mmap bucket. This is from /usr/local/apache2.0.35/corefiles/httpd.core.3 ; .4 and .5 are the same problem.
#0 apr_pool_cleanup_kill (p=0x8152f08, data=0x8152eb8, cleanup_fn=0x280cc700 <mmap_cleanup>) at apr_pools.c:1669 #1 0x280cc90a in apr_mmap_delete (mm=0x8152eb8) at mmap.c:210 #2 0x280ad926 in mmap_destroy (data=0x8131298) at apr_buckets_mmap.c:82 #3 0x280adf08 in apr_brigade_cleanup (data=0x8134ca8) at apr_brigade.c:86 #4 0x280adebe in brigade_cleanup (data=0x8134ca8) at apr_brigade.c:72 #5 0x280cdd3b in run_cleanups (c=0x813cb98) at apr_pools.c:1713 #6 0x280cd51c in apr_pool_destroy (pool=0x814b010) at apr_pools.c:638 #7 0x280cd417 in apr_pool_clear (pool=0x812c010) at apr_pools.c:600 #8 0x8064752 in child_main (child_num_arg=291) at prefork.c:586 the mm looks whacked/previously deleted: (gdb) fr 1 #1 0x280cc90a in apr_mmap_delete (mm=0x8152eb8) at mmap.c:210 (gdb) p *mm $1 = {cntxt = 0x8152f08, mm = 0x0, size = 0, is_owner = 0} (gdb) p *mm.cntxt $2 = {parent = 0x8151010, child = 0x10, sibling = 0x2, ref = 0x28, cleanups = 0x8147010, allocator = 0x0, subprocesses = 0x0, abort_fn = 0, user_data = 0x0, tag = 0x0, active = 0x0, self = 0x0, self_first_avail = 0x0} ...but the bucket structures looks fine: (gdb) fr 3 #3 0x280adf08 in apr_brigade_cleanup (data=0x8134ca8) at apr_brigade.c:86 86 apr_bucket_delete(e); (gdb) dump_bucket e bucket=MMAP(0x8131190), length=475, data=0x8131298 (gdb) p *e $3 = {link = {next = 0x8134cac, prev = 0x8134cac}, type = 0x280b79f0, length = 475, start = 0, data = 0x8131298, free = 0x805cf18 <apr_bucket_free>, list = 0x8131010} (gdb) p *(apr_bucket_mmap *)e.data $4 = {refcount = {refcount = 0}, mmap = 0x8152eb8} If you have an account on daedalus and you want to take a look, cd /usr/local/apache2.0.35 gdb bin/httpd corefiles/httpd.core.[2..5] Dump 2 is a different. It shows up in sendfile intermittently. We hit something similar about a month ago. This one needs a trap to prove whether or not we're passing Good Stuff to the FreeBSD kernel. Greg