Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi Rick,

On Sun, 18 Oct 2015, Rick Macklem wrote:


Christian Kratzer wrote:

Hi Rick,

looks like your latest patch nailed the issue. The box has been up for 3
days:

 ck@noc3:~ % uptime
 12:22PM  up 3 days,  4:11, 1 user, load averages: 0.07, 0.10, 0.08
 ck@noc3:~ %

If it does not crash over the weekend this seems to be it:


When I took a closer look, it appears that PR 172942 was a different crash and
it appears that one was fixed via r264600.

Your problem does not appear to be in the bugs database. (I will commit the
patch in mid-November anyhow, but creating a PR for this might be useful for
others.)

Btw, I think the attached patch (which includes this change) also fixes a
problem that caused a crash during mounting, reported via PR 201912.
(If you`d like to test this one that would be appreciated. It should be
applied to code not already patched with the one below, since the below
patch is included in it.)

Thanks for your help with this, rick



I'll put your patch on the VM in question. Btw. it has been up for 6 days now 
without a crash.

Before I do that I would like to see that it really addresses PR 201912.

Do you have any idea how I could provoke that one ?

Ideally I would like to do the stuff that forces the panic, then apply
the patch and see that the system stays stable despite me doing the
silly moves again.

Greetings
Christian

--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Rick Macklem]

Christian Kratzer wrote:
> Hi Rick,
> 
> On Sun, 18 Oct 2015, Rick Macklem wrote:
> 
> > Christian Kratzer wrote:
> >> Hi Rick,
> >>
> >> looks like your latest patch nailed the issue. The box has been up for 3
> >> days:
> >>
> >>  ck@noc3:~ % uptime
> >>  12:22PM  up 3 days,  4:11, 1 user, load averages: 0.07, 0.10, 0.08
> >>  ck@noc3:~ %
> >>
> >> If it does not crash over the weekend this seems to be it:
> >>
> > When I took a closer look, it appears that PR 172942 was a different crash
> > and
> > it appears that one was fixed via r264600.
> >
> > Your problem does not appear to be in the bugs database. (I will commit the
> > patch in mid-November anyhow, but creating a PR for this might be useful
> > for
> > others.)
> >
> > Btw, I think the attached patch (which includes this change) also fixes a
> > problem that caused a crash during mounting, reported via PR 201912.
> > (If you`d like to test this one that would be appreciated. It should be
> > applied to code not already patched with the one below, since the below
> > patch is included in it.)
> >
> > Thanks for your help with this, rick
> 
> 
> I'll put your patch on the VM in question. Btw. it has been up for 6 days now
> without a crash.
> 
> Before I do that I would like to see that it really addresses PR 201912.
> 
> Do you have any idea how I could provoke that one ?
> 
Not really, I'm afraid. The patch deals with the failure cases in 
smb_vc_create(),
which I think was what caused the crash, given the backtrace in the PR.

You can look at smb_vc_create() and see there is a bunch of "goto fail;" cases,
but I don't know how to specifically tickle any of these?

You could "fake it" by putting a "goto fail;" at line#428, just after
the "error = ENOMEM;". This will break smb_vc_create() big time, but I
think it will generate the same crash as PR 201912.

> Ideally I would like to do the stuff that forces the panic, then apply
> the patch and see that the system stays stable despite me doing the
> silly moves again.
> 
That would be nice, but so long as I know that the patch doesn't cause a
regression, I am comfortable committing it. (This refers to the other 2
changes in the patch. It seems clear that the fix in smbiod2.patch is ok
to commit.)

Thanks for all your help with this, rick

> Greetings
> Christian
> 
> --
> Christian Kratzer   CK Software GmbH
> Email:   c...@cksoft.de   Wildberger Weg 24/2
> Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
> Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
> Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
> Web: http://www.cksoft.de/
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
> 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Rick Macklem]

Christian Kratzer wrote:
> Hi Rick,
> 
> looks like your latest patch nailed the issue. The box has been up for 3
> days:
> 
>  ck@noc3:~ % uptime
>  12:22PM  up 3 days,  4:11, 1 user, load averages: 0.07, 0.10, 0.08
>  ck@noc3:~ %
> 
> If it does not crash over the weekend this seems to be it:
> 
When I took a closer look, it appears that PR 172942 was a different crash and
it appears that one was fixed via r264600.

Your problem does not appear to be in the bugs database. (I will commit the
patch in mid-November anyhow, but creating a PR for this might be useful for
others.)

Btw, I think the attached patch (which includes this change) also fixes a
problem that caused a crash during mounting, reported via PR 201912.
(If you`d like to test this one that would be appreciated. It should be
 applied to code not already patched with the one below, since the below
 patch is included in it.)

Thanks for your help with this, rick

> 
> ck@noc3:/usr/src % svn diff sys/netsmb/smb_iod.c
> Index: sys/netsmb/smb_iod.c
> ===
> --- sys/netsmb/smb_iod.c(revision 289211)
> +++ sys/netsmb/smb_iod.c(working copy)
> @@ -659,6 +659,11 @@
>  break;
>  tsleep(>iod_flags, PWAIT, "90idle",
>  iod->iod_sleeptimo);
>  }
> +
> +   /* We can now safely destroy the mutexes and free the iod structure.
> */
> +   smb_sl_destroy(>iod_rqlock);
> +   smb_sl_destroy(>iod_evlock);
> +   free(iod, M_SMBIOD);
>  mtx_unlock();
>  kproc_exit(0);
>   }
> @@ -695,9 +700,6 @@
>   smb_iod_destroy(struct smbiod *iod)
>   {
>  smb_iod_request(iod, SMBIOD_EV_SHUTDOWN | SMBIOD_EV_SYNC, NULL);
> -   smb_sl_destroy(>iod_rqlock);
> -   smb_sl_destroy(>iod_evlock);
> -   free(iod, M_SMBIOD);
>  return 0;
>   }
> 
> ck@noc3:/usr/src %
> 
> 
> Can you get this committed into current and later stable  ?
> 
> Greetings
> Christian
> 
> 
> 
> On Mon, 12 Oct 2015, Rick Macklem wrote:
> 
> > Christian Kratzer wrote:
> >> Hi Rick,
> >>
> >> On Mon, 12 Oct 2015, Rick Macklem wrote:
> >>
> >>> Christian Kratzer wrote:
>  Hi Rick,
> 
>  there was also a second more recent crash in /var/crash
> 
>   Mon Oct 12 03:01:16 CEST 2015
> 
>   FreeBSD noc3.cksoft.de 10.2-STABLE FreeBSD 10.2-STABLE #2 r288980M:
>   Sun
>   Oct 11 08:37:40 CEST 2015
>   c...@noc3.cksoft.de:/usr/obj/usr/src/sys/NOC  amd64
> 
>   panic: Assertion mtx_unowned(m) failed at
>   /usr/src/sys/kern/kern_mutex.c:955
> 
> >>> Oops, I screwed up. I should have looked at this panic assertion when you
> >>> reported
> >>> it before. Ok, so if I understand the assertion correctly, it means that
> >>> another
> >>> thread has the mutex locked. If this is correct, I'll have to take
> >>> another
> >>> look at
> >>> the code and figure out how to wait for these other threads to finish
> >>> with
> >>> the mutexes.
> >>>
> >>> I do think the patch fixes the race I saw, but there must be other races
> >>> in
> >>> the code.
> >>>
> >>> I'll take another look, but if anyone else is conversant with netsmb,
> >>> feel
> >>> free to
> >>> jump in, because it is all new to me.
> >>>
> >>> Unfortunately, I won't have any way to do testing for the next month or
> >>> so,
> >>> so any
> >>> patches I do come up with will be "try this untested..".
> >>
> >> thats no problem.
> >>
> >> Just keep the patches coming when you have time and tell me when to reset
> >> back to stable,
> >> current or whatever so we don't lose sync of the status.
> >>
> > Well, you can try the attached one instead of the previous ones (ie.
> > against stable).
> > It just delays destroying the mutexes until the iod thread is exiting.
> >
> > I can't quite see why the previous patches wouldn't fix it, but this one
> > leaves
> > smb_iod_main() unchanged, so it is a simpler patch and doesn't affect
> > semantics
> > except for a slight delay in destroying the mutexes.
> >
> >> As it looks like that the race happens on unmount I could try putting a
> >> sleep
> >> 60 into the
> >> script that does the "mount && rsycn && umount" magic just before the
> >> umount.
> >> That would
> >> allow anything that it slow to go away to perhaps release the mutexes
> >> before
> >> the umount.
> >>
> > If it still crashes with this patch, it might be worth a try.
> >
> > Or, if this patch still crashes, you could just delete the 3 lines that the
> > patch moves, so the mutexes are never destroyed. This would result in a
> > leak,
> > but it would tell us if destroying these mutexes is the problem.
> >
> > Thanks for your willingness to test these, rick
> >
> >> Not a real fix of course but might help to verify what's going on.
> >>
> >> Greetings
> >> Christian
> >>
> >>
> >> --
> >> Christian Kratzer   CK Software GmbH
> >> Email:   c...@cksoft.de 

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi Rick,

looks like your latest patch nailed the issue. The box has been up for 3 days:

ck@noc3:~ % uptime
12:22PM  up 3 days,  4:11, 1 user, load averages: 0.07, 0.10, 0.08
ck@noc3:~ %

If it does not crash over the weekend this seems to be it:


ck@noc3:/usr/src % svn diff sys/netsmb/smb_iod.c
Index: sys/netsmb/smb_iod.c
===
--- sys/netsmb/smb_iod.c(revision 289211)
+++ sys/netsmb/smb_iod.c(working copy)
@@ -659,6 +659,11 @@
break;
tsleep(>iod_flags, PWAIT, "90idle", iod->iod_sleeptimo);
}
+
+   /* We can now safely destroy the mutexes and free the iod structure. */
+   smb_sl_destroy(>iod_rqlock);
+   smb_sl_destroy(>iod_evlock);
+   free(iod, M_SMBIOD);
mtx_unlock();
kproc_exit(0);
 }
@@ -695,9 +700,6 @@
 smb_iod_destroy(struct smbiod *iod)
 {
smb_iod_request(iod, SMBIOD_EV_SHUTDOWN | SMBIOD_EV_SYNC, NULL);
-   smb_sl_destroy(>iod_rqlock);
-   smb_sl_destroy(>iod_evlock);
-   free(iod, M_SMBIOD);
return 0;
 }

ck@noc3:/usr/src %


Can you get this committed into current and later stable  ?

Greetings
Christian



On Mon, 12 Oct 2015, Rick Macklem wrote:


Christian Kratzer wrote:

Hi Rick,

On Mon, 12 Oct 2015, Rick Macklem wrote:


Christian Kratzer wrote:

Hi Rick,

there was also a second more recent crash in /var/crash

 Mon Oct 12 03:01:16 CEST 2015

 FreeBSD noc3.cksoft.de 10.2-STABLE FreeBSD 10.2-STABLE #2 r288980M:
 Sun
 Oct 11 08:37:40 CEST 2015
 c...@noc3.cksoft.de:/usr/obj/usr/src/sys/NOC  amd64

 panic: Assertion mtx_unowned(m) failed at
 /usr/src/sys/kern/kern_mutex.c:955


Oops, I screwed up. I should have looked at this panic assertion when you
reported
it before. Ok, so if I understand the assertion correctly, it means that
another
thread has the mutex locked. If this is correct, I'll have to take another
look at
the code and figure out how to wait for these other threads to finish with
the mutexes.

I do think the patch fixes the race I saw, but there must be other races in
the code.

I'll take another look, but if anyone else is conversant with netsmb, feel
free to
jump in, because it is all new to me.

Unfortunately, I won't have any way to do testing for the next month or so,
so any
patches I do come up with will be "try this untested..".


thats no problem.

Just keep the patches coming when you have time and tell me when to reset
back to stable,
current or whatever so we don't lose sync of the status.


Well, you can try the attached one instead of the previous ones (ie. against 
stable).
It just delays destroying the mutexes until the iod thread is exiting.

I can't quite see why the previous patches wouldn't fix it, but this one leaves
smb_iod_main() unchanged, so it is a simpler patch and doesn't affect semantics
except for a slight delay in destroying the mutexes.


As it looks like that the race happens on unmount I could try putting a sleep
60 into the
script that does the "mount && rsycn && umount" magic just before the umount.
That would
allow anything that it slow to go away to perhaps release the mutexes before
the umount.


If it still crashes with this patch, it might be worth a try.

Or, if this patch still crashes, you could just delete the 3 lines that the
patch moves, so the mutexes are never destroyed. This would result in a leak,
but it would tell us if destroying these mutexes is the problem.

Thanks for your willingness to test these, rick


Not a real fix of course but might help to verify what's going on.

Greetings
Christian


--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"





--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Rick Macklem]

Christian Kratzer wrote:
> Hi Rick,
> 
> looks like your latest patch nailed the issue. The box has been up for 3
> days:
> 
>  ck@noc3:~ % uptime
>  12:22PM  up 3 days,  4:11, 1 user, load averages: 0.07, 0.10, 0.08
>  ck@noc3:~ %
> 
> If it does not crash over the weekend this seems to be it:
> 
Sounds good. Although I wouldn't have thought it could happen in practice,
I did spot how a race could still occur with the last patch. Since 
smb_iod_request()
did the msleep() with PDROP, it could return any time after the wakeup(evp) and
destroy the mutexes before the iod thread was done with them.

Anyhow, if it fixes the problem, I guess we're happy.

Btw, I think PR#172942 and 201912 may both be caused by the same problem. I've
asked the people that reported these to test the patch.

> 
> ck@noc3:/usr/src % svn diff sys/netsmb/smb_iod.c
> Index: sys/netsmb/smb_iod.c
> ===
> --- sys/netsmb/smb_iod.c(revision 289211)
> +++ sys/netsmb/smb_iod.c(working copy)
> @@ -659,6 +659,11 @@
>  break;
>  tsleep(>iod_flags, PWAIT, "90idle",
>  iod->iod_sleeptimo);
>  }
> +
> +   /* We can now safely destroy the mutexes and free the iod structure.
> */
> +   smb_sl_destroy(>iod_rqlock);
> +   smb_sl_destroy(>iod_evlock);
> +   free(iod, M_SMBIOD);
>  mtx_unlock();
>  kproc_exit(0);
>   }
> @@ -695,9 +700,6 @@
>   smb_iod_destroy(struct smbiod *iod)
>   {
>  smb_iod_request(iod, SMBIOD_EV_SHUTDOWN | SMBIOD_EV_SYNC, NULL);
> -   smb_sl_destroy(>iod_rqlock);
> -   smb_sl_destroy(>iod_evlock);
> -   free(iod, M_SMBIOD);
>  return 0;
>   }
> 
> ck@noc3:/usr/src %
> 
> 
> Can you get this committed into current and later stable  ?
> 
I should be able to do this, although not until mid-Nov. I'd also like
to hear from the folk that reported the PRs.

John, maybe you could review this?

Thanks for your help testing this, rick

> Greetings
> Christian
> 
> 
> 
> On Mon, 12 Oct 2015, Rick Macklem wrote:
> 
> > Christian Kratzer wrote:
> >> Hi Rick,
> >>
> >> On Mon, 12 Oct 2015, Rick Macklem wrote:
> >>
> >>> Christian Kratzer wrote:
>  Hi Rick,
> 
>  there was also a second more recent crash in /var/crash
> 
>   Mon Oct 12 03:01:16 CEST 2015
> 
>   FreeBSD noc3.cksoft.de 10.2-STABLE FreeBSD 10.2-STABLE #2 r288980M:
>   Sun
>   Oct 11 08:37:40 CEST 2015
>   c...@noc3.cksoft.de:/usr/obj/usr/src/sys/NOC  amd64
> 
>   panic: Assertion mtx_unowned(m) failed at
>   /usr/src/sys/kern/kern_mutex.c:955
> 
> >>> Oops, I screwed up. I should have looked at this panic assertion when you
> >>> reported
> >>> it before. Ok, so if I understand the assertion correctly, it means that
> >>> another
> >>> thread has the mutex locked. If this is correct, I'll have to take
> >>> another
> >>> look at
> >>> the code and figure out how to wait for these other threads to finish
> >>> with
> >>> the mutexes.
> >>>
> >>> I do think the patch fixes the race I saw, but there must be other races
> >>> in
> >>> the code.
> >>>
> >>> I'll take another look, but if anyone else is conversant with netsmb,
> >>> feel
> >>> free to
> >>> jump in, because it is all new to me.
> >>>
> >>> Unfortunately, I won't have any way to do testing for the next month or
> >>> so,
> >>> so any
> >>> patches I do come up with will be "try this untested..".
> >>
> >> thats no problem.
> >>
> >> Just keep the patches coming when you have time and tell me when to reset
> >> back to stable,
> >> current or whatever so we don't lose sync of the status.
> >>
> > Well, you can try the attached one instead of the previous ones (ie.
> > against stable).
> > It just delays destroying the mutexes until the iod thread is exiting.
> >
> > I can't quite see why the previous patches wouldn't fix it, but this one
> > leaves
> > smb_iod_main() unchanged, so it is a simpler patch and doesn't affect
> > semantics
> > except for a slight delay in destroying the mutexes.
> >
> >> As it looks like that the race happens on unmount I could try putting a
> >> sleep
> >> 60 into the
> >> script that does the "mount && rsycn && umount" magic just before the
> >> umount.
> >> That would
> >> allow anything that it slow to go away to perhaps release the mutexes
> >> before
> >> the umount.
> >>
> > If it still crashes with this patch, it might be worth a try.
> >
> > Or, if this patch still crashes, you could just delete the 3 lines that the
> > patch moves, so the mutexes are never destroyed. This would result in a
> > leak,
> > but it would tell us if destroying these mutexes is the problem.
> >
> > Thanks for your willingness to test these, rick
> >
> >> Not a real fix of course but might help to verify what's going on.
> >>
> >> Greetings
> >> Christian
> >>
> >>
> >> --
> >> Christian Kratzer   

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi Rick,

On Mon, 12 Oct 2015, Rick Macklem wrote:


Well, you can try the attached one instead of the previous ones (ie. against 
stable).
It just delays destroying the mutexes until the iod thread is exiting.

I can't quite see why the previous patches wouldn't fix it, but this one leaves
smb_iod_main() unchanged, so it is a simpler patch and doesn't affect semantics
except for a slight delay in destroying the mutexes.


patch applied this morning against plain 10-stable with wittness enabled ...


As it looks like that the race happens on unmount I could try putting a sleep
60 into the
script that does the "mount && rsycn && umount" magic just before the umount.
That would
allow anything that it slow to go away to perhaps release the mutexes before
the umount.


If it still crashes with this patch, it might be worth a try.


I had a sleep 60 before the umount over night and it did not crash.
Could have been to short a wait though.

I have removed the sleep 60 in order to give your patch a good testing


Or, if this patch still crashes, you could just delete the 3 lines that the
patch moves, so the mutexes are never destroyed. This would result in a leak,
but it would tell us if destroying these mutexes is the problem.


Good point.


Thanks for your willingness to test these, rick


No problem.  Thanks to you for wrapping your head around this.

Greetings
Christian

--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi Rick,

On Sat, 10 Oct 2015, Rick Macklem wrote:


Hi again,

Attached is a semantically equivalent patch to the one I posted a few
minutes ago, but I think this one is more readable.

Please let me know if you get it tested, rick


the box crashed again tonight with your patch applied. Here's the new crashinfo:

panic: Assertion mtx_unowned(m) failed at /usr/src/sys/kern/kern_mutex.c:955

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: Assertion mtx_unowned(m) failed at /usr/src/sys/kern/kern_mutex.c:955
cpuid = 2
KDB: stack backtrace:
#0 0x80975bb0 at kdb_backtrace+0x60
#1 0x8093baa6 at vpanic+0x126
#2 0x8093b979 at kassert_panic+0x139
#3 0x80921c47 at _mtx_destroy+0x77
#4 0x81a1c114 at smb_iod_destroy+0xc4
#5 0x81a12eea at smb_vc_free+0x1a
#6 0x81a13e24 at sdp_trydestroy+0xb4
#7 0x81a1cb36 at smbfs_unmount+0xd6
#8 0x809d9e84 at dounmount+0x524
#9 0x809d9936 at sys_unmount+0x3c6
#10 0x80d42235 at amd64_syscall+0x265
#11 0x80d25cfb at Xfast_syscall+0xfb
Uptime: 1d21h59m0s
Dumping 191 out of 999 MB:..9%..17%..26%..34%..42%..51%..67%..76%..84%..92%

Here are the stackframes with line numbers:

(kgdb) frame 0
#0  __curthread () at ./machine/pcpu.h:219
219 __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) frame 1
#1  doadump (textdump=) at 
/usr/src/sys/kern/kern_shutdown.c:263
263 dumptid = curthread->td_tid;
(kgdb) frame 2
#2  0x8093b5f2 in kern_reboot (howto=260) at 
/usr/src/sys/kern/kern_shutdown.c:451
451 doadump(TRUE);
(kgdb) frame 3
#3  0x8093bae5 in vpanic (fmt=, ap=) 
at /usr/src/sys/kern/kern_shutdown.c:758
758 kern_reboot(bootopt);
(kgdb) frame 4
#4  0x8093b979 in kassert_panic (fmt=0x80e931ef "Assertion %s 
failed at %s:%d") at /usr/src/sys/kern/kern_shutdown.c:646
646 vpanic(fmt, ap);
(kgdb) frame 5
#5  0x80921c47 in _mtx_destroy (c=0xf80002db5490) at 
/usr/src/sys/kern/kern_mutex.c:955
955 MPASS(mtx_unowned(m));
(kgdb) frame 6
#6  0x81a1c174 in smb_iod_destroy (iod=0xf80002db5400) at 
/usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:733
733 smb_sl_destroy(>iod_evlock);
(kgdb) frame 7
#7  0x81a12eea in smb_vc_free (cp=0xf80002933000) at 
/usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499
499 smb_iod_destroy(vcp->vc_iod);
(kgdb) frame 8
#8  0x81a13e24 in sdp_trydestroy (sdp=0xf80002904140) at 
/usr/src/sys/modules/smbfs/../../netsmb/smb_dev.c:166
166 smb_vc_rele(vcp, scred);
(kgdb) frame 9
#9  0x81a1cb96 in smbfs_unmount (mp=0xf80015226000, 
mntflags=) at 
/usr/src/sys/modules/smbfs/../../fs/smbfs/smbfs_vfsops.c:297
297 sdp_trydestroy(dev);
(kgdb) frame 10
#10 0x809d9e84 in dounmount (mp=0xf80015226000, 
flags=134217728, td=0xf800151b0940) at /usr/src/sys/kern/vfs_mount.c:1313
1313error = VFS_UNMOUNT(mp, flags);
(kgdb)

Let me know if you need anything else from the stackframes.


Greetings
Christian


--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi Rick,

there was also a second more recent crash in /var/crash

Mon Oct 12 03:01:16 CEST 2015

FreeBSD noc3.cksoft.de 10.2-STABLE FreeBSD 10.2-STABLE #2 r288980M: Sun Oct 
11 08:37:40 CEST 2015 c...@noc3.cksoft.de:/usr/obj/usr/src/sys/NOC  amd64

panic: Assertion mtx_unowned(m) failed at /usr/src/sys/kern/kern_mutex.c:955

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: Assertion mtx_unowned(m) failed at /usr/src/sys/kern/kern_mutex.c:955
cpuid = 3
KDB: stack backtrace:
#0 0x80975bb0 at kdb_backtrace+0x60
#1 0x8093baa6 at vpanic+0x126
#2 0x8093b979 at kassert_panic+0x139
#3 0x80921c47 at _mtx_destroy+0x77
#4 0x81a1c174 at smb_iod_destroy+0xc4
#5 0x81a12eea at smb_vc_free+0x1a
#6 0x81a13e24 at sdp_trydestroy+0xb4
#7 0x81a1cb96 at smbfs_unmount+0xd6
#8 0x809d9e84 at dounmount+0x524
#9 0x809d9936 at sys_unmount+0x3c6
#10 0x80d42235 at amd64_syscall+0x265
#11 0x80d25cfb at Xfast_syscall+0xfb
Uptime: 8h44m10s
Dumping 102 out of 999 MB:..16%..32%..47%..63%..78%..94%

Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
Loaded symbols for /boot/kernel/smbfs.ko.symbols
Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
Loaded symbols for /boot/kernel/libiconv.ko.symbols
Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
Loaded symbols for /boot/kernel/libmchain.ko.symbols
#0  doadump (textdump=) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0  doadump (textdump=) at pcpu.h:219
#1  0x8093b5f2 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:451
#2  0x8093bae5 in vpanic (fmt=,
ap=) at /usr/src/sys/kern/kern_shutdown.c:758
#3  0x8093b979 in kassert_panic (fmt=)
at /usr/src/sys/kern/kern_shutdown.c:646
#4  0x80921c47 in _mtx_destroy (c=0xf80002db5490)
at /usr/src/sys/kern/kern_mutex.c:955
#5  0x81a1c174 in smb_iod_destroy (iod=0xf80002db5400)
at /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:733
#6  0x81a12eea in smb_vc_free (cp=0xf80002933000)
at /usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499
#7  0x81a13e24 in sdp_trydestroy (sdp=0xf80002904140)
at /usr/src/sys/modules/smbfs/../../netsmb/smb_dev.c:166
#8  0x81a1cb96 in smbfs_unmount (mp=0xf80015226000,
mntflags=)
at /usr/src/sys/modules/smbfs/../../fs/smbfs/smbfs_vfsops.c:297
#9  0x809d9e84 in dounmount (mp=0xf80015226000, flags=134217728,
td=0xf800151b0940) at /usr/src/sys/kern/vfs_mount.c:1313
#10 0x809d9936 in sys_unmount (td=0xf800151b0940,
uap=0xfe003d643b80) at /usr/src/sys/kern/vfs_mount.c:1205
#11 0x80d42235 in amd64_syscall (td=0xf800151b0940, traced=0)
at subr_syscall.c:134
#12 0x80d25cfb in Xfast_syscall ()
at /usr/src/sys/amd64/amd64/exception.S:396
#13 0x00080089190a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language:  auto; currently minimal
(kgdb)


Greetings
Christian










On Mon, 12 Oct 2015, Christian Kratzer wrote:


Hi Rick,

On Sat, 10 Oct 2015, Rick Macklem wrote:


Hi again,

Attached is a semantically equivalent patch to the one I posted a few
minutes ago, but I think this one is more readable.

Please let me know if you get it tested, rick


the box crashed again tonight with your patch applied. Here's the new 
crashinfo:


   panic: Assertion mtx_unowned(m) failed at 
/usr/src/sys/kern/kern_mutex.c:955


   GNU gdb 6.1.1 [FreeBSD]
   Copyright 2004 Free Software Foundation, Inc.
   GDB is free software, covered by the GNU General Public License, and you 
are
   welcome to change it and/or distribute copies of it under certain 
conditions.

   Type "show copying" to see the conditions.
   There is absolutely no warranty for GDB.  Type "show warranty" for 
details.

   This GDB was configured as "amd64-marcel-freebsd"...

   Unread portion of the kernel message buffer:
   panic: Assertion mtx_unowned(m) failed at 
/usr/src/sys/kern/kern_mutex.c:955

   cpuid = 2
   KDB: stack backtrace:
   #0 0x80975bb0 at kdb_backtrace+0x60
   #1 0x8093baa6 at vpanic+0x126
   #2 0x8093b979 at kassert_panic+0x139
   #3 0x80921c47 at _mtx_destroy+0x77
   #4 

Re: smbfs crashes since approx. 10.1-RELEASE

[Rick Macklem]

Christian Kratzer wrote:
> Hi Rick,
> 
> there was also a second more recent crash in /var/crash
> 
>  Mon Oct 12 03:01:16 CEST 2015
> 
>  FreeBSD noc3.cksoft.de 10.2-STABLE FreeBSD 10.2-STABLE #2 r288980M: Sun
>  Oct 11 08:37:40 CEST 2015
>  c...@noc3.cksoft.de:/usr/obj/usr/src/sys/NOC  amd64
> 
>  panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955
> 
Oops, I screwed up. I should have looked at this panic assertion when you 
reported
it before. Ok, so if I understand the assertion correctly, it means that another
thread has the mutex locked. If this is correct, I'll have to take another look 
at
the code and figure out how to wait for these other threads to finish with the 
mutexes.

I do think the patch fixes the race I saw, but there must be other races in the 
code.

I'll take another look, but if anyone else is conversant with netsmb, feel free 
to
jump in, because it is all new to me.

Unfortunately, I won't have any way to do testing for the next month or so, so 
any
patches I do come up with will be "try this untested..".

rick

>  GNU gdb 6.1.1 [FreeBSD]
>  Copyright 2004 Free Software Foundation, Inc.
>  GDB is free software, covered by the GNU General Public License, and you
>  are
>  welcome to change it and/or distribute copies of it under certain
>  conditions.
>  Type "show copying" to see the conditions.
>  There is absolutely no warranty for GDB.  Type "show warranty" for
>  details.
>  This GDB was configured as "amd64-marcel-freebsd"...
> 
>  Unread portion of the kernel message buffer:
>  panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955
>  cpuid = 3
>  KDB: stack backtrace:
>  #0 0x80975bb0 at kdb_backtrace+0x60
>  #1 0x8093baa6 at vpanic+0x126
>  #2 0x8093b979 at kassert_panic+0x139
>  #3 0x80921c47 at _mtx_destroy+0x77
>  #4 0x81a1c174 at smb_iod_destroy+0xc4
>  #5 0x81a12eea at smb_vc_free+0x1a
>  #6 0x81a13e24 at sdp_trydestroy+0xb4
>  #7 0x81a1cb96 at smbfs_unmount+0xd6
>  #8 0x809d9e84 at dounmount+0x524
>  #9 0x809d9936 at sys_unmount+0x3c6
>  #10 0x80d42235 at amd64_syscall+0x265
>  #11 0x80d25cfb at Xfast_syscall+0xfb
>  Uptime: 8h44m10s
>  Dumping 102 out of 999 MB:..16%..32%..47%..63%..78%..94%
> 
>  Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
>  Loaded symbols for /boot/kernel/smbfs.ko.symbols
>  Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libiconv.ko.symbols
>  Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libmchain.ko.symbols
>  #0  doadump (textdump=) at pcpu.h:219
>  219 pcpu.h: No such file or directory.
>   in pcpu.h
>  (kgdb) #0  doadump (textdump=) at pcpu.h:219
>  #1  0x8093b5f2 in kern_reboot (howto=260)
>   at /usr/src/sys/kern/kern_shutdown.c:451
>  #2  0x8093bae5 in vpanic (fmt=,
>   ap=) at /usr/src/sys/kern/kern_shutdown.c:758
>  #3  0x8093b979 in kassert_panic (fmt=)
>   at /usr/src/sys/kern/kern_shutdown.c:646
>  #4  0x80921c47 in _mtx_destroy (c=0xf80002db5490)
>   at /usr/src/sys/kern/kern_mutex.c:955
>  #5  0x81a1c174 in smb_iod_destroy (iod=0xf80002db5400)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:733
>  #6  0x81a12eea in smb_vc_free (cp=0xf80002933000)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499
>  #7  0x81a13e24 in sdp_trydestroy (sdp=0xf80002904140)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_dev.c:166
>  #8  0x81a1cb96 in smbfs_unmount (mp=0xf80015226000,
>   mntflags=)
>   at /usr/src/sys/modules/smbfs/../../fs/smbfs/smbfs_vfsops.c:297
>  #9  0x809d9e84 in dounmount (mp=0xf80015226000,
>  flags=134217728,
>   td=0xf800151b0940) at /usr/src/sys/kern/vfs_mount.c:1313
>  #10 0x809d9936 in sys_unmount (td=0xf800151b0940,
>   uap=0xfe003d643b80) at /usr/src/sys/kern/vfs_mount.c:1205
>  #11 0x80d42235 in amd64_syscall (td=0xf800151b0940,
>  traced=0)
>   at subr_syscall.c:134
>  #12 0x80d25cfb in Xfast_syscall ()
>   at /usr/src/sys/amd64/amd64/exception.S:396
>  #13 0x00080089190a in ?? ()
>  Previous frame inner to this frame (corrupt stack?)
>  Current language:  auto; currently minimal
>  (kgdb)
> 
> 
> Greetings
> Christian
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> On Mon, 12 Oct 2015, Christian Kratzer wrote:
> 
> > Hi Rick,
> >
> > On Sat, 10 Oct 2015, Rick Macklem wrote:
> >
> >> Hi again,
> >> 
> >> Attached is a semantically equivalent patch to the one I posted a few
> >> minutes ago, but I think this 

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi Rick,

On Mon, 12 Oct 2015, Rick Macklem wrote:


Christian Kratzer wrote:

Hi Rick,

there was also a second more recent crash in /var/crash

 Mon Oct 12 03:01:16 CEST 2015

 FreeBSD noc3.cksoft.de 10.2-STABLE FreeBSD 10.2-STABLE #2 r288980M: Sun
 Oct 11 08:37:40 CEST 2015
 c...@noc3.cksoft.de:/usr/obj/usr/src/sys/NOC  amd64

 panic: Assertion mtx_unowned(m) failed at
 /usr/src/sys/kern/kern_mutex.c:955


Oops, I screwed up. I should have looked at this panic assertion when you 
reported
it before. Ok, so if I understand the assertion correctly, it means that another
thread has the mutex locked. If this is correct, I'll have to take another look 
at
the code and figure out how to wait for these other threads to finish with the 
mutexes.

I do think the patch fixes the race I saw, but there must be other races in the 
code.

I'll take another look, but if anyone else is conversant with netsmb, feel free 
to
jump in, because it is all new to me.

Unfortunately, I won't have any way to do testing for the next month or so, so 
any
patches I do come up with will be "try this untested..".


thats no problem.

Just keep the patches coming when you have time and tell me when to reset back 
to stable,
current or whatever so we don't lose sync of the status.

As it looks like that the race happens on unmount I could try putting a sleep 
60 into the
script that does the "mount && rsycn && umount" magic just before the umount.  
That would
allow anything that it slow to go away to perhaps release the mutexes before 
the umount.

Not a real fix of course but might help to verify what's going on.

Greetings
Christian


--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Rick Macklem]

Christian Kratzer wrote:
> Hi Rick,
> 
> On Mon, 12 Oct 2015, Rick Macklem wrote:
> 
> > Christian Kratzer wrote:
> >> Hi Rick,
> >>
> >> there was also a second more recent crash in /var/crash
> >>
> >>  Mon Oct 12 03:01:16 CEST 2015
> >>
> >>  FreeBSD noc3.cksoft.de 10.2-STABLE FreeBSD 10.2-STABLE #2 r288980M:
> >>  Sun
> >>  Oct 11 08:37:40 CEST 2015
> >>  c...@noc3.cksoft.de:/usr/obj/usr/src/sys/NOC  amd64
> >>
> >>  panic: Assertion mtx_unowned(m) failed at
> >>  /usr/src/sys/kern/kern_mutex.c:955
> >>
> > Oops, I screwed up. I should have looked at this panic assertion when you
> > reported
> > it before. Ok, so if I understand the assertion correctly, it means that
> > another
> > thread has the mutex locked. If this is correct, I'll have to take another
> > look at
> > the code and figure out how to wait for these other threads to finish with
> > the mutexes.
> >
> > I do think the patch fixes the race I saw, but there must be other races in
> > the code.
> >
> > I'll take another look, but if anyone else is conversant with netsmb, feel
> > free to
> > jump in, because it is all new to me.
> >
> > Unfortunately, I won't have any way to do testing for the next month or so,
> > so any
> > patches I do come up with will be "try this untested..".
> 
> thats no problem.
> 
> Just keep the patches coming when you have time and tell me when to reset
> back to stable,
> current or whatever so we don't lose sync of the status.
> 
Well, you can try the attached one instead of the previous ones (ie. against 
stable).
It just delays destroying the mutexes until the iod thread is exiting.

I can't quite see why the previous patches wouldn't fix it, but this one leaves
smb_iod_main() unchanged, so it is a simpler patch and doesn't affect semantics
except for a slight delay in destroying the mutexes.

> As it looks like that the race happens on unmount I could try putting a sleep
> 60 into the
> script that does the "mount && rsycn && umount" magic just before the umount.
> That would
> allow anything that it slow to go away to perhaps release the mutexes before
> the umount.
> 
If it still crashes with this patch, it might be worth a try.

Or, if this patch still crashes, you could just delete the 3 lines that the
patch moves, so the mutexes are never destroyed. This would result in a leak,
but it would tell us if destroying these mutexes is the problem.

Thanks for your willingness to test these, rick

> Not a real fix of course but might help to verify what's going on.
> 
> Greetings
> Christian
> 
> 
> --
> Christian Kratzer   CK Software GmbH
> Email:   c...@cksoft.de   Wildberger Weg 24/2
> Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
> Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
> Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
> Web: http://www.cksoft.de/
> ___
> freebsd-stable@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
> 
--- smb_iod.c.orig	2015-10-10 18:53:34.0 -0400
+++ smb_iod.c	2015-10-12 20:30:00.0 -0400
@@ -659,6 +659,11 @@ smb_iod_thread(void *arg)
 			break;
 		tsleep(>iod_flags, PWAIT, "90idle", iod->iod_sleeptimo);
 	}
+
+	/* We can now safely destroy the mutexes and free the iod structure. */
+	smb_sl_destroy(>iod_rqlock);
+	smb_sl_destroy(>iod_evlock);
+	free(iod, M_SMBIOD);
 	mtx_unlock();
 	kproc_exit(0);
 }
@@ -695,9 +700,6 @@ int
 smb_iod_destroy(struct smbiod *iod)
 {
 	smb_iod_request(iod, SMBIOD_EV_SHUTDOWN | SMBIOD_EV_SYNC, NULL);
-	smb_sl_destroy(>iod_rqlock);
-	smb_sl_destroy(>iod_evlock);
-	free(iod, M_SMBIOD);
 	return 0;
 }
 
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi  Rick,

On Sat, 10 Oct 2015, Rick Macklem wrote:

Hi again,

Attached is a semantically equivalent patch to the one I posted a few
minutes ago, but I think this one is more readable.

Please let me know if you get it tested, rick


I have patched the 10-stable kernel on the box.  Judging from the previous crash
frequency the issue should be fixed if the box stays up  2-3 days.

Thanks a lot!

Greetings
Christian

--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Rick Macklem]

Hi again,

Attached is a semantically equivalent patch to the one I posted a few
minutes ago, but I think this one is more readable.

Please let me know if you get it tested, rick

- Original Message -
> Hi John,
> 
> the box crashed again running a 10-stable kernel with following patch of
> yours:
> 
> On Wed, 7 Oct 2015, John Baldwin wrote:
> > Index: smb_iod.c
> > ===
> > --- smb_iod.c   (revision 288952)
> > +++ smb_iod.c   (working copy)
> > @@ -624,6 +624,13 @@
> >SMB_IOD_EVUNLOCK(iod);
> >} else
> >free(evp, M_SMBIOD);
> > +   if (iod->iod_flags & SMBIOD_SHUTDOWN) {
> > +   if (!STAILQ_EMPTY(>iod_evlist))
> > +   printf("%s: shutdown with pending
> > events\n",
> > +   __func__);
> > +   }
> > +   return;
> > +   }
> >}
> > #if 0
> >if (iod->iod_state == SMBIOD_ST_VCACTIVE) {
> >
> 
> here is what I got on the kvm console:
> 
>  login: panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955^M
>  cpuid = 1^M
>  KDB: stack backtrace:^M
>  #0 0x80975bb0 at kdb_backtrace+0x60^M
>  #1 0x8093baa6 at vpanic+0x126^M
>  #2 0x8093b979 at kassert_panic+0x139^M
>  #3 0x80921c47 at _mtx_destroy+0x77^M
>  #4 0x81a1c114 at smb_iod_destroy+0xc4^M
>  #5 0x81a12eea at smb_vc_free+0x1a^M
>  #6 0x81a13e24 at sdp_trydestroy+0xb4^M
>  #7 0x81a1cb36 at smbfs_unmount+0xd6^M
>  #8 0x809d9e84 at dounmount+0x524^M
>  #9 0x809d9936 at sys_unmount+0x3c6^M
>  #10 0x80d42235 at amd64_syscall+0x265^M
>  #11 0x80d25cfb at Xfast_syscall+0xfb^M
>  Uptime: 19h48m28s^M
>  Dumping 179 out of 999
>  MB:..9%..18%..27%..36%..45%..54%..63%..72%..81%..98%^M
>  Dump complete^M
>  Automatic reboot in 15 seconds - press a key on the console to abort^M
> 
> heres the crashinfo:
> 
>  panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955
> 
>  GNU gdb 6.1.1 [FreeBSD]
>  Copyright 2004 Free Software Foundation, Inc.
>  GDB is free software, covered by the GNU General Public License, and you
>  are
>  welcome to change it and/or distribute copies of it under certain
>  conditions.
>  Type "show copying" to see the conditions.
>  There is absolutely no warranty for GDB.  Type "show warranty" for
>  details.
>  This GDB was configured as "amd64-marcel-freebsd"...
> 
>  Unread portion of the kernel message buffer:
>  panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955
>  cpuid = 1
>  KDB: stack backtrace:
>  #0 0x80975bb0 at kdb_backtrace+0x60
>  #1 0x8093baa6 at vpanic+0x126
>  #2 0x8093b979 at kassert_panic+0x139
>  #3 0x80921c47 at _mtx_destroy+0x77
>  #4 0x81a1c114 at smb_iod_destroy+0xc4
>  #5 0x81a12eea at smb_vc_free+0x1a
>  #6 0x81a13e24 at sdp_trydestroy+0xb4
>  #7 0x81a1cb36 at smbfs_unmount+0xd6
>  #8 0x809d9e84 at dounmount+0x524
>  #9 0x809d9936 at sys_unmount+0x3c6
>  #10 0x80d42235 at amd64_syscall+0x265
>  #11 0x80d25cfb at Xfast_syscall+0xfb
>  Uptime: 19h48m28s
>  Dumping 179 out of 999
>  MB:..9%..18%..27%..36%..45%..54%..63%..72%..81%..98%
> 
>  Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
>  Loaded symbols for /boot/kernel/smbfs.ko.symbols
>  Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libiconv.ko.symbols
>  Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libmchain.ko.symbols
>  #0  doadump (textdump=) at pcpu.h:219
>  219 pcpu.h: No such file or directory.
>   in pcpu.h
>  (kgdb) #0  doadump (textdump=) at pcpu.h:219
>  #1  0x8093b5f2 in kern_reboot (howto=260)
>   at /usr/src/sys/kern/kern_shutdown.c:451
>  #2  0x8093bae5 in vpanic (fmt=,
>   ap=) at /usr/src/sys/kern/kern_shutdown.c:758
>  #3  0x8093b979 in kassert_panic (fmt=)
>   at /usr/src/sys/kern/kern_shutdown.c:646
>  #4  0x80921c47 in _mtx_destroy (c=0xf80009284690)
>   at /usr/src/sys/kern/kern_mutex.c:955
>  #5  0x81a1c114 in smb_iod_destroy (iod=0xf80009284600)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:706
>  #6  0x81a12eea in smb_vc_free (cp=0xf8003a602a00)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499
>  #7  0x81a13e24 in sdp_trydestroy (sdp=0xf8000a7cbc80)
>   at 

Re: smbfs crashes since approx. 10.1-RELEASE

[Rick Macklem]

Hi,

Maybe you could try the attached patch instead of the one below?
(Completely untested. I haven't even tried to compile it.)

rick

- Original Message -
> Hi John,
> 
> the box crashed again running a 10-stable kernel with following patch of
> yours:
> 
> On Wed, 7 Oct 2015, John Baldwin wrote:
> > Index: smb_iod.c
> > ===
> > --- smb_iod.c   (revision 288952)
> > +++ smb_iod.c   (working copy)
> > @@ -624,6 +624,13 @@
> >SMB_IOD_EVUNLOCK(iod);
> >} else
> >free(evp, M_SMBIOD);
> > +   if (iod->iod_flags & SMBIOD_SHUTDOWN) {
> > +   if (!STAILQ_EMPTY(>iod_evlist))
> > +   printf("%s: shutdown with pending
> > events\n",
> > +   __func__);
> > +   }
> > +   return;
> > +   }
> >}
> > #if 0
> >if (iod->iod_state == SMBIOD_ST_VCACTIVE) {
> >
> 
> here is what I got on the kvm console:
> 
>  login: panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955^M
>  cpuid = 1^M
>  KDB: stack backtrace:^M
>  #0 0x80975bb0 at kdb_backtrace+0x60^M
>  #1 0x8093baa6 at vpanic+0x126^M
>  #2 0x8093b979 at kassert_panic+0x139^M
>  #3 0x80921c47 at _mtx_destroy+0x77^M
>  #4 0x81a1c114 at smb_iod_destroy+0xc4^M
>  #5 0x81a12eea at smb_vc_free+0x1a^M
>  #6 0x81a13e24 at sdp_trydestroy+0xb4^M
>  #7 0x81a1cb36 at smbfs_unmount+0xd6^M
>  #8 0x809d9e84 at dounmount+0x524^M
>  #9 0x809d9936 at sys_unmount+0x3c6^M
>  #10 0x80d42235 at amd64_syscall+0x265^M
>  #11 0x80d25cfb at Xfast_syscall+0xfb^M
>  Uptime: 19h48m28s^M
>  Dumping 179 out of 999
>  MB:..9%..18%..27%..36%..45%..54%..63%..72%..81%..98%^M
>  Dump complete^M
>  Automatic reboot in 15 seconds - press a key on the console to abort^M
> 
> heres the crashinfo:
> 
>  panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955
> 
>  GNU gdb 6.1.1 [FreeBSD]
>  Copyright 2004 Free Software Foundation, Inc.
>  GDB is free software, covered by the GNU General Public License, and you
>  are
>  welcome to change it and/or distribute copies of it under certain
>  conditions.
>  Type "show copying" to see the conditions.
>  There is absolutely no warranty for GDB.  Type "show warranty" for
>  details.
>  This GDB was configured as "amd64-marcel-freebsd"...
> 
>  Unread portion of the kernel message buffer:
>  panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955
>  cpuid = 1
>  KDB: stack backtrace:
>  #0 0x80975bb0 at kdb_backtrace+0x60
>  #1 0x8093baa6 at vpanic+0x126
>  #2 0x8093b979 at kassert_panic+0x139
>  #3 0x80921c47 at _mtx_destroy+0x77
>  #4 0x81a1c114 at smb_iod_destroy+0xc4
>  #5 0x81a12eea at smb_vc_free+0x1a
>  #6 0x81a13e24 at sdp_trydestroy+0xb4
>  #7 0x81a1cb36 at smbfs_unmount+0xd6
>  #8 0x809d9e84 at dounmount+0x524
>  #9 0x809d9936 at sys_unmount+0x3c6
>  #10 0x80d42235 at amd64_syscall+0x265
>  #11 0x80d25cfb at Xfast_syscall+0xfb
>  Uptime: 19h48m28s
>  Dumping 179 out of 999
>  MB:..9%..18%..27%..36%..45%..54%..63%..72%..81%..98%
> 
>  Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
>  Loaded symbols for /boot/kernel/smbfs.ko.symbols
>  Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libiconv.ko.symbols
>  Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libmchain.ko.symbols
>  #0  doadump (textdump=) at pcpu.h:219
>  219 pcpu.h: No such file or directory.
>   in pcpu.h
>  (kgdb) #0  doadump (textdump=) at pcpu.h:219
>  #1  0x8093b5f2 in kern_reboot (howto=260)
>   at /usr/src/sys/kern/kern_shutdown.c:451
>  #2  0x8093bae5 in vpanic (fmt=,
>   ap=) at /usr/src/sys/kern/kern_shutdown.c:758
>  #3  0x8093b979 in kassert_panic (fmt=)
>   at /usr/src/sys/kern/kern_shutdown.c:646
>  #4  0x80921c47 in _mtx_destroy (c=0xf80009284690)
>   at /usr/src/sys/kern/kern_mutex.c:955
>  #5  0x81a1c114 in smb_iod_destroy (iod=0xf80009284600)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:706
>  #6  0x81a12eea in smb_vc_free (cp=0xf8003a602a00)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499
>  #7  0x81a13e24 in sdp_trydestroy (sdp=0xf8000a7cbc80)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_dev.c:166
>  #8  

Re: smbfs crashes since approx. 10.1-RELEASE

[Rick Macklem]

Christian Kratzer wrote:
> Hi John,
> 
> the box crashed again running a 10-stable kernel with following patch of
> yours:
> 
> On Wed, 7 Oct 2015, John Baldwin wrote:
> > Index: smb_iod.c
> > ===
> > --- smb_iod.c   (revision 288952)
> > +++ smb_iod.c   (working copy)
> > @@ -624,6 +624,13 @@
> >SMB_IOD_EVUNLOCK(iod);
> >} else
> >free(evp, M_SMBIOD);
> > +   if (iod->iod_flags & SMBIOD_SHUTDOWN) {
> > +   if (!STAILQ_EMPTY(>iod_evlist))
> > +   printf("%s: shutdown with pending
> > events\n",
> > +   __func__);
> > +   }
> > +   return;
> > +   }
> >}
> > #if 0
> >if (iod->iod_state == SMBIOD_ST_VCACTIVE) {
> >
Ok, I think John has found the race. As soon as smb_iod_main() does the 
wakeup() just
before the patch (line#623), smb_iod_destroy() can go ahead and destroy the 
mtuexes.
I think the above patch failed because the smb_iod_sendall(iod) and 
smb_iod_recvall(iod)
need to be done before smb_iod_main() returns.

I'll email a patch for testing later to-day (I've gotta run and do something 
else now), rick

> 
> here is what I got on the kvm console:
> 
>  login: panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955^M
>  cpuid = 1^M
>  KDB: stack backtrace:^M
>  #0 0x80975bb0 at kdb_backtrace+0x60^M
>  #1 0x8093baa6 at vpanic+0x126^M
>  #2 0x8093b979 at kassert_panic+0x139^M
>  #3 0x80921c47 at _mtx_destroy+0x77^M
>  #4 0x81a1c114 at smb_iod_destroy+0xc4^M
>  #5 0x81a12eea at smb_vc_free+0x1a^M
>  #6 0x81a13e24 at sdp_trydestroy+0xb4^M
>  #7 0x81a1cb36 at smbfs_unmount+0xd6^M
>  #8 0x809d9e84 at dounmount+0x524^M
>  #9 0x809d9936 at sys_unmount+0x3c6^M
>  #10 0x80d42235 at amd64_syscall+0x265^M
>  #11 0x80d25cfb at Xfast_syscall+0xfb^M
>  Uptime: 19h48m28s^M
>  Dumping 179 out of 999
>  MB:..9%..18%..27%..36%..45%..54%..63%..72%..81%..98%^M
>  Dump complete^M
>  Automatic reboot in 15 seconds - press a key on the console to abort^M
> 
> heres the crashinfo:
> 
>  panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955
> 
>  GNU gdb 6.1.1 [FreeBSD]
>  Copyright 2004 Free Software Foundation, Inc.
>  GDB is free software, covered by the GNU General Public License, and you
>  are
>  welcome to change it and/or distribute copies of it under certain
>  conditions.
>  Type "show copying" to see the conditions.
>  There is absolutely no warranty for GDB.  Type "show warranty" for
>  details.
>  This GDB was configured as "amd64-marcel-freebsd"...
> 
>  Unread portion of the kernel message buffer:
>  panic: Assertion mtx_unowned(m) failed at
>  /usr/src/sys/kern/kern_mutex.c:955
>  cpuid = 1
>  KDB: stack backtrace:
>  #0 0x80975bb0 at kdb_backtrace+0x60
>  #1 0x8093baa6 at vpanic+0x126
>  #2 0x8093b979 at kassert_panic+0x139
>  #3 0x80921c47 at _mtx_destroy+0x77
>  #4 0x81a1c114 at smb_iod_destroy+0xc4
>  #5 0x81a12eea at smb_vc_free+0x1a
>  #6 0x81a13e24 at sdp_trydestroy+0xb4
>  #7 0x81a1cb36 at smbfs_unmount+0xd6
>  #8 0x809d9e84 at dounmount+0x524
>  #9 0x809d9936 at sys_unmount+0x3c6
>  #10 0x80d42235 at amd64_syscall+0x265
>  #11 0x80d25cfb at Xfast_syscall+0xfb
>  Uptime: 19h48m28s
>  Dumping 179 out of 999
>  MB:..9%..18%..27%..36%..45%..54%..63%..72%..81%..98%
> 
>  Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
>  Loaded symbols for /boot/kernel/smbfs.ko.symbols
>  Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libiconv.ko.symbols
>  Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libmchain.ko.symbols
>  #0  doadump (textdump=) at pcpu.h:219
>  219 pcpu.h: No such file or directory.
>   in pcpu.h
>  (kgdb) #0  doadump (textdump=) at pcpu.h:219
>  #1  0x8093b5f2 in kern_reboot (howto=260)
>   at /usr/src/sys/kern/kern_shutdown.c:451
>  #2  0x8093bae5 in vpanic (fmt=,
>   ap=) at /usr/src/sys/kern/kern_shutdown.c:758
>  #3  0x8093b979 in kassert_panic (fmt=)
>   at /usr/src/sys/kern/kern_shutdown.c:646
>  #4  0x80921c47 in _mtx_destroy (c=0xf80009284690)
>   at /usr/src/sys/kern/kern_mutex.c:955
>  #5  0x81a1c114 in smb_iod_destroy (iod=0xf80009284600)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:706
>  #6  

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi John,

the box crashed again running a 10-stable kernel with following patch of yours:

On Wed, 7 Oct 2015, John Baldwin wrote:

Index: smb_iod.c
===
--- smb_iod.c   (revision 288952)
+++ smb_iod.c   (working copy)
@@ -624,6 +624,13 @@
   SMB_IOD_EVUNLOCK(iod);
   } else
   free(evp, M_SMBIOD);
+   if (iod->iod_flags & SMBIOD_SHUTDOWN) {
+   if (!STAILQ_EMPTY(>iod_evlist))
+   printf("%s: shutdown with pending events\n",
+   __func__);
+   }
+   return;
+   }
   }
#if 0
   if (iod->iod_state == SMBIOD_ST_VCACTIVE) {



here is what I got on the kvm console:

login: panic: Assertion mtx_unowned(m) failed at
/usr/src/sys/kern/kern_mutex.c:955^M
cpuid = 1^M
KDB: stack backtrace:^M
#0 0x80975bb0 at kdb_backtrace+0x60^M
#1 0x8093baa6 at vpanic+0x126^M
#2 0x8093b979 at kassert_panic+0x139^M
#3 0x80921c47 at _mtx_destroy+0x77^M
#4 0x81a1c114 at smb_iod_destroy+0xc4^M
#5 0x81a12eea at smb_vc_free+0x1a^M
#6 0x81a13e24 at sdp_trydestroy+0xb4^M
#7 0x81a1cb36 at smbfs_unmount+0xd6^M
#8 0x809d9e84 at dounmount+0x524^M
#9 0x809d9936 at sys_unmount+0x3c6^M
#10 0x80d42235 at amd64_syscall+0x265^M
#11 0x80d25cfb at Xfast_syscall+0xfb^M
Uptime: 19h48m28s^M
Dumping 179 out of 999
MB:..9%..18%..27%..36%..45%..54%..63%..72%..81%..98%^M
Dump complete^M
Automatic reboot in 15 seconds - press a key on the console to abort^M

heres the crashinfo:

panic: Assertion mtx_unowned(m) failed at /usr/src/sys/kern/kern_mutex.c:955

GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain 
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
panic: Assertion mtx_unowned(m) failed at /usr/src/sys/kern/kern_mutex.c:955
cpuid = 1
KDB: stack backtrace:
#0 0x80975bb0 at kdb_backtrace+0x60
#1 0x8093baa6 at vpanic+0x126
#2 0x8093b979 at kassert_panic+0x139
#3 0x80921c47 at _mtx_destroy+0x77
#4 0x81a1c114 at smb_iod_destroy+0xc4
#5 0x81a12eea at smb_vc_free+0x1a
#6 0x81a13e24 at sdp_trydestroy+0xb4
#7 0x81a1cb36 at smbfs_unmount+0xd6
#8 0x809d9e84 at dounmount+0x524
#9 0x809d9936 at sys_unmount+0x3c6
#10 0x80d42235 at amd64_syscall+0x265
#11 0x80d25cfb at Xfast_syscall+0xfb
Uptime: 19h48m28s
Dumping 179 out of 999 MB:..9%..18%..27%..36%..45%..54%..63%..72%..81%..98%

Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
Loaded symbols for /boot/kernel/smbfs.ko.symbols
Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
Loaded symbols for /boot/kernel/libiconv.ko.symbols
Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
Loaded symbols for /boot/kernel/libmchain.ko.symbols
#0  doadump (textdump=) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) #0  doadump (textdump=) at pcpu.h:219
#1  0x8093b5f2 in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:451
#2  0x8093bae5 in vpanic (fmt=,
ap=) at /usr/src/sys/kern/kern_shutdown.c:758
#3  0x8093b979 in kassert_panic (fmt=)
at /usr/src/sys/kern/kern_shutdown.c:646
#4  0x80921c47 in _mtx_destroy (c=0xf80009284690)
at /usr/src/sys/kern/kern_mutex.c:955
#5  0x81a1c114 in smb_iod_destroy (iod=0xf80009284600)
at /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:706
#6  0x81a12eea in smb_vc_free (cp=0xf8003a602a00)
at /usr/src/sys/modules/smbfs/../../netsmb/smb_conn.c:499
#7  0x81a13e24 in sdp_trydestroy (sdp=0xf8000a7cbc80)
at /usr/src/sys/modules/smbfs/../../netsmb/smb_dev.c:166
#8  0x81a1cb36 in smbfs_unmount (mp=0xf80039f88330,
mntflags=)
at /usr/src/sys/modules/smbfs/../../fs/smbfs/smbfs_vfsops.c:297
#9  0x809d9e84 in dounmount (mp=0xf80039f88330, flags=134217728,
td=0xf8000f2b) at /usr/src/sys/kern/vfs_mount.c:1313
#10 0x809d9936 in sys_unmount (td=0xf8000f2b,
uap=0xfe003d67fb80) at /usr/src/sys/kern/vfs_mount.c:1205
#11 0x80d42235 in amd64_syscall 

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi John,

On Wed, 7 Oct 2015, John Baldwin wrote:

   mtx_lock = 1444181401}


Ok, so that is a destroyed mutex.  This means it is probably not Giant, and
it might be some mutex in smb_iod_main() that shows up in smb_iod_thread() due
to inlining.

Actually, we know this from your earlier mail:

   if (evp->ev_type & SMBIOD_EV_SYNC) {
   SMB_IOD_EVLOCK(iod);
   wakeup(evp);
   SMB_IOD_EVUNLOCK(iod);

Line 624 is that SMB_IOD_EVUNLOCK().

Hmm, does 'p *evp' work at frame 10?  If not, can you try building the
devel/gdb port from a recent ports tree with the 'KGDB' option enabled and
use 'kgdb710' instead of 'kgdb' to see if you can print out '*evp'?


kgdb hangs when chaning to frame10.  I will build the port later (svn ports 
checkout in progress)

I have cloned the VM so that I have this isolated from my production network.


(kgdb)
--snipp--

I can build a GENERIC kernel with INVARIANTS enabled on the box to see if we 
get a better assertions next time this happens.


That would be great, but please keep the existing core and kernel.  We might
be able to figure this out from that still.

Also, go ahead and put this patch in and let me know if you ever see the
printf logged.  If you do, that could explain this panic (and we might need
a more involved fix to avoid memory leaks).

Index: smb_iod.c
===
--- smb_iod.c   (revision 288952)
+++ smb_iod.c   (working copy)
@@ -624,6 +624,13 @@
   SMB_IOD_EVUNLOCK(iod);
   } else
   free(evp, M_SMBIOD);
+   if (iod->iod_flags & SMBIOD_SHUTDOWN) {
+   if (!STAILQ_EMPTY(>iod_evlist))
+   printf("%s: shutdown with pending events\n",
+   __func__);
+   }
+   return;
+   }
   }
#if 0
   if (iod->iod_state == SMBIOD_ST_VCACTIVE) {


The vm is now runnning lates 10-stable kernel with above patch and invariants 
enabled.

Give it about 2 days to produce the next crash.

Greetings
Christian

--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi,

On Tue, 6 Oct 2015, John Baldwin wrote:


This crash is occurring when doing an mtx_unlock(). Unfortunately, I'm not
conversant w.r.t. this code. I've cc'd jhb@ in case he has some insight.
If you don't get any responses, I'd suggest reposting to freebsd-current@ with
"crashes in mtx_unlock()" in the subject line.

Btw John, the code does tsleep() in a loop before the mtx_unlock(). I do
remember that was once allowed, but am not sure if it still is (ie a tsleep() 
call
while holding Giant)?

Hopefully someone who knows what is special about Giant that might cause this 
will
respond.

Good luck with it, rick


tsleep() with Giant is still allowed.  However, this sort of panic usually means
you unlocked a mutex you didn't hold (but without INVARIANTS enabled or you'd 
get
an assertion failure earlier).

I don't see anything obviously wrong in smb_iod_thread() however.

If you have the crashdump, can you please run this in kgdb:

frame 9
p (struct mtx *)c
p *(struct mtx *)c


yes I have. Here we go:

--snipp--
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address   = 0x20
fault code  = supervisor read data, page not present
instruction pointer = 0x20:0x80996c7c
stack pointer   = 0x28:0xfe004e79bac0
frame pointer   = 0x28:0xfe004e79baf0
code segment= base 0x0, limit 0xf, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags= resume, IOPL = 0
current process = 12235 (smbiod172)
trap number = 12
panic: page fault
cpuid = 0
KDB: stack backtrace:
#0 0x80984e30 at kdb_backtrace+0x60
#1 0x809489e6 at vpanic+0x126
#2 0x809488b3 at panic+0x43
#3 0x80d4aadb at trap_fatal+0x36b
#4 0x80d4addd at trap_pfault+0x2ed
#5 0x80d4a47a at trap+0x47a
#6 0x80d307f2 at calltrap+0x8
#7 0x8092ebe0 at __mtx_unlock_sleep+0x60
#8 0x8092eb69 at __mtx_unlock_flags+0x69
#9 0x81a1b724 at smb_iod_thread+0xb4
#10 0x8091244a at fork_exit+0x9a
#11 0x80d30d2e at fork_trampoline+0xe
Uptime: 1d18h34m4s
Dumping 161 out of 999 MB:..10%..20%..30%..40%..50%..60%..70%..80%..90%..100%

Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
Loaded symbols for /boot/kernel/smbfs.ko.symbols
Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
Loaded symbols for /boot/kernel/libiconv.ko.symbols
Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
Loaded symbols for /boot/kernel/libmchain.ko.symbols
#0  doadump (textdump=) at pcpu.h:219
219 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) frame 9
#9  0x8092ebe0 in __mtx_unlock_sleep (c=0xf8002f531790, opts=,
file=0x81a25801 "%s: Can't handle disordered parameters %d:%d\n", 
line=1) at /usr/src/sys/kern/kern_mutex.c:791
791 /usr/src/sys/kern/kern_mutex.c: No such file or directory.
in /usr/src/sys/kern/kern_mutex.c
Current language:  auto; currently minimal
(kgdb) p (struct mtx *)c
$1 = (struct mtx *) 0xf8002f531790
(kgdb) p *(struct mtx *)c
$2 = {lock_object = {lo_name = 0x6 , lo_flags = 0, 
lo_data = 0, lo_witness = 0xf8002f531798},
  mtx_lock = 1444181401}
(kgdb)
--snipp--

I can build a GENERIC kernel with INVARIANTS enabled on the box to see if we 
get a better assertions next time this happens.

That is in case it happens at all with a debug build.

Greetings
Christian

--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[John Baldwin]

On Wednesday, October 07, 2015 08:52:30 AM Christian Kratzer wrote:
> Hi,
> 
> On Tue, 6 Oct 2015, John Baldwin wrote:
> 
> >> This crash is occurring when doing an mtx_unlock(). Unfortunately, 
> >> I'm not
> >> conversant w.r.t. this code. I've cc'd jhb@ in case he has some insight.
> >> If you don't get any responses, I'd suggest reposting to freebsd-current@ 
> >> with
> >> "crashes in mtx_unlock()" in the subject line.
> >>
> >> Btw John, the code does tsleep() in a loop before the mtx_unlock(). 
> >> I do
> >> remember that was once allowed, but am not sure if it still is (ie a 
> >> tsleep() call
> >> while holding Giant)?
> >>
> >> Hopefully someone who knows what is special about Giant that might cause 
> >> this will
> >> respond.
> >>
> >> Good luck with it, rick
> >
> > tsleep() with Giant is still allowed.  However, this sort of panic usually 
> > means
> > you unlocked a mutex you didn't hold (but without INVARIANTS enabled or 
> > you'd get
> > an assertion failure earlier).
> >
> > I don't see anything obviously wrong in smb_iod_thread() however.
> >
> > If you have the crashdump, can you please run this in kgdb:
> >
> > frame 9
> > p (struct mtx *)c
> > p *(struct mtx *)c
> 
> yes I have. Here we go:
> 
> --snipp--
> Fatal trap 12: page fault while in kernel mode
> cpuid = 0; apic id = 00
> fault virtual address   = 0x20
> fault code  = supervisor read data, page not present
> instruction pointer = 0x20:0x80996c7c
> stack pointer   = 0x28:0xfe004e79bac0
> frame pointer   = 0x28:0xfe004e79baf0
> code segment= base 0x0, limit 0xf, type 0x1b
>  = DPL 0, pres 1, long 1, def32 0, gran 1
> processor eflags= resume, IOPL = 0
> current process = 12235 (smbiod172)
> trap number = 12
> panic: page fault
> cpuid = 0
> KDB: stack backtrace:
> #0 0x80984e30 at kdb_backtrace+0x60
> #1 0x809489e6 at vpanic+0x126
> #2 0x809488b3 at panic+0x43
> #3 0x80d4aadb at trap_fatal+0x36b
> #4 0x80d4addd at trap_pfault+0x2ed
> #5 0x80d4a47a at trap+0x47a
> #6 0x80d307f2 at calltrap+0x8
> #7 0x8092ebe0 at __mtx_unlock_sleep+0x60
> #8 0x8092eb69 at __mtx_unlock_flags+0x69
> #9 0x81a1b724 at smb_iod_thread+0xb4
> #10 0x8091244a at fork_exit+0x9a
> #11 0x80d30d2e at fork_trampoline+0xe
> Uptime: 1d18h34m4s
> Dumping 161 out of 999 MB:..10%..20%..30%..40%..50%..60%..70%..80%..90%..100%
> 
> Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
> Loaded symbols for /boot/kernel/smbfs.ko.symbols
> Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
> Loaded symbols for /boot/kernel/libiconv.ko.symbols
> Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
> Loaded symbols for /boot/kernel/libmchain.ko.symbols
> #0  doadump (textdump=) at pcpu.h:219
> 219 pcpu.h: No such file or directory.
>  in pcpu.h
> (kgdb) frame 9
> #9  0x8092ebe0 in __mtx_unlock_sleep (c=0xf8002f531790, 
> opts=,
>  file=0x81a25801 "%s: Can't handle disordered parameters 
> %d:%d\n", line=1) at /usr/src/sys/kern/kern_mutex.c:791
> 791 /usr/src/sys/kern/kern_mutex.c: No such file or directory.
>  in /usr/src/sys/kern/kern_mutex.c
> Current language:  auto; currently minimal
> (kgdb) p (struct mtx *)c
> $1 = (struct mtx *) 0xf8002f531790
> (kgdb) p *(struct mtx *)c
> $2 = {lock_object = {lo_name = 0x6 , lo_flags = 0, 
> lo_data = 0, lo_witness = 0xf8002f531798},
>mtx_lock = 1444181401}

Ok, so that is a destroyed mutex.  This means it is probably not Giant, and
it might be some mutex in smb_iod_main() that shows up in smb_iod_thread() due
to inlining.

Actually, we know this from your earlier mail:

if (evp->ev_type & SMBIOD_EV_SYNC) {
SMB_IOD_EVLOCK(iod);
wakeup(evp);
SMB_IOD_EVUNLOCK(iod);

Line 624 is that SMB_IOD_EVUNLOCK().

Hmm, does 'p *evp' work at frame 10?  If not, can you try building the
devel/gdb port from a recent ports tree with the 'KGDB' option enabled and
use 'kgdb710' instead of 'kgdb' to see if you can print out '*evp'?

> (kgdb)
> --snipp--
> 
> I can build a GENERIC kernel with INVARIANTS enabled on the box to see if we 
> get a better assertions next time this happens.

That would be great, but please keep the existing core and kernel.  We might
be able to figure this out from that still.

Also, go ahead and put this patch in and let me know if you ever see the
printf logged.  If you do, that could explain this panic (and we might need
a more involved fix to avoid memory leaks).

Index: smb_iod.c
===
--- smb_iod.c   (revision 288952)
+++ smb_iod.c   (working copy)
@@ -624,6 +624,13 @@
SMB_IOD_EVUNLOCK(iod);
} else
free(evp, 

Re: smbfs crashes since approx. 10.1-RELEASE

[Christian Kratzer]

Hi,

On Mon, 5 Oct 2015, dweimer wrote:


 root@noc2:/var/crash # freebsd-version -u
 10.2-RELEASE-p5
 root@noc2:/var/crash # freebsd-version -k
 10.2-RELEASE


Shouldn't the kernel version also be 10.2-RELEASE-p5? Perhaps the user land 
being different than the kernel is causing an issue?


root@freebsd:/home/dweimer # uname -a
FreeBSD freebsd.dweimer.local 10.2-RELEASE-p5 FreeBSD 10.2-RELEASE-p5 #1 
r288512: Fri Oct  2 13:54:14 CDT 2015 
dweimer@freebsd.dweimer.local:/jails/devel/ROOT/usr/obj/jails/devel/ROOT/usr/src/sys/GENERIC 
amd64

root@freebsd:/home/dweimer # freebsd-version -u
10.2-RELEASE-p5
root@freebsd:/home/dweimer # freebsd-version -k
10.2-RELEASE-p5


as this system is maintained by freebsd-update and there have been no kernel 
changes from release to p5 the kernel is still showing 10.2-RELEASE.

It is normal for systems updated by freebsd-update that the kernel version lags 
behind the userland.

Greetings
Christian

--
Christian Kratzer   CK Software GmbH
Email:   c...@cksoft.de   Wildberger Weg 24/2
Phone:   +49 7032 893 997 - 0   D-71126 Gaeufelden
Fax: +49 7032 893 997 - 9   HRB 245288, Amtsgericht Stuttgart
Mobile:  +49 171 1947 843   Geschaeftsfuehrer: Christian Kratzer
Web: http://www.cksoft.de/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[John Baldwin]

On Monday, October 05, 2015 06:16:54 PM Rick Macklem wrote:
> Christian Kratzer wrote:
> > Hi,
> > 
> > I run a regular rsync job that runs from cron and copies stuff that gets
> > created on a Windows smbfs share.
> > 
> > Starting about 10.1-RELEASE the VM has become unstable and started panicing.
> > 
> > I have narrowed the issue down to the aforementioned rsync job.
> > 
> > When I move the job to a different VM the the other VM starts crashing and
> > the VM without the job becomes stable agin.
> > 
> > I have panics and crashinfos stored in /var/crash if anybody is interested:
> > 
> >  root@noc2:/var/crash # uname -a
> >  FreeBSD noc2.cksoft.de 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r28: 
> > Wed
> >  Aug 12 15:26:37 UTC 2015
> >  r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
> >  root@noc2:/var/crash # freebsd-version -u
> >  10.2-RELEASE-p5
> >  root@noc2:/var/crash # freebsd-version -k
> >  10.2-RELEASE
> >  root@noc2:/var/crash #
> > 
> > This is what I have in /var/crash/core.txt.0
> > 
> >  Fatal trap 12: page fault while in kernel mode
> >  cpuid = 0; apic id = 00
> >  fault virtual address   = 0x20
> >  fault code  = supervisor read data, page not present
> >  instruction pointer = 0x20:0x80996c7c
> >  stack pointer   = 0x28:0xfe003d6c0ac0
> >  frame pointer   = 0x28:0xfe003d6c0af0
> >  code segment= base 0x0, limit 0xf, type 0x1b
> > = DPL 0, pres 1, long 1, def32 0, gran 1
> >  processor eflags= resume, IOPL = 0
> >  current process = 1349 (smbiod10)
> >  trap number = 12
> >  panic: page fault
> >  cpuid = 0
> >  KDB: stack backtrace:
> >  #0 0x80984e30 at kdb_backtrace+0x60
> >  #1 0x809489e6 at vpanic+0x126
> >  #2 0x809488b3 at panic+0x43
> >  #3 0x80d4aadb at trap_fatal+0x36b
> >  #4 0x80d4addd at trap_pfault+0x2ed
> >  #5 0x80d4a47a at trap+0x47a
> >  #6 0x80d307f2 at calltrap+0x8
> >  #7 0x8092ebe0 at __mtx_unlock_sleep+0x60
> >  #8 0x8092eb69 at __mtx_unlock_flags+0x69
> >  #9 0x81a1b724 at smb_iod_thread+0xb4
> >  #10 0x8091244a at fork_exit+0x9a
> >  #11 0x80d30d2e at fork_trampoline+0xe
> >  Uptime: 2h43m55s
> >  Dumping 103 out of 999 MB: (CTRL-C to abort)
> >  ..16%..31%..47%..62%..78%..93%
> > 
> This crash is occurring when doing an mtx_unlock(). Unfortunately, I'm 
> not
> conversant w.r.t. this code. I've cc'd jhb@ in case he has some insight.
> If you don't get any responses, I'd suggest reposting to freebsd-current@ with
> "crashes in mtx_unlock()" in the subject line.
> 
> Btw John, the code does tsleep() in a loop before the mtx_unlock(). I do
> remember that was once allowed, but am not sure if it still is (ie a tsleep() 
> call
> while holding Giant)?
> 
> Hopefully someone who knows what is special about Giant that might cause this 
> will
> respond.
> 
> Good luck with it, rick

tsleep() with Giant is still allowed.  However, this sort of panic usually means
you unlocked a mutex you didn't hold (but without INVARIANTS enabled or you'd 
get
an assertion failure earlier).

I don't see anything obviously wrong in smb_iod_thread() however.

If you have the crashdump, can you please run this in kgdb:

frame 9
p (struct mtx *)c
p *(struct mtx *)c

-- 
John Baldwin
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Rick Macklem]

Christian Kratzer wrote:
> Hi,
> 
> I run a regular rsync job that runs from cron and copies stuff that gets
> created on a Windows smbfs share.
> 
> Starting about 10.1-RELEASE the VM has become unstable and started panicing.
> 
> I have narrowed the issue down to the aforementioned rsync job.
> 
> When I move the job to a different VM the the other VM starts crashing and
> the VM without the job becomes stable agin.
> 
> I have panics and crashinfos stored in /var/crash if anybody is interested:
> 
>  root@noc2:/var/crash # uname -a
>  FreeBSD noc2.cksoft.de 10.2-RELEASE FreeBSD 10.2-RELEASE #0 r28: Wed
>  Aug 12 15:26:37 UTC 2015
>  r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
>  root@noc2:/var/crash # freebsd-version -u
>  10.2-RELEASE-p5
>  root@noc2:/var/crash # freebsd-version -k
>  10.2-RELEASE
>  root@noc2:/var/crash #
> 
> This is what I have in /var/crash/core.txt.0
> 
>  Fatal trap 12: page fault while in kernel mode
>  cpuid = 0; apic id = 00
>  fault virtual address   = 0x20
>  fault code  = supervisor read data, page not present
>  instruction pointer = 0x20:0x80996c7c
>  stack pointer   = 0x28:0xfe003d6c0ac0
>  frame pointer   = 0x28:0xfe003d6c0af0
>  code segment= base 0x0, limit 0xf, type 0x1b
>   = DPL 0, pres 1, long 1, def32 0, gran 1
>  processor eflags= resume, IOPL = 0
>  current process = 1349 (smbiod10)
>  trap number = 12
>  panic: page fault
>  cpuid = 0
>  KDB: stack backtrace:
>  #0 0x80984e30 at kdb_backtrace+0x60
>  #1 0x809489e6 at vpanic+0x126
>  #2 0x809488b3 at panic+0x43
>  #3 0x80d4aadb at trap_fatal+0x36b
>  #4 0x80d4addd at trap_pfault+0x2ed
>  #5 0x80d4a47a at trap+0x47a
>  #6 0x80d307f2 at calltrap+0x8
>  #7 0x8092ebe0 at __mtx_unlock_sleep+0x60
>  #8 0x8092eb69 at __mtx_unlock_flags+0x69
>  #9 0x81a1b724 at smb_iod_thread+0xb4
>  #10 0x8091244a at fork_exit+0x9a
>  #11 0x80d30d2e at fork_trampoline+0xe
>  Uptime: 2h43m55s
>  Dumping 103 out of 999 MB: (CTRL-C to abort)
>  ..16%..31%..47%..62%..78%..93%
> 
This crash is occurring when doing an mtx_unlock(). Unfortunately, I'm not
conversant w.r.t. this code. I've cc'd jhb@ in case he has some insight.
If you don't get any responses, I'd suggest reposting to freebsd-current@ with
"crashes in mtx_unlock()" in the subject line.

Btw John, the code does tsleep() in a loop before the mtx_unlock(). I do
remember that was once allowed, but am not sure if it still is (ie a tsleep() 
call
while holding Giant)?

Hopefully someone who knows what is special about Giant that might cause this 
will
respond.

Good luck with it, rick

>  Reading symbols from /boot/kernel/smbfs.ko.symbols...done.
>  Loaded symbols for /boot/kernel/smbfs.ko.symbols
>  Reading symbols from /boot/kernel/libiconv.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libiconv.ko.symbols
>  Reading symbols from /boot/kernel/libmchain.ko.symbols...done.
>  Loaded symbols for /boot/kernel/libmchain.ko.symbols
>  #0  doadump (textdump=) at pcpu.h:219
>  219 pcpu.h: No such file or directory.
>   in pcpu.h
>  (kgdb) #0  doadump (textdump=) at pcpu.h:219
>  #1  0x80948642 in kern_reboot (howto=260)
>   at /usr/src/sys/kern/kern_shutdown.c:451
>  #2  0x80948a25 in vpanic (fmt=,
>   ap=) at /usr/src/sys/kern/kern_shutdown.c:758
>  #3  0x809488b3 in panic (fmt=0x0)
>   at /usr/src/sys/kern/kern_shutdown.c:687
>  #4  0x80d4aadb in trap_fatal (frame=,
>   eva=) at /usr/src/sys/amd64/amd64/trap.c:851
>  #5  0x80d4addd in trap_pfault (frame=0xfe003d6c0a10,
>   usermode=) at /usr/src/sys/amd64/amd64/trap.c:674
>  #6  0x80d4a47a in trap (frame=0xfe003d6c0a10)
>   at /usr/src/sys/amd64/amd64/trap.c:440
>  #7  0x80d307f2 in calltrap ()
>   at /usr/src/sys/amd64/amd64/exception.S:236
>  #8  0x80996c7c in turnstile_broadcast (ts=0x0, queue=0)
>   at /usr/src/sys/kern/subr_turnstile.c:838
>  #9  0x8092ebe0 in __mtx_unlock_sleep (c=0xf80002df0390,
>   opts=,
>   file=0x81a25801 "%s: Can't handle disordered parameters 
> %d:%d\n",
>   line=1) at /usr/src/sys/kern/kern_mutex.c:791
>  #10 0x8092eb69 in __mtx_unlock_flags (c=,
>   opts=,
>   file=0x81a25801 "%s: Can't handle disordered parameters 
> %d:%d\n",
>   line=1) at /usr/src/sys/kern/kern_mutex.c:254
>  #11 0x81a1b724 in smb_iod_thread (arg=0xf80002df0300)
>   at /usr/src/sys/modules/smbfs/../../netsmb/smb_iod.c:624
>  #12 0x8091244a in fork_exit (
>   

Re: smbfs crashes since approx. 10.1-RELEASE

[dweimer]

On 2015-10-05 5:16 pm, Rick Macklem wrote:

Christian Kratzer wrote:

Hi,

I run a regular rsync job that runs from cron and copies stuff that 
gets

created on a Windows smbfs share.

Starting about 10.1-RELEASE the VM has become unstable and started 
panicing.


I have narrowed the issue down to the aforementioned rsync job.

When I move the job to a different VM the the other VM starts crashing 
and

the VM without the job becomes stable agin.

I have panics and crashinfos stored in /var/crash if anybody is 
interested:


 root@noc2:/var/crash # uname -a
 FreeBSD noc2.cksoft.de 10.2-RELEASE FreeBSD 10.2-RELEASE #0 
r28: Wed

 Aug 12 15:26:37 UTC 2015
 r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
 root@noc2:/var/crash # freebsd-version -u
 10.2-RELEASE-p5
 root@noc2:/var/crash # freebsd-version -k
 10.2-RELEASE


Shouldn't the kernel version also be 10.2-RELEASE-p5? Perhaps the user 
land being different than the kernel is causing an issue?


root@freebsd:/home/dweimer # uname -a
FreeBSD freebsd.dweimer.local 10.2-RELEASE-p5 FreeBSD 10.2-RELEASE-p5 #1 
r288512: Fri Oct  2 13:54:14 CDT 2015 
dweimer@freebsd.dweimer.local:/jails/devel/ROOT/usr/obj/jails/devel/ROOT/usr/src/sys/GENERIC 
 amd64

root@freebsd:/home/dweimer # freebsd-version -u
10.2-RELEASE-p5
root@freebsd:/home/dweimer # freebsd-version -k
10.2-RELEASE-p5

--
Thanks,
   Dean E. Weimer
   http://www.dweimer.net/
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"

Re: smbfs crashes since approx. 10.1-RELEASE

[Kevin Oberman]

On Mon, Oct 5, 2015 at 5:00 PM, dweimer  wrote:

> On 2015-10-05 5:16 pm, Rick Macklem wrote:
>
>> Christian Kratzer wrote:
>>
>>> Hi,
>>>
>>> I run a regular rsync job that runs from cron and copies stuff that gets
>>> created on a Windows smbfs share.
>>>
>>> Starting about 10.1-RELEASE the VM has become unstable and started
>>> panicing.
>>>
>>> I have narrowed the issue down to the aforementioned rsync job.
>>>
>>> When I move the job to a different VM the the other VM starts crashing
>>> and
>>> the VM without the job becomes stable agin.
>>>
>>> I have panics and crashinfos stored in /var/crash if anybody is
>>> interested:
>>>
>>>  root@noc2:/var/crash # uname -a
>>>  FreeBSD noc2.cksoft.de 10.2-RELEASE FreeBSD 10.2-RELEASE #0
>>> r28: Wed
>>>  Aug 12 15:26:37 UTC 2015
>>>  r...@releng1.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
>>>  root@noc2:/var/crash # freebsd-version -u
>>>  10.2-RELEASE-p5
>>>  root@noc2:/var/crash # freebsd-version -k
>>>  10.2-RELEASE
>>>
>>
> Shouldn't the kernel version also be 10.2-RELEASE-p5? Perhaps the user
> land being different than the kernel is causing an issue?
>
> root@freebsd:/home/dweimer # uname -a
> FreeBSD freebsd.dweimer.local 10.2-RELEASE-p5 FreeBSD 10.2-RELEASE-p5 #1
> r288512: Fri Oct  2 13:54:14 CDT 2015 
> dweimer@freebsd.dweimer.local:/jails/devel/ROOT/usr/obj/jails/devel/ROOT/usr/src/sys/GENERIC
> amd64
> root@freebsd:/home/dweimer # freebsd-version -u
> 10.2-RELEASE-p5
> root@freebsd:/home/dweimer # freebsd-version -k
> 10.2-RELEASE-p5
>
> --
> Thanks,
>Dean E. Weimer
>http://www.dweimer.net/


Probably not.

If freebsd-update is  used and the update does not change the kernel, the
kernel is not touched and remains at the prior level. If built from
sources, even though the kernel is identical, the kernel version is bumped.
I just confirmed that none of the updates to 10.2-RELEASE appear to update
the kernel.
--
Kevin Oberman, Part time kid herder and retired Network Engineer
E-mail: rkober...@gmail.com
PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"