Re: NFS deadlock on 9.2-Beta1

2013-08-25 Thread J David
On Sun, Aug 25, 2013 at 7:42 AM, Adrian Chadd wrote: > Does -HEAD have this same problem? If I understood kib@ correctly, this is fixed in -HEAD by r253927. > If so, we should likely just revert the patch entirely from -HEAD and -9 > until it's resolved. It was not too difficult to prepare a re

Re: NFS deadlock on 9.2-Beta1

2013-08-25 Thread Rick Macklem
Michael Tratz wrote: > > On Aug 15, 2013, at 2:39 PM, Rick Macklem > wrote: > > > Michael Tratz wrote: > >> > >> On Jul 27, 2013, at 11:25 PM, Konstantin Belousov > >> wrote: > >> > >>> On Sat, Jul 27, 2013 at 03:13:05PM -0700, Michael Tratz wrote: > Let's assume the pid which started th

Re: NFS deadlock on 9.2-Beta1

2013-08-25 Thread Adrian Chadd
Hi, Does -HEAD have this same problem? If so, we should likely just revert the patch entirely from -HEAD and -9 until it's resolved. -adrian On 24 August 2013 23:51, Michael Tratz wrote: > > On Aug 15, 2013, at 2:39 PM, Rick Macklem wrote: > > > Michael Tratz wrote: > >> > >> On Jul 27,

Re: NFS deadlock on 9.2-Beta1

2013-08-25 Thread Michael Tratz
On Aug 15, 2013, at 2:39 PM, Rick Macklem wrote: > Michael Tratz wrote: >> >> On Jul 27, 2013, at 11:25 PM, Konstantin Belousov >> wrote: >> >>> On Sat, Jul 27, 2013 at 03:13:05PM -0700, Michael Tratz wrote: Let's assume the pid which started the deadlock is 14001 (it will be a diff

Re: NFS deadlock on 9.2-Beta1

2013-08-24 Thread Rick Macklem
Kostik wrote: > On Sat, Aug 24, 2013 at 01:08:05PM -0400, J David wrote: > > The requested information about the deadlock was finally obtained > > and > > provided off-list to the requested parties due to size. > > Thank you, the problem is clear now. > > The problematic process backtrace is > >

Re: NFS deadlock on 9.2-Beta1

2013-08-24 Thread J David
On Sat, Aug 24, 2013 at 4:55 PM, Konstantin Belousov wrote: > On Sat, Aug 24, 2013 at 04:11:09PM -0400, J David wrote: >> On Sat, Aug 24, 2013 at 3:38 PM, Konstantin Belousov >> wrote: >> > No, at least not without reverting the r254754 first. The IGN_SBUSY patch >> > is not critical there. >> >

Re: NFS deadlock on 9.2-Beta1

2013-08-24 Thread Konstantin Belousov
On Sat, Aug 24, 2013 at 04:11:09PM -0400, J David wrote: > On Sat, Aug 24, 2013 at 3:38 PM, Konstantin Belousov > wrote: > > No, at least not without reverting the r254754 first. The IGN_SBUSY patch > > is not critical there. > > There is lots of other stuff in r250907 / reverted by r254754. So

Re: NFS deadlock on 9.2-Beta1

2013-08-24 Thread J David
On Sat, Aug 24, 2013 at 3:38 PM, Konstantin Belousov wrote: > No, at least not without reverting the r254754 first. The IGN_SBUSY patch > is not critical there. There is lots of other stuff in r250907 / reverted by r254754. Some of it looks important for sendfile() performance. If testing this

Re: NFS deadlock on 9.2-Beta1

2013-08-24 Thread Konstantin Belousov
On Sat, Aug 24, 2013 at 02:03:50PM -0400, J David wrote: > On Sat, Aug 24, 2013 at 1:41 PM, Konstantin Belousov > wrote: > > I think the easiest route is to a partial merge of the r253927 from HEAD. > > Is it helpful if we restart testing releng/9.2 using your suggested > fix? And if so, the IGN

Re: NFS deadlock on 9.2-Beta1

2013-08-24 Thread J David
On Sat, Aug 24, 2013 at 1:41 PM, Konstantin Belousov wrote: > I think the easiest route is to a partial merge of the r253927 from HEAD. Is it helpful if we restart testing releng/9.2 using your suggested fix? And if so, the IGN_SBUSY patch you posted earlier be applied as well or no? If it ran

Re: NFS deadlock on 9.2-Beta1

2013-08-24 Thread Konstantin Belousov
On Sat, Aug 24, 2013 at 01:08:05PM -0400, J David wrote: > The requested information about the deadlock was finally obtained and > provided off-list to the requested parties due to size. Thank you, the problem is clear now. The problematic process backtrace is Tracing command httpd pid 86383 tid

Re: NFS deadlock on 9.2-Beta1

2013-08-24 Thread J David
The requested information about the deadlock was finally obtained and provided off-list to the requested parties due to size. ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mai

Re: NFS deadlock on 9.2-Beta1

2013-08-23 Thread Rick Macklem
J. David wrote: > One deadlocked process cropped up overnight, but I managed to panic > the box before getting too much debugging info. :( > > The process was in state T instead of D, which I guess must be a side > effect of some of the debugging code compiled in. > > Here are the details I was a

Re: NFS deadlock on 9.2-Beta1

2013-08-22 Thread J David
One deadlocked process cropped up overnight, but I managed to panic the box before getting too much debugging info. :( The process was in state T instead of D, which I guess must be a side effect of some of the debugging code compiled in. Here are the details I was able to capture: db> show pro

Re: NFS deadlock on 9.2-Beta1

2013-08-22 Thread Konstantin Belousov
On Wed, Aug 21, 2013 at 09:08:10PM -0400, Rick Macklem wrote: > Kostik wrote: > > On Tue, Aug 20, 2013 at 06:18:16PM -0400, Rick Macklem wrote: > > > J David wrote: > > > > On Thu, Aug 15, 2013 at 5:39 PM, Rick Macklem > > > > > > > > wrote: > > > > > Have you been able to pass the debugging info

Re: NFS deadlock on 9.2-Beta1

2013-08-22 Thread J David
Now that a kernel with INVARIANTS/WITNESS is finally available on a machine with serial console I am having terrible trouble provoking this to happen. (Machine grinds to a halt if I put the usual test load on it due to all the debug code in the kernel.) Did get this interesting LOR, though it did

Re: NFS deadlock on 9.2-Beta1

2013-08-21 Thread Rick Macklem
Kostik wrote: > On Tue, Aug 20, 2013 at 06:18:16PM -0400, Rick Macklem wrote: > > J David wrote: > > > On Thu, Aug 15, 2013 at 5:39 PM, Rick Macklem > > > > > > wrote: > > > > Have you been able to pass the debugging info on to Kostik? > > > > > > > > It would be really nice to get this fixed for

Re: NFS deadlock on 9.2-Beta1

2013-08-21 Thread Konstantin Belousov
On Wed, Aug 21, 2013 at 08:03:35PM +0200, Yamagi Burmeister wrote: > Could the problem be related to this deadlock / LOR? - > http://lists.freebsd.org/pipermail/freebsd-fs/2013-August/018052.html This is not related. > > My test setup is still in place. Will test with r250907 reverted > tomorrow

Re: NFS deadlock on 9.2-Beta1

2013-08-21 Thread Yamagi Burmeister
On Wed, 21 Aug 2013 16:10:32 +0300 Konstantin Belousov wrote: > I already described what to do with this. I need the debugging > information to see what is going on. Without the data, it is only > wasted time of everybody involved. > > Some technical notes. The sendfile() uses shared lock for

Re: NFS deadlock on 9.2-Beta1

2013-08-21 Thread Konstantin Belousov
On Tue, Aug 20, 2013 at 06:18:16PM -0400, Rick Macklem wrote: > J David wrote: > > On Thu, Aug 15, 2013 at 5:39 PM, Rick Macklem > > wrote: > > > Have you been able to pass the debugging info on to Kostik? > > > > > > It would be really nice to get this fixed for FreeBSD9.2. > > > > You're probab

Re: NFS deadlock on 9.2-Beta1

2013-08-20 Thread Oliver Pinter
On 8/20/13, J David wrote: > On Thu, Aug 15, 2013 at 5:39 PM, Rick Macklem wrote: >> Have you been able to pass the debugging info on to Kostik? >> >> It would be really nice to get this fixed for FreeBSD9.2. > > You're probably not talking to me, but headway here is slow. At our > location, we

Re: NFS deadlock on 9.2-Beta1

2013-08-20 Thread Rick Macklem
J David wrote: > On Thu, Aug 15, 2013 at 5:39 PM, Rick Macklem > wrote: > > Have you been able to pass the debugging info on to Kostik? > > > > It would be really nice to get this fixed for FreeBSD9.2. > > You're probably not talking to me, but headway here is slow. At our > location, we have be

Re: NFS deadlock on 9.2-Beta1

2013-08-19 Thread J David
On Thu, Aug 15, 2013 at 5:39 PM, Rick Macklem wrote: > Have you been able to pass the debugging info on to Kostik? > > It would be really nice to get this fixed for FreeBSD9.2. You're probably not talking to me, but headway here is slow. At our location, we have been continuing to test releng/9.

Re: NFS deadlock on 9.2-Beta1

2013-08-15 Thread Rick Macklem
Michael Tratz wrote: > > On Jul 27, 2013, at 11:25 PM, Konstantin Belousov > wrote: > > > On Sat, Jul 27, 2013 at 03:13:05PM -0700, Michael Tratz wrote: > >> Let's assume the pid which started the deadlock is 14001 (it will > >> be a different pid when we get the results, because the machine > >

Re: NFS deadlock on 9.2-Beta1

2013-08-05 Thread J David
On Mon, Aug 5, 2013 at 12:06 PM, Mark Saad wrote: > Is there any updates on this issue ? Has anyone tested it or see it happen > on the release candidate ? It's a bit premature for that; the RC has been out for a few hours. We put BETA2 on 25 nodes and only saw the problem on five after 24 hou

Re: NFS deadlock on 9.2-Beta1

2013-08-05 Thread Mark Saad
On Jul 29, 2013, at 10:48 PM, J David wrote: > If it is helpful, we have 25 nodes testing the 9.2-BETA1 build and > without especially trying to exercise this bug, we found > sendfile()-using processes deadlocked in WCHAN newnfs on 5 of the 25 > nodes. The ones with highest uptime (about 3 day

Re: NFS deadlock on 9.2-Beta1

2013-07-29 Thread J David
If it is helpful, we have 25 nodes testing the 9.2-BETA1 build and without especially trying to exercise this bug, we found sendfile()-using processes deadlocked in WCHAN newnfs on 5 of the 25 nodes. The ones with highest uptime (about 3 days) seem most affected, so it does seem like a "sooner or

Re: NFS deadlock on 9.2-Beta1

2013-07-29 Thread Rick Macklem
Michael Tratz wrote: > > On Jul 27, 2013, at 11:25 PM, Konstantin Belousov > wrote: > > > On Sat, Jul 27, 2013 at 03:13:05PM -0700, Michael Tratz wrote: > >> Let's assume the pid which started the deadlock is 14001 (it will > >> be a different pid when we get the results, because the machine > >

Re: NFS deadlock on 9.2-Beta1

2013-07-29 Thread Michael Tratz
On Jul 27, 2013, at 11:25 PM, Konstantin Belousov wrote: > On Sat, Jul 27, 2013 at 03:13:05PM -0700, Michael Tratz wrote: >> Let's assume the pid which started the deadlock is 14001 (it will be a >> different pid when we get the results, because the machine has been >> restarted) >> >> I type

Re: NFS deadlock on 9.2-Beta1

2013-07-27 Thread Konstantin Belousov
On Sat, Jul 27, 2013 at 03:13:05PM -0700, Michael Tratz wrote: > Let's assume the pid which started the deadlock is 14001 (it will be a > different pid when we get the results, because the machine has been restarted) > > I type: > > show proc 14001 > > I get the thread numbers from that output

Re: NFS deadlock on 9.2-Beta1

2013-07-27 Thread Michael Tratz
On Jul 27, 2013, at 1:58 PM, Konstantin Belousov wrote: > On Sat, Jul 27, 2013 at 04:20:49PM -0400, Rick Macklem wrote: >> Michael Tratz wrote: >>> >>> On Jul 24, 2013, at 5:25 PM, Rick Macklem >>> wrote: >>> Michael Tratz wrote: > Two machines (NFS Server: running ZFS / Client: disk

Re: NFS deadlock on 9.2-Beta1

2013-07-27 Thread Michael Tratz
On Jul 27, 2013, at 1:20 PM, Rick Macklem wrote: > Michael Tratz wrote: >> >> On Jul 24, 2013, at 5:25 PM, Rick Macklem >> wrote: >> >>> Michael Tratz wrote: Two machines (NFS Server: running ZFS / Client: disk-less), both are running FreeBSD r253506. The NFS client starts to d

Re: NFS deadlock on 9.2-Beta1

2013-07-27 Thread Konstantin Belousov
On Sat, Jul 27, 2013 at 04:20:49PM -0400, Rick Macklem wrote: > Michael Tratz wrote: > > > > On Jul 24, 2013, at 5:25 PM, Rick Macklem > > wrote: > > > > > Michael Tratz wrote: > > >> Two machines (NFS Server: running ZFS / Client: disk-less), both > > >> are > > >> running FreeBSD r253506. The

Re: NFS deadlock on 9.2-Beta1

2013-07-27 Thread Rick Macklem
Michael Tratz wrote: > > On Jul 24, 2013, at 5:25 PM, Rick Macklem > wrote: > > > Michael Tratz wrote: > >> Two machines (NFS Server: running ZFS / Client: disk-less), both > >> are > >> running FreeBSD r253506. The NFS client starts to deadlock > >> processes > >> within a few hours. It usually

Re: NFS deadlock on 9.2-Beta1

2013-07-26 Thread Daniel Braniss
> > On Jul 24, 2013, at 5:25 PM, Rick Macklem wrote: > > > Michael Tratz wrote: > >> Two machines (NFS Server: running ZFS / Client: disk-less), both are > >> running FreeBSD r253506. The NFS client starts to deadlock processes > >> within a few hours. It usually gets worse from there on. The >

Re: NFS deadlock on 9.2-Beta1

2013-07-25 Thread Michael Tratz
On Jul 24, 2013, at 5:25 PM, Rick Macklem wrote: > Michael Tratz wrote: >> Two machines (NFS Server: running ZFS / Client: disk-less), both are >> running FreeBSD r253506. The NFS client starts to deadlock processes >> within a few hours. It usually gets worse from there on. The >> processes sta

Re: NFS deadlock on 9.2-Beta1

2013-07-24 Thread Steven Hartland
- Original Message - From: "Rick Macklem" To: "Michael Tratz" Cc: Sent: Thursday, July 25, 2013 1:25 AM Subject: Re: NFS deadlock on 9.2-Beta1 Michael Tratz wrote: Two machines (NFS Server: running ZFS / Client: disk-less), both are running FreeBSD r253506. T

Re: NFS deadlock on 9.2-Beta1

2013-07-24 Thread Rick Macklem
Michael Tratz wrote: > Two machines (NFS Server: running ZFS / Client: disk-less), both are > running FreeBSD r253506. The NFS client starts to deadlock processes > within a few hours. It usually gets worse from there on. The > processes stay in "D" state. I haven't been able to reproduce it > when

NFS deadlock on 9.2-Beta1

2013-07-24 Thread Michael Tratz
Two machines (NFS Server: running ZFS / Client: disk-less), both are running FreeBSD r253506. The NFS client starts to deadlock processes within a few hours. It usually gets worse from there on. The processes stay in "D" state. I haven't been able to reproduce it when I want it to happen. I only