Re: [OpenAFS] Re: Tuning the -daemons.

2011-07-25 Thread Jan Johansson
Jan Johansson wrote: > I will try my best to post what we did in the end. After another hang I was able to get a thread dump and it matched the dynamic vcache problem so we added "-disable-dynamic-vcaches" to the cache manager and it has been trouble free since. Thank you for the invaluable help

Re: [OpenAFS] Re: Tuning the -daemons.

2011-04-28 Thread Jan Johansson
Andrew Deason wrote: > It suggests that it could be the problem, but technically > really anything holding xvcache could cause that (or anything > else causing the callback thread to hang). But certainly the > issue in this thread is the most likely cause. > > If you want to really be sure that t

[OpenAFS] Re: Tuning the -daemons.

2011-04-28 Thread Andrew Deason
On Thu, 28 Apr 2011 10:46:25 +0200 Jan Johansson wrote: > So when reading the thread more closely I found a command that I > had missed. > > cmdebug > > So this time around I tried it when the IMAP server broke and got no > response (it timed out). > > Would it be correct to assume that this

Re: [OpenAFS] Re: Tuning the -daemons.

2011-04-28 Thread Jan Johansson
Andrew Deason wrote: > Assuming I am correctly thinking of the same issue... to > clarify: technically it's the thread handling incoming RPCs, > not the thread handling all Rx traffic. But the result is > xvcache staying write-locked for a long time, so the whole CM > pretty much grinds to a halt.

[OpenAFS] Re: Tuning the -daemons.

2011-04-20 Thread Andrew Deason
On Tue, 19 Apr 2011 10:21:10 -0500 Andrew Deason wrote: > The underlying issue I think has always existed: xvcache must be > write-locked for vcache traversal, and we traverse vcaches looking for > something to flush, and a flush may hit a fileserver for a > GiveUpCallBacks call when we flush VCB

Re: [OpenAFS] Re: Tuning the -daemons.

2011-04-19 Thread Harald Barth
> Maybe dynamic vcaches made this more likely to be hit, though (which > would be 1.4.10, Linux-only). That makes sense as I think we were running something that was 1.4.9-ish a long time without seeing any such issues. Harald. ___ OpenAFS-info mailing

Re: [OpenAFS] Re: Tuning the -daemons.

2011-04-19 Thread Simon Wilkinson
On 19 Apr 2011, at 16:21, Andrew Deason wrote: > > Maybe dynamic vcaches made this more likely to be hit, though (which > would be 1.4.10, Linux-only). Before/without those, I think you have to > run out of free vcache entries before you hit the relevant code path, > which I expect happens less o

[OpenAFS] Re: Tuning the -daemons.

2011-04-19 Thread Andrew Deason
On Tue, 19 Apr 2011 14:54:38 +0200 (CEST) Harald Barth wrote: > > We believe that this behaviour is fixed in 1.6.0pre4. > > Do you have any idea when it was introduced? The underlying issue I think has always existed: xvcache must be write-locked for vcache traversal, and we traverse vcaches lo

[OpenAFS] Re: Tuning the -daemons.

2011-04-19 Thread Andrew Deason
On Tue, 19 Apr 2011 09:28:17 +0200 Jan Johansson wrote: > Simon Wilkinson wrote: > > Reviewing your original post, it has occurred to me that your > > problem could be a symptom of an issue a number of sites are seeing > > with callback breaks. Essentially, it is possible for the thread in > > c

Re: [OpenAFS] Re: Tuning the -daemons.

2011-04-19 Thread Harald Barth
> We believe that this behaviour is fixed in 1.6.0pre4. Do you have any idea when it was introduced? Harald. ___ OpenAFS-info mailing list OpenAFS-info@openafs.org https://lists.openafs.org/mailman/listinfo/openafs-info

Re: [OpenAFS] Re: Tuning the -daemons.

2011-04-19 Thread Jan Johansson
Simon Wilkinson wrote: > Reviewing your original post, it has occurred to me that your > problem could be a symptom of an issue a number of sites are > seeing with callback breaks. Essentially, it is possible for > the thread in client that handles incoming network traffic to > hang whilst handlin

Re: [OpenAFS] Re: Tuning the -daemons.

2011-04-18 Thread Simon Wilkinson
On 18 Apr 2011, at 12:33, Jan Johansson wrote: > Some time ago (in thread > https://lists.openafs.org/pipermail/openafs-info/2011-February/035407.html) > I asked about the client -daemons flag. Reviewing your original post, it has occurred to me that your problem could be a symptom of an issue a

Re: [OpenAFS] Re: Tuning the -daemons.

2011-04-18 Thread Jan Johansson
Hello! Some time ago (in thread https://lists.openafs.org/pipermail/openafs-info/2011-February/035407.html) I asked about the client -daemons flag. The simple answer I got was that there was no easy way to see how many of the daemons that where used. The helpful community also provided a number

[OpenAFS] Re: Tuning the -daemons.

2011-02-08 Thread Andrew Deason
On Mon, 7 Feb 2011 20:55:11 +0100 Jan Johansson wrote: > We had this kind of problems before. > > In the first round the client made the server crash. An upgrade > of the client from Ubuntu Karmic to Ubuntu Lucid solved that. If the client made the server crash, there was a bug in the server. C

Re: [OpenAFS] Re: Tuning the -daemons.

2011-02-07 Thread Jan Johansson
Thank you for your interest in helping out here. So I will start with the easy questions and try to get into the kernel later. Based on the History I believe that the problem is the client/cache manager. We had this kind of problems before. In the first round the client made the server crash. A

Re: [OpenAFS] Re: Tuning the -daemons.

2011-02-07 Thread Harald Barth
Jan, I'd start like this > > afs: Lost contact with file server AAA.BBB.CCC.133 in cell example.com Please upgrade to newer version and restart with -p 128 or something like this. Do this on all servers which are contacted by your mbox server. When the server does not have threads, the clien

Re: [OpenAFS] Re: Tuning the -daemons.

2011-02-07 Thread Harald Barth
> I think you mean -p 128. Yes I do (of course, for the server). > I believe Jan is asking about the client > background I/O daemons, not the number of server threads/processes. I believe that he might need both because for example afsfs4.su.se reports 18 idle threads and afs-prod-fs618.it.su.se

[OpenAFS] Re: Tuning the -daemons.

2011-02-07 Thread Andrew Deason
On Mon, 7 Feb 2011 16:53:25 +0100 Jan Johansson wrote: > Short version: > > How do I see how many of the "background daemons" that are in use? We don't offer an easy way to see this. I believe you are on Linux, in which case you can look at the process backtraces via 'echo t > /proc/sysrq-trigg

[OpenAFS] Re: Tuning the -daemons.

2011-02-07 Thread Andrew Deason
On Mon, 07 Feb 2011 18:02:23 +0100 (CET) Harald Barth wrote: > > Long version: > > > > We have a pretty busy IMAP server with Maildir's in AFS (yeah its > > probably crazy but we have been doing it for a number of years). > > Longer answer: You want to tune your servers to -daemons 128 which is