Re: Lock contention executing cvs log
MacDonald, Ian writes: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary91921517666352637== Please do not send MIME and/or HTML encrypted messages to the list. Plain text only, PLEASE! Shouldn't cvs simply be using read locks while executing the log command? It should, and it is. If that's the case, why is there a contention for the read locks? Because you need an exclusive lock to create a read lock. The exclusive lock is only held for a very short time (just long enough to create the read lock), but it's still possible for there to be contention for it. -Larry Jones I must have been delirious from having so much fun. -- Calvin ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
RE: Lock contention executing cvs log
Title: RE: Lock contention executing cvs log Thanks Larry, Sorry for the MIME post. I realized that I forgot to turn it off after sending it (and you can't un-send a message). Not sure if you saw my follow up. But I'm now thinking that the lock contention is not the source of my performance problems, just a side effect. My theory is that the CVS processes are effectively serializing themselves by virtue of using up most of the available memory while they operate on a particular directory. When a large cvs process releases enough memory for another cvs process to operate freely, the new process runs into several of other cvs processes which were also now trying to do the same thing which is acquire a read lock on the next directory. Another clue is that I can start several build cycles on different hosts at random start times (15-20 minutes apart), but they eventually become synchronized and all finish within a few seconds of each other. I guess my question now is why the cvs processed grow so large. Can you characterize how cvs should behave with respect to memory allocations when performing a log command over a set of directories? Regards, -- Ian MacDonald -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, February 28, 2003 8:13 AM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: Lock contention executing cvs log MacDonald, Ian writes: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary91921517666352637== Please do not send MIME and/or HTML encrypted messages to the list. Plain text only, PLEASE! Shouldn't cvs simply be using read locks while executing the log command? It should, and it is. If that's the case, why is there a contention for the read locks? Because you need an exclusive lock to create a read lock. The exclusive lock is only held for a very short time (just long enough to create the read lock), but it's still possible for there to be contention for it. -Larry Jones I must have been delirious from having so much fun. -- Calvin ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
RE: Lock contention executing cvs log
Title: RE: Lock contention executing cvs log (Hopefully this time at plain-text) - Sorry again Thanks Larry, Sorry for the MIME post. I realized that I forgot to turn it off after sending it (and you can't un-send a message). Not sure if you saw my follow up. But I'm now thinking that the lock contention is not the source of my performance problems, just a side effect. My theory is that the CVS processes are effectively serializing themselves by virtue of using up most of the available memory while they operate on a particular directory. When a large cvs process releases enough memory for another cvs process to operate freely, the new process runs into several of other cvs processes which were also now trying to do the same thing which is acquire a read lock on the next directory. Another clue is that I can start several build cycles on different hosts at random start times (15-20 minutes apart), but they eventually become synchronized and all finish within a few seconds of each other. I guess my question now is why the cvs processed grow so large. Can you characterize how cvs should behave with respect to memory allocations when performing a log command over a set of directories? Regards, -- Ian MacDonald -Original Message- From: MacDonald, Ian [mailto:[EMAIL PROTECTED]] Sent: Friday, February 28, 2003 11:02 AM To: [EMAIL PROTECTED] Subject: RE: Lock contention executing cvs log Thanks Larry, Sorry for the MIME post. I realized that I forgot to turn it off after sending it (and you can't un-send a message). Not sure if you saw my follow up. But I'm now thinking that the lock contention is not the source of my performance problems, just a side effect. My theory is that the CVS processes are effectively serializing themselves by virtue of using up most of the available memory while they operate on a particular directory. When a large cvs process releases enough memory for another cvs process to operate freely, the new process runs into several of other cvs processes which were also now trying to do the same thing which is acquire a read lock on the next directory. Another clue is that I can start several build cycles on different hosts at random start times (15-20 minutes apart), but they eventually become synchronized and all finish within a few seconds of each other. I guess my question now is why the cvs processed grow so large. Can you characterize how cvs should behave with respect to memory allocations when performing a log command over a set of directories? Regards, -- Ian MacDonald -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, February 28, 2003 8:13 AM To: [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: Lock contention executing cvs log MacDonald, Ian writes: MIME-Version: 1.0 Content-Type: multipart/mixed; boundary91921517666352637== Please do not send MIME and/or HTML encrypted messages to the list. Plain text only, PLEASE! Shouldn't cvs simply be using read locks while executing the log command? It should, and it is. If that's the case, why is there a contention for the read locks? Because you need an exclusive lock to create a read lock. The exclusive lock is only held for a very short time (just long enough to create the read lock), but it's still possible for there to be contention for it. -Larry Jones I must have been delirious from having so much fun. -- Calvin ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Re: Lock contention executing cvs log
MacDonald, Ian writes: While several of my build processes were running, I was monitoring the CVS server with 'top'. I noticed a strange occurrence with two of cvs processes that were currently executing a log command - both processes grew to consume nearly all available memory (in my case 512MB). While this was occurring, the cvs clients associated with these growing cvs processes started dumping the 'waiting for blahs lock' messages to stdout. This went on for about 15 minutes before the log commands completed. Does anyone know if this is normal behavior? Probably. CVS can be quite memory intensive. Now, that's generally *virtual* memory, so it doesn't matter so much. If you were running a version of top that's similar to the one I have, SIZE is the amount of virtual memory a process is using, RES is the amount of real memory. You had to have been looking at SIZE rather than RES since it isn't possible for two processes to both be using nearly all the physical memory. Don't forget that virtual memory is paged rather than processes being swapped, so the two processes don't interfere with each other -- it's not necessary for one to wait for the other to release some virtual memory before it can procede. In fact, I suspect you'll find that the processes *never* release virtual memory; they start out growing, they may or may not eventually stabilize; if they do, they stay that size until they end; they never shrink. I think what you're seeing is, indeed, lock contention. Whichever process starts first has to actually read the data off the disk and is constantly stopping and waiting for that to happen. The second process, however, finds all the data it needs in the filesystem cache (courtesy of the first process) and thus can proceed much faster until it finally catches up. Once that happens, the two process procede more-or-less in lockstep and end up running into each other the next time they try to create read locks. Waiting for a lock is a very time consuming process -- the current code just sleeps for 30 seconds and then tries again. I just checked in a change that modifies the algorithm slightly. When contention is encountered, the code now makes a few retry attempts with a very short time wait time (initially 2 us with an exponential backoff) before giving up and going into the fullblown wait with messages and the 30 second wait. That should significantly reduce the impact of contention. In my test with two simultaneous log processes, there was lots of contention, but the second process usually got through after a single retry and never needed more than 3 (I allow up to 9 retries which is a total delay of about 1ms). Of course, if there's enough contention, you'll still end up in the old 30 second wait with a message code. Better would be to make the whole waiting process use a random wait with exponential backoff; that should help avoid process that have gotten in lockstep all trying to grab the master lock at the same time. -Larry Jones I've never seen a sled catch fire before. -- Hobbes ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
Lock contention executing cvs log
Title: Lock contention executing cvs log Hi Folks, I'm sorry if this posting might appear twice. I first posted this to the NNTP group gnu.cvs.help but was not sure my posts were actually making it. Here's my problem: We have an automated build environment (CruiseControl) running on multiple hosts that poll for changes in a cvs repository using the cvs log command. We are seeing a problem that seems to be resulting in an inordinate amount of time executing the log command. When to two or more of the hosts are executing cvs -q log -N -d upperdate Lowerdate -b on the same module they seem to get stuck in a lockstep mode of waiting for each others locks with each module taking upwards of a minute. For our respository this adds up to 30 minutes to each build cycle scanning for changes. Shouldn't cvs simply be using read locks while executing the log command? If that's the case, why is there a contention for the read locks? Perhaps there is a write lock getting created? Any assistance would be appreciated. Thanks, -- Ian MacDonald ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
RE: Lock contention executing cvs log
This is a follow up with some more observations: While several of my build processes were running, I was monitoring the CVS server with 'top'. I noticed a strange occurrence with two of cvs processes that were currently executing a log command - both processes grew to consume nearly all available memory (in my case 512MB). While this was occurring, the cvs clients associated with these growing cvs processes started dumping the 'waiting for blahs lock' messages to stdout. This went on for about 15 minutes before the log commands completed. Does anyone know if this is normal behavior? BTW, in case you're curious, the log command is running against a source tree with ~1.5 GB of mixed binary and text source files. I think the biggest binary file is ~30MB. Regards, -- Ian MacDonald -Original Message- From: MacDonald, Ian [mailto:[EMAIL PROTECTED] Sent: Thursday, February 27, 2003 12:10 PM To: '[EMAIL PROTECTED]' Subject: Lock contention executing cvs log Hi Folks, I'm sorry if this posting might appear twice. I first posted this to the NNTP group gnu.cvs.help but was not sure my posts were actually making it. Here's my problem: We have an automated build environment (CruiseControl) running on multiple hosts that poll for changes in a cvs repository using the cvs log command. We are seeing a problem that seems to be resulting in an inordinate amount of time executing the log command. When to two or more of the hosts are executing cvs -q log -N -d upperdate Lowerdate -b on the same module they seem to get stuck in a lockstep mode of waiting for each others locks with each module taking upwards of a minute. For our respository this adds up to 30 minutes to each build cycle scanning for changes. Shouldn't cvs simply be using read locks while executing the log command? If that's the case, why is there a contention for the read locks? Perhaps there is a write lock getting created? Any assistance would be appreciated. Thanks, -- Ian MacDonald ___ Info-cvs mailing list [EMAIL PROTECTED] http://mail.gnu.org/mailman/listinfo/info-cvs
