Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets )

On Fri, May 12, 2006 at 11:21:54AM -0700, Jin Guojun wrote: > I simply replaced gettimeofday() syscall with a super simple > user space code, for testing with no time accuracy, in libc > (libc.so.6.test, which is mapped for mysqld and mysql via libmap -- > see details at the end), Stronger argume

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets )

Scott Long wrote: Sven Petai wrote: On Tuesday 09 May 2006 20:35, Julian Elischer wrote: Sven Petai wrote: are there any patches that take the gettimeofday() calls and replace them with something that is cheap such as only doing every 10th one and just returning the last value ++ 1 uSec f

Re: Fine-grained locking for POSIX local sockets ( UNIX domain sockets )

On Friday 12 May 2006 01:58, Robert Watson wrote: > On Thu, 11 May 2006, Scott Long wrote: > >> So I guess the real question is: do we want to merge the UNIX domain > >> socket locking work? The MySQL gains sound good, the performance drop > >> under very high load seems problematic, and there are

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Thu, 11 May 2006, Scott Long wrote: So I guess the real question is: do we want to merge the UNIX domain socket locking work? The MySQL gains sound good, the performance drop under very high load seems problematic, and there are more general questions about performance with other workload

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Robert Watson wrote: On Wed, 10 May 2006, David Xu wrote: Fixing one of big lock contentions is not enough, you have to fix them all, it is easy to see that a second contention becomes a top one. :-) So I guess the real question is: do we want to merge the UNIX domain socket locking work?

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Wed, 10 May 2006, David Xu wrote: Fixing one of big lock contentions is not enough, you have to fix them all, it is easy to see that a second contention becomes a top one. :-) So I guess the real question is: do we want to merge the UNIX domain socket locking work? The MySQL gains sound

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets )

Sven Petai wrote: On Wednesday 10 May 2006 04:38, Scott Long wrote: Were you testing on SMP, and if so, was Hyperthreading enabled? I tested it on 3 amd64 machines, so no hyperthreading possible. The library preload hack was tested on: 1 * dualcore amd X2 4 * dualcore opteron Hacked mys

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Wed, 2006-May-10 10:42:39 +0200, Suleiman Souhlal wrote: >That's not enough. On some CPUs (like the current Opterons), the TSC >slows down when the CPU executes a HLT instruction, so if you want good >accuracy, you'll need to take that into account too. The CPU can't be executing any instruct

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Paul Allen wrote: From Julian Elischer <[EMAIL PROTECTED]>, Tue, May 09, 2006 at 10:35:06AM -0700: Sven Petai wrote: are there any patches that take the gettimeofday() calls and replace them with something that is cheap such as only doing every 10th one and just returning the last value ++ 1

Re: Fine-grained locking for POSIX local sockets ( UNIX domain sockets )

On Wednesday 10 May 2006 04:38, Scott Long wrote: > > Were you testing on SMP, and if so, was Hyperthreading enabled? > I tested it on 3 amd64 machines, so no hyperthreading possible. The library preload hack was tested on: 1 * dualcore amd X2 4 * dualcore opteron Hacked mysqld that used TSC w

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Peter Jeremy wrote: On Wed, 2006-May-10 10:42:39 +0200, Suleiman Souhlal wrote: That's not enough. On some CPUs (like the current Opterons), the TSC slows down when the CPU executes a HLT instruction, so if you want good accuracy, you'll need to take that into account too. The CPU can't be

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Tim Kientzle wrote: I recall Matt talking about implementing gettimeofday() without a syscall. The basic idea is to have the kernel record some constants in a page that's mapped across all processes, then libc can just read the time from a known location. It might be nice to combine this with

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets )

Sven Petai wrote: On Tuesday 09 May 2006 20:35, Julian Elischer wrote: Sven Petai wrote: are there any patches that take the gettimeofday() calls and replace them with something that is cheap such as only doing every 10th one and just returning the last value ++ 1 uSec for the other ones.. a

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

David Xu wrote: On Wednesday 10 May 2006 02:23, Kris Kennaway wrote: There are at least several issues here: ...This may be general issues like gettimeofday() on Linux vs FreeBSD; clearly there is something *very big* to blame here. Mysql does do *lots* of such calls ... My last recall is t

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets )

On Tuesday 09 May 2006 20:35, Julian Elischer wrote: > Sven Petai wrote: > > are there any patches that take the gettimeofday() calls and replace > them with something that is cheap > such as only doing every 10th one and just returning the last value ++ 1 > uSec for the other ones.. > > a ktrace o

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Wednesday 10 May 2006 02:23, Kris Kennaway wrote: > There are at least several issues here: > > * Factor of >two difference in performance across the board (all > loads) relative to Linux. This may be general issues like > gettimeofday() on Linux vs FreeBSD; clearly there is something *very > b

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

>From Julian Elischer <[EMAIL PROTECTED]>, Tue, May 09, 2006 at 10:35:06AM >-0700: > Sven Petai wrote: > > are there any patches that take the gettimeofday() calls and replace > them with something that is cheap > such as only doing every 10th one and just returning the last value ++ 1 > uSec f

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Tue, May 09, 2006 at 11:13:02AM -0700, Greg 'groggy' Lehey wrote: > On Monday, 8 May 2006 at 21:11:09 -0400, Kris Kennaway wrote: > > On Mon, May 08, 2006 at 08:43:28PM -0400, Kris Kennaway wrote: > >> On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: > >>> OK, David's patch fixes

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Tue, May 09, 2006 at 06:18:11PM +0300, Sven Petai wrote: > On Tuesday 09 May 2006 03:42, Kris Kennaway wrote: > > On Tue, May 09, 2006 at 03:34:59AM +0300, Sven Petai wrote: > > > > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > > > > yet had time to investigate. Is anyo

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Monday, 8 May 2006 at 21:11:09 -0400, Kris Kennaway wrote: > On Mon, May 08, 2006 at 08:43:28PM -0400, Kris Kennaway wrote: >> On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: >>> OK, David's patch fixes the umtx thundering herd (and seems to give a >>> 4-6% boost). I also fixed

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Tue, 9 May 2006, 10:35-0700, Julian Elischer wrote: > Sven Petai wrote: > > are there any patches that take the gettimeofday() calls and replace > them with something that is cheap such as only doing every 10th one > and just returning the last value ++ 1 uSec for the other ones.. > > a ktrace

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Sven Petai wrote: are there any patches that take the gettimeofday() calls and replace them with something that is cheap such as only doing every 10th one and just returning the last value ++ 1 uSec for the other ones.. a ktrace of Mysql shows a LOT of gettimeofday() calls. On Tuesday 09 M

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Tuesday 09 May 2006 03:42, Kris Kennaway wrote: > On Tue, May 09, 2006 at 03:34:59AM +0300, Sven Petai wrote: > > > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > > > yet had time to investigate. Is anyone else seeing this? > > > > Seems to run fine here with 4.1.18 on a

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Kris Kennaway wrote: On Mon, May 08, 2006 at 06:18:18PM -0700, Maxim Sobolev wrote: Kris Kennaway wrote: On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: OK, David's patch fixes the umtx thundering herd (and seems to give a 4-6% boost). I also fixed a thundering herd in FILEDESC

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Kris Kennaway wrote: On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: OK, David's patch fixes the umtx thundering herd (and seems to give a 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which was also waking up 2-7 CPUs at once about 30% of the time) by doing s/w

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Mon, May 08, 2006 at 06:18:18PM -0700, Maxim Sobolev wrote: > Kris Kennaway wrote: > >On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: > >>OK, David's patch fixes the umtx thundering herd (and seems to give a > >>4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (whic

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Mon, May 08, 2006 at 08:43:28PM -0400, Kris Kennaway wrote: > On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: > > OK, David's patch fixes the umtx thundering herd (and seems to give a > > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > > was also waking up

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Mon, May 08, 2006 at 02:52:07AM -0400, Kris Kennaway wrote: > OK, David's patch fixes the umtx thundering herd (and seems to give a > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > was also waking up 2-7 CPUs at once about 30% of the time) by doing > s/wakeup/wakeup_one

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Tue, May 09, 2006 at 03:34:59AM +0300, Sven Petai wrote: > > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > > yet had time to investigate. Is anyone else seeing this? > > > > Seems to run fine here with 4.1.18 on amd64, but doesn't seem to make much > difference though

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

> Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > yet had time to investigate. Is anyone else seeing this? > Seems to run fine here with 4.1.18 on amd64, but doesn't seem to make much difference though. I ran the tests again on the 8 core machine with and without rwatsons

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

David Xu wrote: On Tuesday 09 May 2006 02:43, Kris Kennaway wrote: Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't yet had time to investigate. Is anyone else seeing this? Kris I only have tested mysql 4.0, I will try 4.1 later. What about 5.0? Have any of

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Tuesday 09 May 2006 02:43, Kris Kennaway wrote: > Hmm, with this patch mysql 4.1 seems to crash at startup. I haven't > yet had time to investigate. Is anyone else seeing this? > > Kris I only have tested mysql 4.0, I will try 4.1 later. ___ freebs

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Mon, May 08, 2006 at 06:43:31PM +0800, David Xu wrote: > On Monday 08 May 2006 14:52, Kris Kennaway wrote: > > OK, David's patch fixes the umtx thundering herd (and seems to give a > > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > > was also waking up 2-7 CPUs at once

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Monday 08 May 2006 14:52, Kris Kennaway wrote: > OK, David's patch fixes the umtx thundering herd (and seems to give a > 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which > was also waking up 2-7 CPUs at once about 30% of the time) by doing > s/wakeup/wakeup_one/. This did

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Mon, 8 May 2006, Kris Kennaway wrote: unp contention has risen a bit. The other big gain is to sleep mtxpool contention, which roughly doubled: In the general case, you can increase the size of the mutex pool. However, since this is per-uid for the socket buffer resource limits, that wo

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

OK, David's patch fixes the umtx thundering herd (and seems to give a 4-6% boost). I also fixed a thundering herd in FILEDESC_UNLOCK (which was also waking up 2-7 CPUs at once about 30% of the time) by doing s/wakeup/wakeup_one/. This did not seem to give a performance impact on this test though.

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 07:47:27PM +0100 I heard the voice of Robert Watson, and lo! it spake thus: > > In past discussion, I think a reasonable conclusion has been some > amount of both. I've still got sitting around in my bookmarks f

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Mon, May 08, 2006 at 07:22:06AM +0800, David Xu wrote: > On Monday 08 May 2006 07:04, Kris Kennaway wrote: > > > i.e. apparently not a large difference, but still a large proportion > > of cases where multiple CPUs are woken at once on the same chain. > > > > Kris > This becauses there is no s

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Monday 08 May 2006 07:04, Kris Kennaway wrote: > i.e. apparently not a large difference, but still a large proportion > of cases where multiple CPUs are woken at once on the same chain. > > Kris This becauses there is no sleepable mutex available, so I had to use msleep and wakeup, this is subo

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 05:41:53PM -0400, Kris Kennaway wrote: > static int > kern_sigtimedwait(struct thread *td, sigset_t waitset, ksiginfo_t *ksi, > struct timespec *timeout) > { > ... > td->td_sigmask = savedmask; > SIGSETNAND(td->td_sigmask, waitset); > signoti

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 05:04:26PM -0400, Kris Kennaway wrote: > >477 23472709 2810986 8 5671248 1900047 > > kern/kern_synch.c:220 (process lock) > > > > The top 10 heavily contended mutexes are very different (but note the > > number of mutex acquisitions, column 3, is

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sat, May 06, 2006 at 06:19:08PM -0400, Kris Kennaway wrote: > x norwatson-8 > + rwatson-8 > ++ > | + | > | + + + x x| > |+

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 11:27:22PM +0300, Sven Petai wrote: > On Sunday 07 May 2006 22:16, you wrote: > > On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > > > The results in my mail were mean values over 2 runs, > > > only once did I see really huge (more than 10%) differences between

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sunday 07 May 2006 22:16, you wrote: > On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > > The results in my mail were mean values over 2 runs, > > only once did I see really huge (more than 10%) differences between > > several subsequent runs with same settings, this case was clearl

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, 7 May 2006, Kris Kennaway wrote: Typically, I do 12 runs of supersmack in each configuration, and discard the first 2 runs in which the cache and scheduler (etc) are still settling, as I'm interested in the steady state. Yeah, forgot to mention that too. Also keeping in mind that my

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 08:57:21PM +0100, Robert Watson wrote: > > >On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > > > >>The results in my mail were mean values over 2 runs, > >>only once did I see really huge (more than 10%) differences between > >>several > >>subsequent runs with

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: The results in my mail were mean values over 2 runs, only once did I see really huge (more than 10%) differences between several subsequent runs with same settings, this case was clearly mentioned in the results. FYI, 2 is not real

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 10:00:41PM +0300, Sven Petai wrote: > The results in my mail were mean values over 2 runs, > only once did I see really huge (more than 10%) differences between several > subsequent runs with same settings, this case was clearly mentioned in the > results. FYI, 2 is not

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Kris Kennaway wrote: Also, I see a slow but statistically significant deterioration in performance over time. Maybe mysql's memory is getting fragmented or something. I have tools to qualitatively assess fragmentation. I can help with this, should you be interested in looking into it. Jas

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

> Anyhow, what I'm getting at is this: make sure when you measure MySQL > performance, you do a series of runs, discard the first few, and then take > an average of the remainder, and watch out for outliers. The results in my mail were mean values over 2 runs, only once did I see really huge (more

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, 7 May 2006, Mike Jakubik wrote: The difference in performance is just ridiculous. Is mysql written to be slow on freebsd or is there a problem with freebsd? In past discussion, I think a reasonable conclusion has been some amount of both. We've identified a few particular areas wher

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Sven Petai wrote: scheduler: ULE thr_lib socket nicequeries threads update select thr unix0 1 100 477913724 thr unix0 10 10 647325172 thr unix-10 1 100 496920662 thr unix-10 10 10 6418

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, May 07, 2006 at 07:16:34PM +0100, Robert Watson wrote: > > On Sun, 7 May 2006, Sven Petai wrote: > > >I performed tests on a 4 * dualcore 2Ghz opteron system (so 8 cores in > >total). > > > >In general with 10 parallel smacker threads the performance seems to go up > >with your patch by

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sun, 7 May 2006, Sven Petai wrote: I performed tests on a 4 * dualcore 2Ghz opteron system (so 8 cores in total). In general with 10 parallel smacker threads the performance seems to go up with your patch by ~44% and with 100 parallel threads it goes down ~25% This is an interesting eff

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

hi I performed tests on a 4 * dualcore 2Ghz opteron system (so 8 cores in total). In general with 10 parallel smacker threads the performance seems to go up with your patch by ~44% and with 100 parallel threads it goes down ~25% Here's a graph of select smack performance with your patch: http://

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Saturday 06 May 2006 22:16, Robert Watson wrote: > Dear all, > > Attached, please find a patch implementing more fine-grained locking for > the POSIX local socket subsystem (UNIX domain socket subsystem). In the > current implementation, we use a single global subsystem lock to protect > all lo

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sunday 07 May 2006 06:39, Attila Nagy wrote: > On 2006. 05. 06. 22:50, Robert Watson wrote: > >> The machine is a quad core Xeon LV server, the client side is > >> sysbench, accessing mysql 4.1.8 on a socket. Heap table, simple test. > > > > Which threading library is that with, btw? If libpthr

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On 2006. 05. 06. 22:50, Robert Watson wrote: The machine is a quad core Xeon LV server, the client side is sysbench, accessing mysql 4.1.8 on a socket. Heap table, simple test. Which threading library is that with, btw? If libpthread, could you run the same test with libthr, and vice versa? thr

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sat, May 06, 2006 at 03:16:48PM +0100, Robert Watson wrote: > > Dear all, > > Attached, please find a patch implementing more fine-grained locking for > the POSIX local socket subsystem (UNIX domain socket subsystem). Dear Sir, Per your request, please find attached the results of my measur

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sat, 6 May 2006, Attila Nagy wrote: On 2006. 05. 06. 16:16, Robert Watson wrote: In local measurements, I have observed a 0% change in MySQL performance on uniprocessor systems, and on a dual-processor system I have observed a 4%-5% performance improvement with two client MySQL threads. Ju

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Hello, On 2006. 05. 06. 16:16, Robert Watson wrote: In local measurements, I have observed a 0% change in MySQL performance on uniprocessor systems, and on a dual-processor system I have observed a 4%-5% performance improvement with two client MySQL threads. Just a quick, nowhere correct test:

Re: Fine-grained locking for POSIX local sockets (UNIX domain sockets)

On Sat, 6 May 2006, Robert Watson wrote: Attached, please find a patch implementing more fine-grained locking for the POSIX local socket subsystem (UNIX domain socket subsystem). In the current implementation, we use a single global subsystem lock to protect all local IPC over the PF_LOCAL s

Fine-grained locking for POSIX local sockets (UNIX domain sockets)

Dear all, Attached, please find a patch implementing more fine-grained locking for the POSIX local socket subsystem (UNIX domain socket subsystem). In the current implementation, we use a single global subsystem lock to protect all local IPC over the PF_LOCAL socket type. This has low overh