Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

tn <kgri...@gmail.com>, "Alvaro Herrera" <alvhe...@2ndquadrant.com>, > "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org> > Subject: Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from > rw-conflict tracking in serializable transactions >

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

> From: "Kevin Grittner" > wrote: > > > "vmstat 1" output is as follow. Because I used only 30 cores (1/4 of all), > > cpu user time should be about 12*4 = 48. > > There seems to be no process blocked by IO. > > > > procs

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

On Sun, Jun 4, 2017 at 11:27 AM, Mengxing Liu wrote: > "vmstat 1" output is as follow. Because I used only 30 cores (1/4 of all), > cpu user time should be about 12*4 = 48. > There seems to be no process blocked by IO. > > procs ---memory--

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

On Tue, Jun 6, 2017 at 12:16 PM, Mengxing Liu wrote: > I think disk I/O is not the bottleneck in our experiment, but the global lock > is. A handy way to figure this kind of thing out is to run a query like this repeatedly during the benchmark: SELECT

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

ent Time: 2017-06-05 00:27:51 (Monday) > To: "Kevin Grittner" <kgri...@gmail.com> > Cc: "Alvaro Herrera" <alvhe...@2ndquadrant.com>, > "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org> > Subject: Re: Re: [HACKERS] Re: [GSOC 17] Elim

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

On Sat, Jun 3, 2017 at 1:51 AM, Mengxing Liu wrote: > I tried 30 cores. But the CPU utilization is about 45%~70%. > How can we distinguish where the problem is? Is disk I/O or Lock? A simple way is to run `vmstat 1` for a bit during the test. Can you post a

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

> "pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org> > Subject: Re: Re: Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from > rw-conflict tracking in serializable transactions > > > Mengxing Liu wrote: > > >> The CPU utilization of Chec

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

> Mengxing Liu wrote: >> The CPU utilization of CheckForSerializableConflictOut/In is >> 0.71%/0.69%. How many cores were on the system used for this test? The paper specifically said that they didn't see performance degradation on the PostgreSQL implementation until 32 concurrent connections

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

Mengxing Liu wrote: > Hi, Alvaro and Kevin. > > > Anyway, this is just my analysis. > > So I want to hack the PG and count the conflict lists' size of > > transactions. That would be more accurate. > > In the last week, I hacked the PG to add an additional thread to count > RWConflict list

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

Hi, Alvaro and Kevin. > Anyway, this is just my analysis. > So I want to hack the PG and count the conflict lists' size of transactions. > That would be more accurate. In the last week, I hacked the PG to add an additional thread to count RWConflict list lengths. And tune the benchmark to

[HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

Hi Mengxing, Mengxing Liu wrote: > Hi, Alvaro and Kevin. I'm Mengxing. > > This is a “synchronization” email to tell you what I've done and my next > plan. I'm looking forward to your advice. Welcome! > According to my proposal, I want to prepare the experimental environment > during

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

On Wed, Mar 15, 2017 at 11:35 AM, Mengxing Liu wrote: >> On a NUMA machine It is not at all unusual to see bifurcated results >> -- with each run coming in very close to one number or a second >> number, often at about a 50/50 rate, with no numbers falling >>

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

l.org" <pgsql-hackers@postgresql.org> > Subject: Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from > rw-conflict tracking in serializable transactions > > On Tue, Mar 14, 2017 at 3:45 PM, Kevin Grittner <kgri...@gmail.com> wrote: > >> On 3/14/1

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

On Tue, Mar 14, 2017 at 3:45 PM, Kevin Grittner wrote: >> On 3/14/17 17:34, Mengxing Liu wrote: >>> There are several alternative benchmarks. Tony suggested that we >>> should use TPC-E and TPC-DS. > > More benchmarks is better, all other things being equal. Keep in > mind

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

On Tue, Mar 14, 2017 at 6:00 AM, DEV_OPS wrote: > On 3/14/17 17:34, Mengxing Liu wrote: > The worst problems have been > seen with 32 or more cores on 4 or more sockets with a large number > of active connections. I don't know whether you have access to a >

Re: [HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

Hi Mengxing Please read my comments : On 3/14/17 17:34, Mengxing Liu wrote: > I send this email to Tony, too. Because he promised to help me with testing > and benchmarking. > The worst problems have been seen with 32 or more cores on 4 or more sockets with a large number of

[HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

I send this email to Tony, too. Because he promised to help me with testing and benchmarking. > > >> The worst problems have been > >> seen with 32 or more cores on 4 or more sockets with a large number > >> of active connections. I don't know whether you have access to a > >> machine capable

[HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

On Sat, Mar 11, 2017 at 8:39 PM, Mengxing Liu wrote: >> The worst problems have been >> seen with 32 or more cores on 4 or more sockets with a large number >> of active connections. I don't know whether you have access to a >> machine capable of putting this kind

[HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

> -Original Messages- > From: "Kevin Grittner" > Sent Time: 2017-03-12 04:24:29 (Sunday) > To: "Mengxing Liu" > Cc: "pgsql-hackers@postgresql.org" > Subject: Re: [GSOC 17] Eliminate O(N^2) scaling from

[HACKERS] Re: [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions

On Fri, Mar 10, 2017 at 9:12 PM, Mengxing Liu wrote: > My name is Mengxing Liu. I am interested in the project "Eliminate > O(N^2) scaling from rw-conflict tracking in serializable > transactions”. After discussing with Kevin off-list, I think it's > time to post