Re: [HACKERS] Separate connection handling from backends

2016-12-07 Thread Craig Ringer
On 7 December 2016 at 22:27, Kevin Grittner wrote: > I don't know how that execution model would compare to what we use > now in terms of performance, but its popularity makes it hard to > ignore as something to consider. Those engines also tend to be threaded. They can stash state in memory and

Re: [HACKERS] Separate connection handling from backends

2016-12-07 Thread Kevin Grittner
On Wed, Dec 7, 2016 at 12:36 AM, Jim Nasby wrote: > The way I'm picturing it backends would no longer be directly > tied to connections. The code that directly handles connections > would grab an available backend when a statement actually came in > (and certainly it'd need to worry about transac

Re: [HACKERS] Separate connection handling from backends

2016-12-06 Thread Jim Nasby
On 12/6/16 10:34 PM, Craig Ringer wrote: In other words, we could start to separate session state from executor state in a limited manner. That'd definitely be valuable, IMO; it's a real shame that Pg's architecture so closely couples the two. So - is just doing "PgInCoreBouncer" a good idea? No

Re: [HACKERS] Separate connection handling from backends

2016-12-06 Thread Jim Nasby
On 12/6/16 6:19 PM, Tom Lane wrote: I'm kind of mystified how a simple code restructuring could solve the fundamental problems with a large number of backends. It sounds like what you're describing would just push the problem around, you would end up with some other maximum instead, max_backends,

Re: [HACKERS] Separate connection handling from backends

2016-12-06 Thread Craig Ringer
On 7 December 2016 at 10:19, Tom Lane wrote: > What it sounds like to me is building a connection pooler into the > backend. I'm not really convinced we ought to go there. If we do, it probably needs to be able to offer things that out-of-tree ones can't. The main things I see that you can't d

Re: [HACKERS] Separate connection handling from backends

2016-12-06 Thread Jim Nasby
On 12/6/16 1:46 PM, Adam Brusselback wrote: BTW, it just occurred to me that having this separation would make it relatively easy to support re-directing DML queries from a replica to the master; if the backend throws the error indicating you tried to write data, the connection la

Re: [HACKERS] Separate connection handling from backends

2016-12-06 Thread Tom Lane
Greg Stark writes: > On 5 December 2016 at 19:48, Jim Nasby wrote: >> One solution to this would be to segregate connection handling from actual >> backends, somewhere along the lines of separating the main loop from the >> switch() that handles libpq commands. Benefits: > I'm kind of mystified

Re: [HACKERS] Separate connection handling from backends

2016-12-06 Thread Greg Stark
On 5 December 2016 at 19:48, Jim Nasby wrote: > One solution to this would be to segregate connection handling from actual > backends, somewhere along the lines of separating the main loop from the > switch() that handles libpq commands. Benefits: I'm kind of mystified how a simple code restructu

Re: [HACKERS] Separate connection handling from backends

2016-12-06 Thread Adam Brusselback
> > BTW, it just occurred to me that having this separation would make it > relatively easy to support re-directing DML queries from a replica to the > master; if the backend throws the error indicating you tried to write data, > the connection layer could re-route that. This also sounds like it

Re: [HACKERS] Separate connection handling from backends

2016-12-06 Thread Kevin Grittner
On Mon, Dec 5, 2016 at 6:54 PM, Jim Nasby wrote: > On 12/5/16 2:14 PM, David Fetter wrote: >> What do you see as the relationship between this proposal and the >> earlier one for admission control? >> >> https://www.postgresql.org/message-id/4b38c1c502250002d...@gw.wicourts.gov > > Without ha

Re: [HACKERS] Separate connection handling from backends

2016-12-05 Thread Jim Nasby
On 12/5/16 2:14 PM, David Fetter wrote: One solution to this would be to segregate connection handling from actual backends, somewhere along the lines of separating the main loop from the switch() that handles libpq commands. Benefits: [interesting stuff elided] What do you see as the relations

Re: [HACKERS] Separate connection handling from backends

2016-12-05 Thread David Fetter
On Mon, Dec 05, 2016 at 01:48:03PM -0600, Jim Nasby wrote: > max_connections is a frequent point of contention between users and > developers. Users want to set it high so they don't have to deal with Yet > More Software (pgpool or pgBouncer); PG developers freak out because > backends are pretty h

[HACKERS] Separate connection handling from backends

2016-12-05 Thread Jim Nasby
max_connections is a frequent point of contention between users and developers. Users want to set it high so they don't have to deal with Yet More Software (pgpool or pgBouncer); PG developers freak out because backends are pretty heavyweight, there's some very hot code that's sensitive to the