Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Simon Riggs
On 8 June 2012 05:01, Tom Lane wrote: > Peter Geoghegan writes: >> On 7 June 2012 18:03, Robert Haas wrote: >>> On Thu, Jun 7, 2012 at 12:52 PM, Simon Riggs wrote: Clearly, delaying checkpoint indefinitely would be a high risk choice. But they won't be delayed indefinitely, since chan

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Peter Geoghegan writes: > On 7 June 2012 18:03, Robert Haas wrote: >> On Thu, Jun 7, 2012 at 12:52 PM, Simon Riggs wrote: >>> Clearly, delaying checkpoint indefinitely would be a high risk choice. >>> But they won't be delayed indefinitely, since changes cause WAL >>> records to be written and t

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Peter Geoghegan
On 7 June 2012 18:03, Robert Haas wrote: > On Thu, Jun 7, 2012 at 12:52 PM, Simon Riggs wrote: >> Clearly, delaying checkpoint indefinitely would be a high risk choice. >> But they won't be delayed indefinitely, since changes cause WAL >> records to be written and that would soon cause another ch

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Simon Riggs
On 7 June 2012 18:03, Robert Haas wrote: > On Thu, Jun 7, 2012 at 12:52 PM, Simon Riggs wrote: >> Clearly, delaying checkpoint indefinitely would be a high risk choice. >> But they won't be delayed indefinitely, since changes cause WAL >> records to be written and that would soon cause another ch

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Sergey Koposov
On Thu, 7 Jun 2012, Tom Lane wrote: I extended the patch to also cover DropDatabaseBuffers, FlushRelationBuffers, and FlushDatabaseBuffers, which have got the exact same type of full-pool scan loop, and committed it. Thanks everybody for the patches/commits. Sergey ***

Re: [HACKERS] log_newpage header comment

2012-06-07 Thread Tom Lane
Robert Haas writes: > It seems that in implementing ginbuildempty(), I falsified the first > "note" in the header comment for log_newpage(): > * Note: all current callers build pages in private memory and write them > * directly to smgr, rather than using bufmgr. Therefore there is no need >

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Simon Riggs
On 7 June 2012 22:54, Tom Lane wrote: > I thought it would be a lot safer and probably a little bit quicker > if we just split DropRelFileNodeBuffers into two routines, one for > the specific-fork case and one for the all-forks case; and then the > same for its main caller smgrdounlink.  So I mod

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-06-07 Thread Simon Riggs
On 7 June 2012 21:08, Andres Freund wrote: >> Moved the wakeup to a logical place outside a critical section. > Hm. I don't really like the way you implemented that. While it reduces the > likelihood quite a bit it will still miss wakeups if an XLogInsert pushes out > the data because of missing

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 7:09 PM, Jaime Casanova wrote: > On Thu, Jun 7, 2012 at 5:15 PM, Tom Lane wrote: >> Please join me in welcoming him aboard. >> > > Congratulations Kevin +1. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hac

[HACKERS] log_newpage header comment

2012-06-07 Thread Robert Haas
It seems that in implementing ginbuildempty(), I falsified the first "note" in the header comment for log_newpage(): * Note: all current callers build pages in private memory and write them * directly to smgr, rather than using bufmgr. Therefore there is no need * to pass a buffer ID to XLogIn

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Jaime Casanova
On Thu, Jun 7, 2012 at 5:15 PM, Tom Lane wrote: > Please join me in welcoming him aboard. > Congratulations Kevin -- Jaime Casanova         www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make ch

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Peter Geoghegan
On 7 June 2012 23:40, Peter Geoghegan wrote: > On 7 June 2012 23:15, Tom Lane wrote: >> Please join me in welcoming him aboard. > > Congratulations, Kevin. Idle thought for the web team: Now might be a good time to take down the blurb on .org in which Kevin describes the mailing list support as

Re: [HACKERS] WIP: parameterized function scan

2012-06-07 Thread Antonin Houska
On 05/24/2012 12:46 AM, Tom Lane wrote: Well, it's not per spec: what you did accepts queries that are invalid per spec and are very likely to be errors rather than intentional invocations of the LATERAL facility. This might be all right for I think I saw queries where function is joined with

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Peter Geoghegan
On 7 June 2012 23:15, Tom Lane wrote: > Please join me in welcoming him aboard. Congratulations, Kevin. -- Peter Geoghegan       http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To mak

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Thom Brown
On 7 June 2012 23:15, Tom Lane wrote: > I am pleased to announce that Kevin Grittner has accepted the core > committee's invitation to become our newest committer.  As you all > know, Kevin's done a good deal of work on the project over the past > couple of years.  We judged that he has the requis

Re: [HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Michael Glaesemann
On Jun 7, 2012, at 18:15, Tom Lane wrote: > I am pleased to announce that Kevin Grittner has accepted the core > committee's invitation to become our newest committer. As you all > know, Kevin's done a good deal of work on the project over the past > couple of years. We judged that he has the r

[HACKERS] New Postgres committer: Kevin Grittner

2012-06-07 Thread Tom Lane
I am pleased to announce that Kevin Grittner has accepted the core committee's invitation to become our newest committer. As you all know, Kevin's done a good deal of work on the project over the past couple of years. We judged that he has the requisite skills, dedication to the project, and a su

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
I wrote: > Simon Riggs writes: >> Both of these, as attached up thread. >> Simon's patch - dropallforks.v1.patch >> Jeff's patch - DropRelFileNodeBuffers_unlock_v1.patch >> (needs a little tidyup) > OK, will take a look. I didn't like dropallforks.v1.patch at all as presented, for several reason

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
Jeff Janes writes: >> On 30 May 2012 12:10, Heikki Linnakangas >> wrote: >>> Also, I wonder if DropRelFileNodeBuffers() could scan the pool without >>> grabbing the spinlocks on every buffer? It could do an unlocked test first, >>> and only grab the spinlock on buffers that need to be dropped. >

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 08:41:23 PM Simon Riggs wrote: > On 6 June 2012 20:11, Andres Freund wrote: > > On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: > >> Hi, > >> > >> On Monday, May 28, 2012 07:11:53 PM Tom Lane wrote: > >> > Andres Freund writes: > >> > > Does anybody have a be

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
Simon Riggs writes: > On 7 June 2012 17:34, Tom Lane wrote: >> Oh, I must be confused about which patch we are talking about --- I >> thought this was in reference to some of the WIP ideas that were being >> thrown about with respect to using lock-free access primitives.  Which >> patch are you p

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Simon Riggs writes: > On 7 June 2012 19:52, Tom Lane wrote: >> This seems like nonsense to me.  How many external scripts are likely to >> know that we skip the FF page?  There might be some, but not many. > If that is the only change in filenames, then all is forgiven. Oh, now I see what you'r

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Simon Riggs
On 7 June 2012 19:52, Tom Lane wrote: > Simon Riggs writes: >> Anything changing filenames will break every HA config anybody has >> anywhere. > > This seems like nonsense to me.  How many external scripts are likely to > know that we skip the FF page?  There might be some, but not many. If that

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Simon Riggs writes: > Anything changing filenames will break every HA config anybody has > anywhere. This seems like nonsense to me. How many external scripts are likely to know that we skip the FF page? There might be some, but not many. > So you can pretty much kiss goodbye to the idea of pg

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-06-07 Thread Simon Riggs
On 6 June 2012 20:11, Andres Freund wrote: > On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: >> Hi, >> >> On Monday, May 28, 2012 07:11:53 PM Tom Lane wrote: >> > Andres Freund writes: >> > > Does anybody have a better idea than to either call WalSndWakeup() at >> > > essentially the wr

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 1:15 PM, Kevin Grittner wrote: > Simon Riggs wrote: > >> Anything changing filenames will break every HA config anybody has >> anywhere. > > It will impact our scripts related to backup and archiving, but I > think we're talking about two or three staff days to cover it in

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 1:40 PM, Kevin Grittner wrote: > Robert Haas wrote: > >> But if you're just using regexp matching against pathnames, your >> tool will be just fine.  Do your tools actually rely on the >> occasional absence of a file in what would otherwise be the usual >> sequence of files

Re: [HACKERS] Early hint bit setting

2012-06-07 Thread Robert Haas
On Wed, Jun 6, 2012 at 6:41 PM, Jim Nasby wrote: > Except that's only true when there are no other transactions running. That's > been one of the big sticking points about trying to proactively set hint > bits; in a real system you're not going to gain very much unless you wait a > while before se

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Kevin Grittner
Robert Haas wrote: > But if you're just using regexp matching against pathnames, your > tool will be just fine. Do your tools actually rely on the > occasional absence of a file in what would otherwise be the usual > sequence of files? To save "snapshot" backups for the long term, we generate

Re: [HACKERS] Could we replace SysV semaphores with latches?

2012-06-07 Thread Stefan Kaltenbrunner
On 06/07/2012 06:09 AM, Tom Lane wrote: > There has been regular griping in this list about our dependence on SysV > shared memory, but not so much about SysV semaphores, even though the > latter cause their fair share of issues; as seen for example in > buildfarm member spoonbill's recent string o

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Robert Haas writes: > ... It's better to have a few unnecessary > checkpoints than to risk losing somebody's data, especially since the > unnecessary checkpoints only happen with wal_level=hot_standby, but > the data loss risk exists for everyone. Yeah, that's another point here: the benefit of t

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Kevin Grittner
Simon Riggs wrote: > Anything changing filenames will break every HA config anybody has > anywhere. It will impact our scripts related to backup and archiving, but I think we're talking about two or three staff days to cover it in our shop. We should definitely make sure that this change is

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 07:03:32 PM Simon Riggs wrote: > On 7 June 2012 17:12, Heikki Linnakangas > > wrote: > > On 07.06.2012 18:51, Simon Riggs wrote: > >> On 7 June 2012 14:50, Heikki Linnakangas > >> > >> wrote: > >>> These changes will help the XLogInsert scaling patch > >> > >> ...an

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Simon Riggs writes: > On 7 June 2012 17:27, Tom Lane wrote: >> Simon Riggs writes: >>> If that is the concern, then its a one line fix to add the missing clog >>> flush. >> To where, and what performance impact will that have? > To the point where we decide to skip the other parts of the > ch

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Simon Riggs
On 7 June 2012 17:12, Heikki Linnakangas wrote: > On 07.06.2012 18:51, Simon Riggs wrote: >> >> On 7 June 2012 14:50, Heikki Linnakangas >>  wrote: >> >>> These changes will help the XLogInsert scaling patch >> >> >> ...and as I'm sure you're aware will junk much of the replication code >> and al

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 12:52 PM, Simon Riggs wrote: > Clearly, delaying checkpoint indefinitely would be a high risk choice. > But they won't be delayed indefinitely, since changes cause WAL > records to be written and that would soon cause another checkpoint. But that's exactly the problem - it

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:53:58 PM Tom Lane wrote: > Andres Freund writes: > > dance. If the record can be smeared over two pages there is no point in > > storing it aligned. > > I think this is not true. The value of requiring alignment is that you > can read the record-length field without

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Simon Riggs
On 7 June 2012 17:34, Tom Lane wrote: > Simon Riggs writes: >> On 7 June 2012 14:56, Tom Lane wrote: >>> Say what?  That's a performance result and proves not a damn thing about >>> safety. > >> Of course not. > >> Based on the rationale explained in the code comments in the patch, it >> seems l

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Andres Freund writes: > dance. If the record can be smeared over two pages there is no point in > storing it aligned. I think this is not true. The value of requiring alignment is that you can read the record-length field without first having to copy it somewhere. In particular, it will get rea

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Simon Riggs
On 7 June 2012 17:27, Tom Lane wrote: > Simon Riggs writes: >>> Robert Haas wrote: I know of real customers who would have suffered real data loss had this code been present in the server version they were using. > >> If that is the concern, then its a one line fix to add the missing c

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 12:41 PM, Tom Lane wrote: > Robert Haas writes: >> Updated patch attached. > > The comments need a pass of copy-editing, eg here and here: > >> + * so somebody else could be change the bit just after we look at it.  In >> fact, >                       ^^^ > >>

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Tom Lane
Robert Haas writes: > Updated patch attached. The comments need a pass of copy-editing, eg here and here: > + * so somebody else could be change the bit just after we look at it. In > fact, ^^^ > + * got cleared after we checked it and before we got

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 05:35:11 PM Heikki Linnakangas wrote: > On 07.06.2012 17:18, Andres Freund wrote: > > On Thursday, June 07, 2012 03:50:35 PM Heikki Linnakangas wrote: > >> 3. Move the only field, xl_rem_len, from the continuation record header > >> straight to the xlog page header, elimi

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
Simon Riggs writes: > On 7 June 2012 14:56, Tom Lane wrote: >> Say what?  That's a performance result and proves not a damn thing about >> safety. > Of course not. > Based on the rationale explained in the code comments in the patch, it > seems like a reasonable thing to me now. > The argument

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:23:58 PM Robert Haas wrote: > On Thu, Jun 7, 2012 at 12:19 PM, Andres Freund wrote: > > You could do a visibilitymap_pin outside, but I don't really see the > > point as the page is already locked. There might be some slight benefit > > in doing so in multi_insert bu

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Simon Riggs writes: >> Robert Haas wrote: >>> I know of real customers who would have suffered real data loss >>> had this code been present in the server version they were using. > If that is the concern, then its a one line fix to add the missing clog flush. To where, and what performance imp

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Simon Riggs
On 7 June 2012 14:56, Tom Lane wrote: > Sergey Koposov writes: >> On Tue, 5 Jun 2012, Simon Riggs wrote: > Sounds less good and we'd need reasonable proof it actually did > anything useful without being dangerous. > Doing an initial unlocked test speeds things up another 2.69 fold (o

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 12:19 PM, Andres Freund wrote: > You could do a visibilitymap_pin outside, but I don't really see the point as > the page is already locked. There might be some slight benefit in doing so in > multi_insert but that would be more complicated. And of doubtful benefit. Doesn't

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:20:32 PM Tom Lane wrote: > Andres Freund writes: > > On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: > >> Honza Horak writes: > > On 06/06/2012 04:50 PM, Andres Freund wrote: > >>> I wonder if the whole issue doesn't require libpq to also try multiple > >>> ha

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 11:51 AM, Simon Riggs wrote: > So this is a very large curve ball you're throwing there. This is not exactly unexpected. At least the first two of these items were previous discussed in the context of the XLOG scaling patch, many months ago. It shouldn't come as a surpris

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Tom Lane
Andres Freund writes: > On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: >> Honza Horak writes: > On 06/06/2012 04:50 PM, Andres Freund wrote: >>> I wonder if the whole issue doesn't require libpq to also try multiple >>> hardcoded socket locations. >> I don't really want to go there. > I

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:08:34 PM Robert Haas wrote: > On Thu, Jun 7, 2012 at 11:04 AM, Andres Freund wrote: > > On Thursday, June 07, 2012 04:27:32 PM Robert Haas wrote: > >> On Thu, Jun 7, 2012 at 9:41 AM, Andres Freund > > > > wrote: > >> >> Proposed patch attached. This adds some more

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 11:57 AM, Andres Freund wrote: > On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: >> Honza Horak writes: >> > On 06/06/2012 04:50 PM, Andres Freund wrote: >> >> I wonder if the whole issue doesn't require libpq to also try multiple >> >> hardcoded socket locations. >>

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Heikki Linnakangas
On 07.06.2012 18:51, Simon Riggs wrote: On 7 June 2012 14:50, Heikki Linnakangas wrote: These changes will help the XLogInsert scaling patch ...and as I'm sure you're aware will junk much of the replication code and almost certainly set back the other work that we have brewing for 9.3. So t

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 11:04 AM, Andres Freund wrote: > On Thursday, June 07, 2012 04:27:32 PM Robert Haas wrote: >> On Thu, Jun 7, 2012 at 9:41 AM, Andres Freund > wrote: >> >> Proposed patch attached.  This adds some more comments in various >> >> places, and implements your suggestion of retes

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:02:12 PM Magnus Hagander wrote: > On Thu, Jun 7, 2012 at 5:56 PM, Andres Freund wrote: > > Hi, > > > > On Thursday, June 07, 2012 05:51:07 PM Simon Riggs wrote: > >> On 7 June 2012 14:50, Heikki Linnakangas > >> > >> wrote: > >> > These changes will help the XLogI

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Magnus Hagander
On Thu, Jun 7, 2012 at 5:56 PM, Andres Freund wrote: > Hi, > > On Thursday, June 07, 2012 05:51:07 PM Simon Riggs wrote: >> On 7 June 2012 14:50, Heikki Linnakangas >> >> wrote: >> > These changes will help the XLogInsert scaling patch >> >> ...and as I'm sure you're aware will junk much of the r

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Simon Riggs
On 7 June 2012 14:59, Kevin Grittner wrote: > Tom Lane wrote: > >> there is no guarantee that we'll manage to reach a database state >> that is consistent with data already flushed out to disk during >> the last checkpoint. > > Robert Haas wrote: > >> I know of real customers who would have suff

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 05:55:11 PM Tom Lane wrote: > Honza Horak writes: > > On 06/06/2012 04:50 PM, Andres Freund wrote: > >> I wonder if the whole issue doesn't require libpq to also try multiple > >> hardcoded socket locations. > > > > I guess so. > > I don't really want to go there. Som

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
Hi, On Thursday, June 07, 2012 05:51:07 PM Simon Riggs wrote: > On 7 June 2012 14:50, Heikki Linnakangas > > wrote: > > These changes will help the XLogInsert scaling patch > > ...and as I'm sure you're aware will junk much of the replication code > and almost certainly set back the other work

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Tom Lane
Honza Horak writes: > On 06/06/2012 04:50 PM, Andres Freund wrote: >> I wonder if the whole issue doesn't require libpq to also try multiple >> hardcoded socket locations. > I guess so. I don't really want to go there. Some use cases have been shown in this thread for having a server listen in

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Simon Riggs
On 7 June 2012 14:50, Heikki Linnakangas wrote: > These changes will help the XLogInsert scaling patch ...and as I'm sure you're aware will junk much of the replication code and almost certainly set back the other work that we have brewing for 9.3. So this is a very large curve ball you're throw

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Tom Lane
Robert Haas writes: > On Wed, Jun 6, 2012 at 6:46 PM, Tom Lane wrote: >> If we don't like that, I can think of a couple of other ways to get there, >> but they have their own downsides: >> >> * Instead of trying to detect after-the-fact whether any concurrent >> WAL activity happened during the

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Honza Horak
On 06/06/2012 04:50 PM, Andres Freund wrote: On Wednesday, June 06, 2012 04:38:42 PM Tom Lane wrote: You might think we should design this exactly like the TCP-socket multiple-listen-addresses case, ie just have a config variable containing a list of directory names. The sticking point there is

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Heikki Linnakangas
On 07.06.2012 17:18, Andres Freund wrote: On Thursday, June 07, 2012 03:50:35 PM Heikki Linnakangas wrote: 3. Move the only field, xl_rem_len, from the continuation record header straight to the xlog page header, eliminating XLogContRecord altogether. This makes it easier to calculate in advance

Re: [HACKERS] Could we replace SysV semaphores with latches?

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 10:13 AM, Tom Lane wrote: > I would be more inclined to look into OS-specific primitives such as > futexes on Linux.  (No idea whether those actually would be suitable, > just pointing out that they exist.)  Our semaphore-based API was always > both overcomplicated and under

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 04:27:32 PM Robert Haas wrote: > On Thu, Jun 7, 2012 at 9:41 AM, Andres Freund wrote: > >> Proposed patch attached. This adds some more comments in various > >> places, and implements your suggestion of retesting the visibility-map > >> bit when we detect a possible mi

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Heikki Linnakangas writes: > When I worked on the XLogInsert scaling patch, it became apparent that > some changes to the WAL format would make it a lot easier. So for 9.3, > I'd like to do some refactoring: > 1. Use a 64-bit integer instead of the two-variable log/seg > representation, for id

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 9:41 AM, Andres Freund wrote: >> Well, one, commits are irrelevant; the page ceases to be all-visible >> as soon as the delete happens. > Its not irrelevant for the code as it stands if non-mvcc snapshots were > allowed. Unless I miss something, even disregarding memory orde

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 03:50:35 PM Heikki Linnakangas wrote: > When I worked on the XLogInsert scaling patch, it became apparent that > some changes to the WAL format would make it a lot easier. So for 9.3, > I'd like to do some refactoring: > 1. Use a 64-bit integer instead of the two-variabl

Re: [HACKERS] Could we replace SysV semaphores with latches?

2012-06-07 Thread Tom Lane
Heikki Linnakangas writes: > On 07.06.2012 07:09, Tom Lane wrote: >> It strikes me that we have recently put together an independent but just >> about equivalent waiting mechanism in the form of latches. And not only >> that, but there's already a latch for each process. Could we replace >> our

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Robert Haas
On Wed, Jun 6, 2012 at 6:46 PM, Tom Lane wrote: > I wrote: >> Actually, it looks like there is an extremely simple way to handle this, >> which is to move the call of LogStandbySnapshot (which generates the WAL >> record in question) to before the checkpoint's REDO pointer is set, but >> after we

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Kevin Grittner
Tom Lane wrote: > there is no guarantee that we'll manage to reach a database state > that is consistent with data already flushed out to disk during > the last checkpoint. Robert Haas wrote: > I know of real customers who would have suffered real data loss > had this code been present in t

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Tom Lane
Sergey Koposov writes: > On Tue, 5 Jun 2012, Simon Riggs wrote: Sounds less good and we'd need reasonable proof it actually did anything useful without being dangerous. >>> Doing an initial unlocked test speeds things up another 2.69 fold (on >>> top of 3.55 for your patch) for me, with

[HACKERS] XLog changes for 9.3

2012-06-07 Thread Heikki Linnakangas
When I worked on the XLogInsert scaling patch, it became apparent that some changes to the WAL format would make it a lot easier. So for 9.3, I'd like to do some refactoring: 1. Use a 64-bit integer instead of the two-variable log/seg representation, for identifying a WAL segment. This has no

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 03:20:50 PM Robert Haas wrote: > On Wed, Jun 6, 2012 at 3:07 PM, Andres Freund wrote: > > Hm. For a short while I thought there would be an issue with heap_delete > > and IOS because the deleting transaction can commit without any barriers > > happening on the IOS side.

Re: [HACKERS] "page is not marked all-visible" warning in regression tests

2012-06-07 Thread Robert Haas
On Wed, Jun 6, 2012 at 3:07 PM, Andres Freund wrote: > Hm. For a short while I thought there would be an issue with heap_delete and > IOS because the deleting transaction can commit without any barriers happening > on the IOS side. But that only seems to be possible with non MVCC snapshots > which

Re: [HACKERS] Early hint bit setting

2012-06-07 Thread Merlin Moncure
On Wed, Jun 6, 2012 at 5:41 PM, Jim Nasby wrote: > On 5/30/12 4:42 PM, Ants Aasma wrote: >> >> I was thinking about what is the earliest time where we could set hint >> bits. This would be just after the commit has been made visible. > > > Except that's only true when there are no other transactio

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-06-07 Thread Sergey Koposov
On Tue, 5 Jun 2012, Simon Riggs wrote: Sounds less good and we'd need reasonable proof it actually did anything useful without being dangerous. Doing an initial unlocked test speeds things up another 2.69 fold (on top of 3.55 for your patch) for me, with 1GB of shared buffers.  That seems like

Re: [HACKERS] SP-GIST docs and examples

2012-06-07 Thread Oleg Bartunov
On Thu, 7 Jun 2012, Thom Brown wrote: On 7 June 2012 12:52, Robert Haas wrote: On Thu, Jun 7, 2012 at 5:19 AM, Simon Riggs wrote: I know Oleg has been discussing quad trees for years now, so SP-GIST sounds like a great feature. The docs on SP-GIST simply suggest people read the code, which

Re: [HACKERS] creating objects in pg_catalog

2012-06-07 Thread Robert Haas
On Wed, Jun 6, 2012 at 5:21 PM, Tom Lane wrote: > Robert Haas writes: >> rhaas=# create table pg_catalog.tom (a int); >> ERROR:  permission denied to create "pg_catalog.tom" > >> The offending error check is in heap_create(), and based on what >> you're saying here it seems like we should just ri

Re: [HACKERS] SP-GIST docs and examples

2012-06-07 Thread Thom Brown
On 7 June 2012 12:52, Robert Haas wrote: > On Thu, Jun 7, 2012 at 5:19 AM, Simon Riggs wrote: >> I know Oleg has been discussing quad trees for years now, so SP-GIST >> sounds like a great feature. >> >> The docs on SP-GIST simply suggest people read the code, which is way >> below our normal sta

Re: [HACKERS] SP-GIST docs and examples

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 5:19 AM, Simon Riggs wrote: > I know Oleg has been discussing quad trees for years now, so SP-GIST > sounds like a great feature. > > The docs on SP-GIST simply suggest people read the code, which is way > below our normal standard of documentation. I completely agree. The

Re: [HACKERS] pg_receivexlog and feedback message

2012-06-07 Thread Magnus Hagander
On Thursday, June 7, 2012, Fujii Masao wrote: > On Thu, Jun 7, 2012 at 6:25 PM, Magnus Hagander > wrote: > > On Thursday, June 7, 2012, Fujii Masao wrote: > >> > >> On Thu, Jun 7, 2012 at 5:05 AM, Magnus Hagander > >> wrote: > >> > On Wed, Jun 6, 2012 at 8:26 PM, Fujii Masao > >> > wrote: > >>

Re: [HACKERS] pg_receivexlog and feedback message

2012-06-07 Thread Fujii Masao
On Thu, Jun 7, 2012 at 6:25 PM, Magnus Hagander wrote: > On Thursday, June 7, 2012, Fujii Masao wrote: >> >> On Thu, Jun 7, 2012 at 5:05 AM, Magnus Hagander >> wrote: >> > On Wed, Jun 6, 2012 at 8:26 PM, Fujii Masao >> > wrote: >> >> On Tue, Jun 5, 2012 at 11:44 PM, Magnus Hagander >> >> wrote:

Re: [HACKERS] Interrupting long external library calls

2012-06-07 Thread Florian Pflug
On Jun7, 2012, at 12:08 , Sandro Santilli wrote: > On Thu, Jun 07, 2012 at 12:00:09PM +0200, Florian Pflug wrote: >> On Jun7, 2012, at 10:20 , Sandro Santilli wrote: > >>> In that case I can understand Tom's advice about providing a callback, >>> and then I would only need to perform the "events

Re: [HACKERS] Interrupting long external library calls

2012-06-07 Thread Sandro Santilli
On Thu, Jun 07, 2012 at 12:00:09PM +0200, Florian Pflug wrote: > On Jun7, 2012, at 10:20 , Sandro Santilli wrote: > > In that case I can understand Tom's advice about providing a callback, > > and then I would only need to perform the "events flushing" part of > > from within the callback, and onl

Re: [HACKERS] Interrupting long external library calls

2012-06-07 Thread Florian Pflug
On Jun7, 2012, at 10:20 , Sandro Santilli wrote: > On Sat, May 26, 2012 at 01:24:26PM +0200, Florian Pflug wrote: >> On May26, 2012, at 12:40 , Simon Riggs wrote: >>> On 25 May 2012 17:34, Tom Lane wrote: I assume that the geos::util::Interrupt::request() call sets a flag somewhere that'

Re: [HACKERS] SP-GIST docs and examples

2012-06-07 Thread Magnus Hagander
On Thursday, June 7, 2012, Simon Riggs wrote: > I know Oleg has been discussing quad trees for years now, so SP-GIST > sounds like a great feature. > > The docs on SP-GIST simply suggest people read the code, which is way > below our normal standard of documentation. > It's the same we have for n

Re: [HACKERS] pg_receivexlog and feedback message

2012-06-07 Thread Magnus Hagander
On Thursday, June 7, 2012, Fujii Masao wrote: > On Thu, Jun 7, 2012 at 5:05 AM, Magnus Hagander > > > wrote: > > On Wed, Jun 6, 2012 at 8:26 PM, Fujii Masao > > > > wrote: > >> On Tue, Jun 5, 2012 at 11:44 PM, Magnus Hagander > >> > > wrote: > >>> On Tue, Jun 5, 2012 at 4:42 PM, Fujii Masao >

Re: [HACKERS] \conninfo and SSL

2012-06-07 Thread Magnus Hagander
On Wednesday, June 6, 2012, Alvaro Herrera wrote: > > Excerpts from Robert Haas's message of mié jun 06 14:45:46 -0400 2012: > > On Sun, Jun 3, 2012 at 5:30 AM, Alastair Turner > > > > wrote: > > > A one-line change adds the SSL info on its own line like > > > > > > -- > > > You are connected

[HACKERS] SP-GIST docs and examples

2012-06-07 Thread Simon Riggs
I know Oleg has been discussing quad trees for years now, so SP-GIST sounds like a great feature. The docs on SP-GIST simply suggest people read the code, which is way below our normal standard of documentation. CREATE INDEX contains no examples involving SP-GIST indexes. It seems likely that th

Re: [HACKERS] Time for pgindent run?

2012-06-07 Thread Magnus Hagander
On Thursday, June 7, 2012, Robert Haas wrote: > On Tue, Jun 5, 2012 at 10:25 AM, Bruce Momjian > > > wrote: > > On Tue, Jun 05, 2012 at 10:21:14AM -0400, Tom Lane wrote: > >> Bruce Momjian > writes: > >> > Is everyone ready for me to run pgindent? We are nearing the first > >> > commit-fest (Jun

Re: [HACKERS] Avoiding adjacent checkpoint records

2012-06-07 Thread Simon Riggs
On 6 June 2012 20:08, Tom Lane wrote: > In commit 18fb9d8d21a28caddb72c7ffbdd7b96d52ff9724, Simon modified the > rule for when to skip checkpoints on the grounds that not enough > activity has happened since the last one.  However, that commit left the > comment block about it in a nonsensical st

Re: [HACKERS] Ability to listen on two unix sockets

2012-06-07 Thread Gurjeet Singh
On Wed, Jun 6, 2012 at 1:55 PM, Robert Haas wrote: > On Wed, Jun 6, 2012 at 10:38 AM, Tom Lane wrote: > > Well, that's what I wanted to discuss before Honza starts coding. > > It's not obvious that there are any use-cases for more than two. > > It's also not clear whether there is any value in s

Re: [HACKERS] Interrupting long external library calls

2012-06-07 Thread Sandro Santilli
On Sat, May 26, 2012 at 01:24:26PM +0200, Florian Pflug wrote: > On May26, 2012, at 12:40 , Simon Riggs wrote: > > On 25 May 2012 17:34, Tom Lane wrote: > >> I assume that the geos::util::Interrupt::request() call sets a flag > >> somewhere that's going to be periodically checked in long-running >

Re: [HACKERS] Could we replace SysV semaphores with latches?

2012-06-07 Thread Heikki Linnakangas
On 07.06.2012 07:09, Tom Lane wrote: There has been regular griping in this list about our dependence on SysV shared memory, but not so much about SysV semaphores, even though the latter cause their fair share of issues; as seen for example in buildfarm member spoonbill's recent string of failure