Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Amit Kapila
From: Robert Haas [mailto:robertmh...@gmail.com] Sent: Thursday, August 09, 2012 11:18 PM On Thu, Aug 9, 2012 at 12:43 PM, Heikki Linnakangas wrote: >>> So suppose that the following sequence of events occurs: >> >>> 1. Tuple A on page 1 is updated. The new version, tuple B, is placed on >>> pag

Re: [HACKERS] patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-08-09 Thread Tom Lane
Jameison Martin writes: > [separate topic: pluggable heap manager] > I'm quite interested in pursuing more aggressive compression > strategies, and I'd like to do so in the context of the heap > manager. I'm exploring having a pluggable heap manager implementation > and would be interested in feed

Re: [HACKERS] WIP patch for consolidating misplaced-aggregate checks

2012-08-09 Thread Tom Lane
Here's an updated version taking into account the discussion so far. It's still a net addition of code (about +200 lines according to diffstat), but I think the consolidation of logic is probably worth that. Any further comments? regards, tom lane binHLYugMFK3Z.bin Desc

Re: [HACKERS] patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-08-09 Thread Jim Nasby
On 8/9/12 10:56 AM, Jameison Martin wrote: [separate topic: pluggable heap manager] I'm quite interested in pursuing more aggressive compression strategies, and I'd like to do so in the context of the heap manager. I'm exploring having a pluggable heap manager implementation and would be intere

Re: [HACKERS] proposal - assign result of query to psql variable

2012-08-09 Thread Pavel Stehule
Hello there is new version of this patch * cleaned var list parser * new regress tests * support FETCH_COUNT > 0 Regards Pavel Stehule 2012/8/1 David Fetter : > On Sat, Jul 28, 2012 at 06:11:21PM +0200, Pavel Stehule wrote: >> Hello >> >> 2012/7/27 Tom Lane : >> > Pavel Stehule writes: >> >>

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Robert Haas
On Thu, Aug 9, 2012 at 12:43 PM, Heikki Linnakangas wrote: >> So suppose that the following sequence of events occurs: >> >> 1. Tuple A on page 1 is updated. The new version, tuple B, is placed on >> page 2. >> 2. The table is vacuumed, removing tuple A. >> 3. Page 1 is written durably to disk. >

Re: [HACKERS] WIP patch for consolidating misplaced-aggregate checks

2012-08-09 Thread Alvaro Herrera
Excerpts from Tom Lane's message of jue ago 09 12:40:08 -0400 2012: > Robert Haas writes: > > On Thu, Aug 9, 2012 at 11:56 AM, Tom Lane wrote: > >> If we do go with the %s-for-a-SQL-keyword approach, it would then become > >> tempting to force-fit all of the cases into that style. > > > I don't

Re: [HACKERS] Wiki link for max_connections? (Fwd: Re: [ADMIN] PostgreSQL oom_adj postmaster process to -17)

2012-08-09 Thread Kevin Grittner
Robert Haas wrote: > Kevin Grittner wrote: >> we *really* need to actually get *something* into the docs on >> this, > Sure, I'm not opposed to documenting it. Putting it in the actual > documentation is not a demotion relative to putting a link to it > in the documentation! I didn't figur

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Heikki Linnakangas
On 09.08.2012 19:39, Robert Haas wrote: On Thu, Aug 9, 2012 at 9:09 AM, Heikki Linnakangas wrote: I meant corruption caused by anything, like disk failure, bugs, cosmic rays, etc. The point is that currently the WAL record contains all the information required to reconstruct the old tuple. Wit

Re: [HACKERS] WIP patch for consolidating misplaced-aggregate checks

2012-08-09 Thread Tom Lane
Robert Haas writes: > On Thu, Aug 9, 2012 at 11:56 AM, Tom Lane wrote: >> If we do go with the %s-for-a-SQL-keyword approach, it would then become >> tempting to force-fit all of the cases into that style. > I don't really like this, though. I don't think an error cursor is a > good substitute

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Robert Haas
On Thu, Aug 9, 2012 at 9:09 AM, Heikki Linnakangas wrote: > I meant corruption caused by anything, like disk failure, bugs, cosmic rays, > etc. The point is that currently the WAL record contains all the information > required to reconstruct the old tuple. With a diff method, that's no longer > th

Re: [HACKERS] WIP patch for consolidating misplaced-aggregate checks

2012-08-09 Thread Robert Haas
On Thu, Aug 9, 2012 at 11:56 AM, Tom Lane wrote: > At the moment, the patch faithfully preserves (well, 99% preserves) > the current spellings of the error messages, so that no regression > test entries change. Once all those messages were brought together, > it became painfully obvious that we h

Re: [HACKERS] Re: [COMMITTERS] pgsql: Send new protocol keepalive messages to standby servers.

2012-08-09 Thread Magnus Hagander
On Thu, Aug 9, 2012 at 6:11 PM, Simon Riggs wrote: > On 5 June 2012 23:55, Tom Lane wrote: >> Simon Riggs writes: >>> Can't we have a trial branch where quarantined patches can be placed >>> on trial for inclusion in main release? >> >> [ shrug... ] You're welcome to publish a personal repo som

Re: [HACKERS] Re: [COMMITTERS] pgsql: Send new protocol keepalive messages to standby servers.

2012-08-09 Thread Simon Riggs
On 5 June 2012 23:55, Tom Lane wrote: > Simon Riggs writes: >> Can't we have a trial branch where quarantined patches can be placed >> on trial for inclusion in main release? > > [ shrug... ] You're welcome to publish a personal repo somewhere with > such things. But even if we did that in the

Re: [HACKERS] [PATCH] Make "psql -1 < file.sql" work as with "-f"

2012-08-09 Thread Fabien COELHO
OK, done. I had to revise the original patch pretty heavily before committing; the original patch assumed that it was OK to make psql -1 Yep. I did that with a "smallest" and "simplest" change in mind, and beside when doing "psql -1 < file", you're hardly going to do anything else anyway, s

Re: [HACKERS] Wiki link for max_connections? (Fwd: Re: [ADMIN] PostgreSQL oom_adj postmaster process to -17)

2012-08-09 Thread Robert Haas
On Thu, Aug 9, 2012 at 11:27 AM, Kevin Grittner wrote: > Robert Haas wrote: >> Kevin Grittner wrote: >>> I see that we currently have five links to wiki.postgresql.org in >>> release notes and four more in the rest of the docs. Are people >>> OK with adding this link to the docs on max_connecti

[HACKERS] WIP patch for consolidating misplaced-aggregate checks

2012-08-09 Thread Tom Lane
I wrote: > I find it fairly annoying though that parseCheckAggregates (and likewise > parseCheckWindowFuncs) have to dig through previously parsed query trees > to look for misplaced aggregates; so adding even more of that is grating > on me. It would be a lot cleaner if transformAggregateCall and

Re: [HACKERS] patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-08-09 Thread Jameison Martin
Simon, Tom is correct, the patch doesn't change the existing row format contract or the format of the null bitmap. The change only affects how new rows are written out. And it uses the same supported format that has always been there (which is why alter table add col null works the way it does).

Re: [HACKERS] SP-GiST for ranges based on 2d-mapping and quad-tree

2012-08-09 Thread Alexander Korotkov
In this revision of patch I tried to handle conditions more generally using variables minLower, maxLower, minUpper, maxUpper, inclusive and strictEmpty. However some strategies still contain additional logic. What is our conclusion about saving previous choice for RANGESTRAT_ADJACENT strategy? ---

Re: [HACKERS] Statistics and selectivity estimation for ranges

2012-08-09 Thread Alexander Korotkov
New revision of patch with two fixes: 1) Check if histogram bin width is zero in get_position. 2) Check statsTuple is valid tuple in rangecontsel. -- With best regards, Alexander Korotkov. range_stat-0.3.patch.gz Description: GNU Zip compressed data -- Sent via pgsql-hackers mailing list (

Re: [HACKERS] Wiki link for max_connections? (Fwd: Re: [ADMIN] PostgreSQL oom_adj postmaster process to -17)

2012-08-09 Thread Kevin Grittner
Robert Haas wrote: > Kevin Grittner wrote: >> I see that we currently have five links to wiki.postgresql.org in >> release notes and four more in the rest of the docs. Are people >> OK with adding this link to the docs on max_connections? (Feel >> free to improve it before answering if you have

Re: [HACKERS] Wiki link for max_connections? (Fwd: Re: [ADMIN] PostgreSQL oom_adj postmaster process to -17)

2012-08-09 Thread Robert Haas
On Thu, Aug 9, 2012 at 10:22 AM, Kevin Grittner wrote: > I see that we currently have five links to wiki.postgresql.org in > release notes and four more in the rest of the docs. Are people OK > with adding this link to the docs on max_connections? (Feel free to > improve it before answering if y

Re: [HACKERS] patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-08-09 Thread Simon Riggs
On 9 August 2012 15:27, Tom Lane wrote: > Simon Riggs writes: >> On 17 April 2012 17:22, Jameison Martin wrote: >>> The following patch truncates trailing null attributes from heap rows to >>> reduce the size of the row bitmap. > >> This is an interesting patch, but its has had various comments

[HACKERS] Re: Wiki link for max_connections? (Fwd: Re: [ADMIN] PostgreSQL oom_adj postmaster process to -17)

2012-08-09 Thread Magnus Hagander
On Thu, Aug 9, 2012 at 4:22 PM, Kevin Grittner wrote: > [Forwarding to the -hackers list. Please respond there.] > > Craig Ringer wrote: >> On 08/09/2012 04:24 AM, Kevin Grittner wrote: >>> http://wiki.postgresql.org/wiki/Number_Of_Database_Connections >>> >> Can we please please PLEASE link to

[HACKERS] Regarding Patch [Rearrange and cross-reference notes on sequences and rollback exemption]

2012-08-09 Thread Amit Kapila
Hi Craig Ringer, It seems you have uploaded your patch https://commitfest.postgresql.org/action/patch_view?id=901 to previous CF (2012-06). Shouldn't it be in CF (2012-09)? With Regards, Amit Kapila.

Re: [HACKERS] patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-08-09 Thread Tom Lane
Simon Riggs writes: > On 17 April 2012 17:22, Jameison Martin wrote: >> The following patch truncates trailing null attributes from heap rows to >> reduce the size of the row bitmap. > This is an interesting patch, but its has had various comments made about it. > When I look at this I see that

Re: [HACKERS] Prevent restored WAL files from being archived again Re: Unnecessary WAL archiving after failover

2012-08-09 Thread Simon Riggs
On 9 August 2012 00:08, Simon Riggs wrote: > Will look at the refactored version tomorrow. Rather than implement the minimal patch onto HEAD, I've written a refactored patch. [attached] This fixes the bugs discussed here, though allows for multiple archive statuses, rather than just two, as wel

[HACKERS] Wiki link for max_connections? (Fwd: Re: [ADMIN] PostgreSQL oom_adj postmaster process to -17)

2012-08-09 Thread Kevin Grittner
[Forwarding to the -hackers list. Please respond there.] Craig Ringer wrote: > On 08/09/2012 04:24 AM, Kevin Grittner wrote: >> http://wiki.postgresql.org/wiki/Number_Of_Database_Connections >> > Can we please please PLEASE link to that as a comment above > max_connections? > > Last time this c

Re: [HACKERS] Bugs in superuser_reserved_connections and max_wal_senders vs max_connections

2012-08-09 Thread Tom Lane
Magnus Hagander writes: > The check in PostmasterMain(): > if (ReservedBackends >= MaxBackends) > { > write_stderr("%s: superuser_reserved_connections must be less > than > max_connections\n", progname); > ExitPostmaster(1); > } > should probably che

Re: [HACKERS] Inserting heap tuples in bulk in COPY

2012-08-09 Thread Robert Haas
On Thu, Aug 9, 2012 at 2:59 AM, Jesper Krogh wrote: > If it is an implementation artifact or an result of this > approach I dont know. But currently, when the GIN fastupdate > code finally decides to "flush" the buffer, it is going to stall all > other processes doing updates while doing it. If yo

Re: [HACKERS] -Wformat-zero-length

2012-08-09 Thread Bruce Momjian
On Thu, Aug 9, 2012 at 09:20:23AM -0400, Robert Haas wrote: > On Wed, Aug 8, 2012 at 7:04 PM, Bruce Momjian wrote: > >> The point I think Robert was trying to make is that we need to cut down > >> not only the complexity of running pg_upgrade, but the number of failure > >> modes. At least that'

Re: [HACKERS] -Wformat-zero-length

2012-08-09 Thread Robert Haas
On Wed, Aug 8, 2012 at 7:04 PM, Bruce Momjian wrote: >> The point I think Robert was trying to make is that we need to cut down >> not only the complexity of running pg_upgrade, but the number of failure >> modes. At least that's how I'd define improvement here. > > Agreed. Even with these chang

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Amit Kapila
From: Simon Riggs [mailto:si...@2ndquadrant.com] Sent: Thursday, August 09, 2012 5:29 PM On 9 August 2012 12:17, Amit Kapila wrote: >> This optimization is to reduce the amount of WAL and definitely adding >> anything extra will have some impact. > Of course. The question is "How much impact?".

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Heikki Linnakangas
On 09.08.2012 15:56, Amit Kapila wrote: From: Heikki Linnakangas [mailto:heikki.linnakan...@enterprisedb.com] Sent: Thursday, August 09, 2012 4:59 PM On 09.08.2012 14:11, Simon Riggs wrote: But then again, full-page writes cover that too. There will be a full-page image of the old block in the

Re: [HACKERS] [PATCH] Make "psql -1 < file.sql" work as with "-f"

2012-08-09 Thread Robert Haas
On Wed, Aug 8, 2012 at 6:50 PM, David Fetter wrote: >> I'm wondering if perhaps -- in addition to what you've done here -- we >> should make "psql -1" error out if reading from a terminal. > > +1 for this. OK, done. I had to revise the original patch pretty heavily before committing; the origina

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Amit Kapila
From: Heikki Linnakangas [mailto:heikki.linnakan...@enterprisedb.com] Sent: Thursday, August 09, 2012 4:59 PM On 09.08.2012 14:11, Simon Riggs wrote: >>> But then again, full-page writes cover that too. There >>> will be a full-page image of the old block in the WAL anyway. > >> Right, but we're

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Simon Riggs
On 9 August 2012 12:17, Amit Kapila wrote: > This optimization is to reduce the amount of WAL and definitely adding > anything extra will have some impact. Of course. The question is "How much impact?". Each tweak has progressively less and less gain. This isn't a binary choice. Squeezing the l

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Heikki Linnakangas
On 09.08.2012 14:11, Simon Riggs wrote: Given the marginal gain because of a low percentage of cross-block updates, I'm not keen. Low percentage because HOT tries hard to keep things on same block - even for non-HOT updates (which is the case, even though it sounds weird). That depends entirely

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Amit Kapila
From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Simon Riggs Sent: Thursday, August 09, 2012 2:49 PM On 9 August 2012 09:49, Amit Kapila wrote: >>> I'd suggest we do this only when the saving is large enough for >>> benefit, rather than do this eve

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Simon Riggs
On 9 August 2012 11:30, Heikki Linnakangas wrote: > On 09.08.2012 12:18, Simon Riggs wrote: >> >> On 9 August 2012 09:49, Amit Kapila wrote: >> >>>WAL reduction is done for the case even when old and new are on >>> different >>> data blocks as well. >> >> >> That makes me feel nervous. I doub

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Heikki Linnakangas
On 09.08.2012 12:18, Simon Riggs wrote: On 9 August 2012 09:49, Amit Kapila wrote: WAL reduction is done for the case even when old and new are on different data blocks as well. That makes me feel nervous. I doubt the marginal gain is worth it. Most updates don't cross blocks. That was

Re: [HACKERS] patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-08-09 Thread Simon Riggs
On 17 April 2012 17:22, Jameison Martin wrote: > The following patch truncates trailing null attributes from heap rows to > reduce the size of the row bitmap. > The intuition for this change is that ALTER TABLE t ADD COLUMN c type NULL > is a metadata only change. Postgres works fine when a row'

[HACKERS] Bugs in superuser_reserved_connections and max_wal_senders vs max_connections

2012-08-09 Thread Magnus Hagander
Both superuser_reserved_connections and max_wal_senders can be set to a value that's higher than max_connections, which is quite pointless, and annoying :) The docs for superuser_reserved_connections say "The value must be less than the value of max_connections.", but this is never enforced. Well,

Re: [HACKERS] Re: [COMMITTERS] pgsql: Fix mapping of PostgreSQL encodings to Python encodings.

2012-08-09 Thread Jan Urbański
On 06/08/12 13:59, Heikki Linnakangas wrote: On 20.07.2012 10:13, Jan Urbański wrote: On 20/07/12 08:59, Jan Urbański wrote: On 18/07/12 17:17, Heikki Linnakangas wrote: On 14.07.2012 17:50, Jan Urbański wrote: If pg_do_encoding_conversion() throws an error, you don't get a chance to call Py_

Re: [HACKERS] Skip checkpoint on promoting from streaming replication

2012-08-09 Thread Simon Riggs
On 22 June 2012 05:03, Kyotaro HORIGUCHI wrote: >I hope this is promising. I've reviewed this and thought about it over some time. At first I was unhappy that you'd removed the restriction that timelines only change on a shutdown checkpoint. But the reality is timelines change at any point

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Simon Riggs
On 9 August 2012 09:49, Amit Kapila wrote: >> I'd suggest we do this only when the saving is large enough for >> benefit, rather than do this every time. > Do you mean to say that when length of updated values of tuple is less > than some threshold(1/3 or 2/3, etc..) value of > total length?

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Amit Kapila
From: Simon Riggs [mailto:si...@2ndquadrant.com] Sent: Thursday, August 09, 2012 12:36 PM On 3 August 2012 12:46, Amit kapila wrote: >> Frame the new tuple from old tuple and WAL record: > Sounds good. Thanks. > I'd suggest we do this only when the saving is large enough for > benefit, rathe

Re: [HACKERS] [WIP] Performance Improvement by reducing WAL for Update Operation

2012-08-09 Thread Simon Riggs
On 3 August 2012 12:46, Amit kapila wrote: > Frame the new tuple from old tuple and WAL record: Sounds good. I'd suggest we do this only when the saving is large enough for benefit, rather than do this every time. You don't mention whether or not the old and the new tuple are on the same data

Re: [HACKERS] Inserting heap tuples in bulk in COPY

2012-08-09 Thread Jesper Krogh
On 08/08/12 21:34, Robert Haas wrote: I think we need to implement buffering both to end of statement or end of transaction, not just one or the other. Another (not necessarily better) idea is to use a buffer that's part of the index, like the GIN fastupdate stuff, so that there's no particular