Re: [HACKERS] [PATCH] binary heap implementation

2012-11-14 Thread Andres Freund
On 2012-11-14 18:41:12 +0530, Abhijit Menon-Sen wrote: There are two or three places in the Postgres source that implement heap sort (e.g. tuplesort.c, nodeMergeAppend.c), and it's also needed by the BDR code. It seemed reasonable to factor out the functionality. This replaces the

[HACKERS] S_ISLNK

2012-11-14 Thread Nicholas White
Hi - I'm cross-compiling the master branch (cygwin/mingw) and have found a reference to S_ISLNK that isn't guarded by #ifndef WIN32 like the ones in basebackup.c are. Could you merge the attached fix? Thanks - Nick exec-symlink-ifdef.patch Description: Binary data -- Sent via pgsql-hackers

Re: [HACKERS] S_ISLNK

2012-11-14 Thread Tom Lane
Nicholas White n.j.wh...@gmail.com writes: Hi - I'm cross-compiling the master branch (cygwin/mingw) and have found a reference to S_ISLNK that isn't guarded by #ifndef WIN32 like the ones in basebackup.c are. That whole function is guarded by HAVE_READLINK, so I'm not seeing the problem (and

Re: [HACKERS] Why does delete from table where not exists (select 1 from ... LIMIT 1) perform badly?

2012-11-14 Thread Tom Lane
Palle Girgensohn gir...@pingpong.net writes: How come the planner treats the delete from table where not extists(select 1 from table2 where ... LIMIT 1) so differently, and usually badly, when the LIMIT 1 is there. Because it can't optimize it into an antijoin. In older version of

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-14 Thread Bruce Momjian
On Wed, Nov 14, 2012 at 06:11:27AM +0200, Ants Aasma wrote: On Wed, Nov 14, 2012 at 2:03 AM, Bruce Momjian br...@momjian.us wrote: At 64k I see pg_upgrade taking 12% of the duration time, if I subtract out the dump/restore times. My percentage numbers only included CPU time and I used SSD

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Fujii Masao
On Wed, Nov 14, 2012 at 8:30 PM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I wrote: From: Tom Lane [mailto:t...@sss.pgh.pa.us] I wrote: Etsuro Fujita fujita.ets...@lab.ntt.co.jp writes: I have a question. I think it would be also better to extend the syntax for the SQL COPY

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL

2012-11-14 Thread Amit kapila
On Tuesday, November 13, 2012 9:29 AM Amit kapila wrote: On Monday, November 12, 2012 12:07 PM Greg Smith wrote: On 11/9/12 11:59 PM, Amit kapila wrote: 1) Change to add scanning a .conf directory in the default configuration using include-dir. This is a quick fix. I predict most of the

Re: [HACKERS] [PATCH] Patch to compute Max LSN of Data Pages

2012-11-14 Thread Fujii Masao
On Wed, Nov 14, 2012 at 5:53 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 13, 2012 at 11:46 AM, Fujii Masao masao.fu...@gmail.com wrote: Without this utility, it's difficult to calculate the maximum LSN of data page, so basically we needed to take a backup when starting the

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-14 Thread Andrew Dunstan
On 11/14/2012 10:08 AM, Bruce Momjian wrote: On Wed, Nov 14, 2012 at 06:11:27AM +0200, Ants Aasma wrote: I agree that parallel restore for schemas is a hard problem. But I didn't mean parallelism within the restore, I meant that we could start both postmasters and pipe the output from dump

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Simon Riggs
On 14 November 2012 15:09, Fujii Masao masao.fu...@gmail.com wrote: Here, progname for COPY IN is the user-supplied program that takes filename as its argument and that writes on standard output. What about further extending the COPY IN syntax to the following? COPY table_name FROM

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-14 Thread Bruce Momjian
On Wed, Nov 14, 2012 at 10:25:24AM -0500, Andrew Dunstan wrote: On 11/14/2012 10:08 AM, Bruce Momjian wrote: On Wed, Nov 14, 2012 at 06:11:27AM +0200, Ants Aasma wrote: I agree that parallel restore for schemas is a hard problem. But I didn't mean parallelism within the restore, I meant

Re: [HACKERS] [PATCH] Patch to compute Max LSN of Data Pages

2012-11-14 Thread Fujii Masao
On Wed, Nov 14, 2012 at 5:35 PM, Amit Kapila amit.kap...@huawei.com wrote: On Tuesday, November 13, 2012 10:17 PM Fujii Masao wrote: On Tue, Nov 13, 2012 at 1:23 PM, Amit kapila amit.kap...@huawei.com wrote: On Monday, November 12, 2012 9:56 PM Alvaro Herrera wrote: Robert Haas escribió:

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Fujii Masao
On Thu, Nov 15, 2012 at 12:31 AM, Simon Riggs si...@2ndquadrant.com wrote: On 14 November 2012 15:09, Fujii Masao masao.fu...@gmail.com wrote: Here, progname for COPY IN is the user-supplied program that takes filename as its argument and that writes on standard output. What about further

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Alvaro Herrera
Simon Riggs escribió: On 14 November 2012 15:09, Fujii Masao masao.fu...@gmail.com wrote: Here, progname for COPY IN is the user-supplied program that takes filename as its argument and that writes on standard output. What about further extending the COPY IN syntax to the following?

Re: [HACKERS] [PATCH] Patch to compute Max LSN of Data Pages

2012-11-14 Thread Amit Kapila
On Wednesday, November 14, 2012 9:12 PM Fujii Masao wrote: On Wed, Nov 14, 2012 at 5:35 PM, Amit Kapila amit.kap...@huawei.com wrote: On Tuesday, November 13, 2012 10:17 PM Fujii Masao wrote: On Tue, Nov 13, 2012 at 1:23 PM, Amit kapila amit.kap...@huawei.com wrote: On Monday, November

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Tom Lane
Alvaro Herrera alvhe...@2ndquadrant.com writes: Simon Riggs escribió: On 14 November 2012 15:09, Fujii Masao masao.fu...@gmail.com wrote: Here, progname for COPY IN is the user-supplied program that takes filename as its argument and that writes on standard output. I think we should be

Re: [HACKERS] foreign key locks

2012-11-14 Thread Alvaro Herrera
Alvaro Herrera wrote: * In heap_lock_tuple's XMAX_IS_MULTI case [snip] why is it membermode mode and not membermode = mode? Uh, that's a bug. Fixed. As noticed in the comment above that snippet, there was a deadlock possible here. Maybe I should add a test to ensure this

Re: [HACKERS] S_ISLNK

2012-11-14 Thread Nicholas White
Ah - OK. It turns out I'd run ./configure in the postgresql directory before running it in my build directory, so I had two (different) pg_config.hs! The below builds for me from a clean source tree now. Thanks - Nick ./path/to/build/configure CC=x86_64-w64-mingw32-gcc

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Andrew Dunstan
On 11/14/2012 11:20 AM, Tom Lane wrote: I disagree with Simon's objection also, because neither reading from nor writing to an external program is likely to fit the model of reading/updating a table very well. For instance, there's no good reason to suppose that reading twice will give the

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Simon Riggs
On 14 November 2012 16:20, Tom Lane t...@sss.pgh.pa.us wrote: Alvaro Herrera alvhe...@2ndquadrant.com writes: Simon Riggs escribió: On 14 November 2012 15:09, Fujii Masao masao.fu...@gmail.com wrote: Here, progname for COPY IN is the user-supplied program that takes filename as its argument

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Simon Riggs
On 14 November 2012 15:09, Fujii Masao masao.fu...@gmail.com wrote: What about further extending the COPY IN syntax to the following? COPY table_name FROM 'progname [ option, ... ]' WITH ... I'd just like to execute COPY vmstat_table FROM 'vmstat' WITH ... If we go ahead with

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Yeah, I agree, although the syntax looks a bit unclean. Oh, I had not looked at the syntax closely. I agree, that basically sucks: it's overcomplicated and under-featured, because you can't control the actual program command line very conveniently.

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Robert Haas
On Tue, Nov 13, 2012 at 4:48 PM, Tom Lane t...@sss.pgh.pa.us wrote: What happens when you get an I/O failure on the checksum fork? Assuming you're using 8K pages there, that would mean you can no longer verify the integrity of between one and four thousand pages of data. True... but you'll

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 14 November 2012 15:09, Fujii Masao masao.fu...@gmail.com wrote: I'd just like to execute COPY vmstat_table FROM 'vmstat' WITH ... If we go ahead with this, I think it needs additional keyword to indicate that we will execute the file rather than

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Andrew Dunstan
On 11/14/2012 11:39 AM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Yeah, I agree, although the syntax looks a bit unclean. Oh, I had not looked at the syntax closely. I agree, that basically sucks: it's overcomplicated and under-featured, because you can't control the actual

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: On 11/14/2012 11:39 AM, Tom Lane wrote: What happened to the previous proposal of treating the COPY target as a pipe specification, ie I'd like to be able to filter STDIN if possible - with this syntax how is COPY going to know to hook up STDIN to

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Andrew Dunstan
On 11/14/2012 11:56 AM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: On 11/14/2012 11:39 AM, Tom Lane wrote: What happened to the previous proposal of treating the COPY target as a pipe specification, ie I'd like to be able to filter STDIN if possible - with this syntax how is

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Bruce Momjian
On Mon, Nov 12, 2012 at 04:42:57PM -0800, Josh Berkus wrote: Jeff, OK, so here's my proposal for a first patch (changes from Simon's patch): * Add a flag to the postgres executable indicating that it should use checksums on everything. This would only be valid if bootstrap mode is

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-14 Thread Jeff Janes
On Thu, Nov 8, 2012 at 9:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Jeff Janes jeff.ja...@gmail.com writes: Are sure the server you are dumping out of is head? I experimented a bit with dumping/restoring 16000 tables matching Bruce's test case (ie, one serial column apiece). The pg_dump

Re: [HACKERS] [PATCH] Patch to compute Max LSN of Data Pages

2012-11-14 Thread Fujii Masao
On Thu, Nov 15, 2012 at 12:55 AM, Amit Kapila amit.kap...@huawei.com wrote: Now user can use this utility to decide if new-standby has max LSN greater than max LSN of new-master he needs to use fullback-up on new-standby. Is my understanding right? No. The maximum LSN of data pages in

Re: [HACKERS] WIP checksums patch

2012-11-14 Thread Bruce Momjian
On Mon, Nov 12, 2012 at 12:39:17AM -0500, Greg Smith wrote: On 11/9/12 6:14 PM, Jeff Davis wrote: On Mon, 2012-11-05 at 12:19 -0500, Robert Haas wrote: Yeah. I definitely think that we could shed an enormous amount of complexity by deciding that this is, for now, an option that can only be

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Peter Eisentraut
On 11/11/12 6:59 PM, Andrew Dunstan wrote: I haven't followed this too closely, but I did wonder several days ago why this wasn't being made an initdb-time decision. One problem I see with this is that it would make regression testing much more cumbersome. Basically, to do a proper job, you'd

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Alvaro Herrera
Peter Eisentraut escribió: On 11/11/12 6:59 PM, Andrew Dunstan wrote: I haven't followed this too closely, but I did wonder several days ago why this wasn't being made an initdb-time decision. One problem I see with this is that it would make regression testing much more cumbersome.

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Peter Eisentraut
On 11/14/12 11:50 AM, Andrew Dunstan wrote: COPY table FROM 'some command line |'; COPY table TO '| some command line'; I'd like to be able to filter STDIN if possible - with this syntax how is COPY going to know to hook up STDIN to the program? Why don't you filter the data

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Andrew Dunstan
On 11/14/2012 02:01 PM, Alvaro Herrera wrote: Peter Eisentraut escribió: On 11/11/12 6:59 PM, Andrew Dunstan wrote: I haven't followed this too closely, but I did wonder several days ago why this wasn't being made an initdb-time decision. One problem I see with this is that it would make

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-14 Thread Atri Sharma
On Wed, Nov 7, 2012 at 9:47 PM, Merlin Moncure mmonc...@gmail.com wrote: On Wed, Nov 7, 2012 at 6:01 AM, Amit Kapila amit.kap...@huawei.com wrote: Following the sig is a first cut at a patch (written by Atri) that attempts to mitigate hint bit i/o penalty when many pages worth of

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Andrew Dunstan
On 11/14/2012 02:05 PM, Peter Eisentraut wrote: On 11/14/12 11:50 AM, Andrew Dunstan wrote: COPY table FROM 'some command line |'; COPY table TO '| some command line'; I'd like to be able to filter STDIN if possible - with this syntax how is COPY going to know to hook up STDIN to

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: On 11/14/2012 02:05 PM, Peter Eisentraut wrote: Why don't you filter the data before it gets to stdin? Some program is feeding the data to stdin on the client side. Why doesn't that do the filtering? I don't see a large advantage in having the data

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-14 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes: On Thu, Nov 8, 2012 at 9:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: There are at least three ways we could whack that mole: ... * Keep a separate list (or data structure of your choice) so that relcache entries created in the current xact could be found

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-14 Thread Alvaro Herrera
Tom Lane escribió: Jeff Janes jeff.ja...@gmail.com writes: The next quadratic behavior is in init_sequence. Yeah, that's another place that is using a linear list that perhaps should be a hashtable. OTOH, probably most sessions don't touch enough different sequences for that to be a win.

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes: Regarding checksums, I can add an option for the initdb that the buildfarm script runs. We already run different tests for different encodings. Of course, constant expanding like this won't scale, so we need to pick the options we want to exrecise

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Andrew Dunstan
On 11/14/2012 02:37 PM, Tom Lane wrote: What I'm imagining is a very very simple addition to COPY that just allows it to execute popen() instead of fopen() to read or write the data source/sink. What you suggest would require hundreds of lines and create many opportunities for new bugs.

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Andrew Dunstan
On 11/14/2012 03:06 PM, Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Regarding checksums, I can add an option for the initdb that the buildfarm script runs. We already run different tests for different encodings. Of course, constant expanding like this won't scale, so we need to

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-11-14 Thread Atri Sharma
On Thu, Nov 15, 2012 at 12:42 AM, Atri Sharma atri.j...@gmail.com wrote: On Wed, Nov 7, 2012 at 9:47 PM, Merlin Moncure mmonc...@gmail.com wrote: On Wed, Nov 7, 2012 at 6:01 AM, Amit Kapila amit.kap...@huawei.com wrote: Following the sig is a first cut at a patch (written by Atri) that

Re: [HACKERS] Doc patch, further describe and-mask nature of the permission system

2012-11-14 Thread Karl O. Pinc
On 11/13/2012 08:50:55 PM, Peter Eisentraut wrote: On Sat, 2012-09-29 at 01:16 -0500, Karl O. Pinc wrote: This patch makes some sweeping statements. Unfortunately, they are wrong. I will see if anything can be salvaged. Regards, Karl k...@meme.com Free Software: You don't pay back, you

[HACKERS] [PATCH] Fix INT_MIN % -1 overflow in int8mod().

2012-11-14 Thread Xi Wang
Return 0 for INT_MIN % -1 (64-bit) instead of throwing an exception. This patch complements commit f9ac414c that fixed int4mod(). --- src/backend/utils/adt/int8.c |4 src/include/c.h |7 +++ 2 files changed, 11 insertions(+) diff --git a/src/backend/utils/adt/int8.c

[HACKERS] Doc patch making firm recommendation for setting the value of commit_delay

2012-11-14 Thread Peter Geoghegan
Some of you will be aware that I've tried to formulate a good general recommendation for setting the value of commit_delay, in light of the improvements made in 9.3. I previously asked for help finding a methodology for optimising commit_delay [1]. The documentation related to commit_delay still

Re: [HACKERS] [PATCH] Fix INT_MIN % -1 overflow in int8mod().

2012-11-14 Thread Tom Lane
Xi Wang xi.w...@gmail.com writes: Return 0 for INT_MIN % -1 (64-bit) instead of throwing an exception. This patch complements commit f9ac414c that fixed int4mod(). Meh. I didn't care for the explicit dependency on INT_MIN in the previous patch, and I like introducing INT64_MIN even less. I

Re: [HACKERS] [PATCH] Fix INT_MIN % -1 overflow in int8mod().

2012-11-14 Thread Xi Wang
On 11/14/12 4:46 PM, Tom Lane wrote: Meh. I didn't care for the explicit dependency on INT_MIN in the previous patch, and I like introducing INT64_MIN even less. I think we should be able to reduce the test to just if (arg2 == -1) return 0; since zero is the correct

Re: [HACKERS] [PATCH] Fix INT_MIN % -1 overflow in int8mod().

2012-11-14 Thread Tom Lane
Xi Wang xi.w...@gmail.com writes: I agree. Will send a v2. Thanks. :) No need, I'm already patching it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

[HACKERS] [PATCH 1/2] Fix INT_MIN % -1 overflow in int8mod().

2012-11-14 Thread Xi Wang
Return 0 for x % -1 instead of throwing an exception (e.g., when x is INT_MIN). Suggested by Tom Lane. --- src/backend/utils/adt/int8.c |4 1 file changed, 4 insertions(+) diff --git a/src/backend/utils/adt/int8.c b/src/backend/utils/adt/int8.c index 0e59956..a30ab36 100644 ---

[HACKERS] [PATCH 2/2] Clean up INT_MIN % -1 overflow in int4mod().

2012-11-14 Thread Xi Wang
Since x % -1 returns 0 for any x, we don't need the check x == INT_MIN. Suggested by Tom Lane. --- src/backend/utils/adt/int.c |2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/backend/utils/adt/int.c b/src/backend/utils/adt/int.c index 4be3901..3e423fe 100644 ---

Re: [HACKERS] [PATCH] Fix INT_MIN % -1 overflow in int8mod().

2012-11-14 Thread Xi Wang
On 11/14/12 5:03 PM, Tom Lane wrote: No need, I'm already patching it. Oops. Sorry. Ignore my patches. - xi -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP checksums patch

2012-11-14 Thread Merlin Moncure
On Tue, Oct 9, 2012 at 1:51 AM, Amit Kapila amit.kap...@huawei.com wrote: On Monday, October 01, 2012 11:11 PM Jeff Davis wrote: On Mon, 2012-10-01 at 18:14 +0100, Simon Riggs wrote: You are missing large parts of the previous thread, giving various opinions on what the UI should look like

Re: [HACKERS] Further pg_upgrade analysis for many tables

2012-11-14 Thread Bruce Momjian
Patch applied to git head. Thanks Ants Aasma for the analysis that lead to the patch. --- On Tue, Nov 13, 2012 at 07:03:51PM -0500, Bruce Momjian wrote: On Tue, Nov 13, 2012 at 05:44:54AM +0200, Ants Aasma wrote: On

Re: [HACKERS] My first patch! (to \df output)

2012-11-14 Thread Robert Haas
On Fri, Nov 9, 2012 at 3:22 PM, Jon Erdman postgre...@thewickedtribe.net wrote: Oops! Here it is in the proper diff format. I didn't have my env set up correctly :( Better add it here so it doesn't get lost: https://commitfest.postgresql.org/action/commitfest_view/open -- Robert Haas

Re: [HACKERS] Add contrib module functions to docs' function index

2012-11-14 Thread Robert Haas
On Tue, Nov 13, 2012 at 7:10 PM, Craig Ringer cr...@2ndquadrant.com wrote: I'm talking about making sure that contrib module functions (and settings) appear in the documentation index ( http://www.postgresql.org/docs/current/static/bookindex.html) so it's easy to find a function by name

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Jeff Davis
Hmm... what if we took this a step further and actually stored the checksums in a separate relation fork? That would make it pretty simple to support enabling/disabling checksums for particular relations. It would also allow us to have a wider checksum, like 32 or 64 bits rather than 16.

Re: [HACKERS] Pg_upgrade speed for many tables

2012-11-14 Thread Bruce Momjian
On Mon, Nov 12, 2012 at 10:29:39AM -0800, Jeff Janes wrote: On Mon, Nov 5, 2012 at 12:08 PM, Bruce Momjian br...@momjian.us wrote: Magnus reported that a customer with a million tables was finding pg_upgrade slow. I had never considered many table to be a problem, but decided to test it.

[HACKERS] logical changeset generation v3

2012-11-14 Thread Andres Freund
Hi, In response to this you will soon find the 14 patches that currently implement $subject. I'll go over each one after showing off for a bit: Start postgres: Start postgres instance (with pg_hba.conf allowing replication cons): $ postgres -D ~/tmp/pgdev-lcr \ -c wal_level=logical \

[HACKERS] [PATCH 05/14] Add a new relmapper.c function RelationMapFilenodeToOid that acts as a reverse of RelationMapOidToFilenode

2012-11-14 Thread Andres Freund
--- src/backend/utils/cache/relmapper.c | 53 + src/include/catalog/indexing.h | 4 +-- src/include/utils/relmapper.h | 2 ++ 3 files changed, 57 insertions(+), 2 deletions(-) diff --git a/src/backend/utils/cache/relmapper.c

[HACKERS] [PATCH 01/14] Add minimal binary heap implementation

2012-11-14 Thread Andres Freund
Will be replaces by the binaryheap.[ch] from Abhijit once its been reviewed. --- src/backend/lib/Makefile | 3 +- src/backend/lib/simpleheap.c | 255 +++ src/include/lib/simpleheap.h | 91 +++ 3 files changed, 348 insertions(+), 1

[HACKERS] [PATCH 06/14] Add a new function pg_relation_by_filenode to lookup up a relation given the tablespace and the filenode OIDs

2012-11-14 Thread Andres Freund
This requires the previously added RELFILENODE syscache and the added RelationMapFilenodeToOid function added in previous commits. --- doc/src/sgml/func.sgml | 23 +++- src/backend/utils/adt/dbsize.c | 79 ++ src/include/catalog/pg_proc.h

[HACKERS] [PATCH 04/14] Add a new RELFILENODE syscache to fetch a pg_class entry via (reltablespace, relfilenode)

2012-11-14 Thread Andres Freund
This cache is somewhat problematic because formally indexes used by syscaches needs to be unique, this one is not. This is just because of 0/InvalidOids stored in pg_class.relfilenode for nailed/shared catalog relations. The syscache will never be queried for InvalidOid relfilenodes however so

[HACKERS] [PATCH 08/14] Store the number of subtransactions in xl_running_xacts separately from toplevel xids

2012-11-14 Thread Andres Freund
To avoid complicating logic we store both, the toplevel and the subxids, in -xip, first -xcnt toplevel ones, and then -subxcnt subxids. Also skip logging any subxids if the snapshot is suboverflowed, they aren't useful in that case anyway. This allows to make some operations cheaper and it

[HACKERS] [PATCH 10/14] Allow walsender's to connect to a specific database

2012-11-14 Thread Andres Freund
Currently the decision whether to connect to a database or not is made by checking whether the passed dbname parameter is replication. Unfortunately this makes it impossible to connect a to a database named replication... This is useful for future walsender commands which need database

[HACKERS] [PATCH 09/14] Adjust all *Satisfies routines to take a HeapTuple instead of a HeapTupleHeader

2012-11-14 Thread Andres Freund
For the regular satisfies routines this is needed in prepareation of logical decoding. I changed the non-regular ones for consistency as well. The naming between htup, tuple and similar is rather confused, I could not find any consistent naming anywhere. This is preparatory work for the logical

[HACKERS] [PATCH 12/14] Add a simple decoding module in contrib named 'test_decoding'

2012-11-14 Thread Andres Freund
--- contrib/Makefile | 1 + contrib/test_decoding/Makefile| 16 +++ contrib/test_decoding/test_decoding.c | 192 ++ 3 files changed, 209 insertions(+) create mode 100644 contrib/test_decoding/Makefile create mode 100644

[HACKERS] [PATCH 13/14] Introduce pg_receivellog, the pg_receivexlog equivalent for logical changes

2012-11-14 Thread Andres Freund
--- src/bin/pg_basebackup/Makefile | 7 +- src/bin/pg_basebackup/pg_receivellog.c | 717 + src/bin/pg_basebackup/streamutil.c | 3 +- src/bin/pg_basebackup/streamutil.h | 1 + 4 files changed, 725 insertions(+), 3 deletions(-) create mode

[HACKERS] [PATCH 02/14] Add support for a generic wal reading facility dubbed XLogReader

2012-11-14 Thread Andres Freund
Features: - streaming reading/writing - filtering - reassembly of records Reusing the ReadRecord infrastructure in situations where the code that wants to do so is not tightly integrated into xlog.c is rather hard and would require changes to rather integral parts of the recovery code which

Re: [HACKERS] logical changeset generation v3 - git repository

2012-11-14 Thread Andres Freund
On 2012-11-15 01:27:46 +0100, Andres Freund wrote: In response to this you will soon find the 14 patches that currently implement $subject. As its not very wieldly to send around that many/big patches all the time, until the next major version I will just update the git tree at: Web:

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Jeff Davis
On Tue, 2012-11-13 at 15:27 -0500, Robert Haas wrote: A small patch that gets committed is better than a big one that doesn't. Here's a small patch (two, actually, because the TLI one is uninteresting and noisy). It's based on Simon's patch, but with some significant changes: * I ripped out

Re: [HACKERS] lcr - walsender integration

2012-11-14 Thread Andres Freund
Hi, The current logical walsender integration looks like the following: =# INIT_LOGICAL_REPLICATION 'text'; WARNING: Initiating logical rep WARNING: reached consistent point, stopping! replication_id | consistent_point | snapshot_name | plugin

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Craig Ringer
On 11/15/2012 12:46 AM, Tom Lane wrote: Agreed, and there's also the question of passing switches etc to the program, so the string can't be a bare file name anyway. I proposed pipe symbols (|) in the string previously, but if you find that too Unix-centric I suppose we could do COPY

Re: [HACKERS] logical changeset generation v3

2012-11-14 Thread Josh Berkus
On 11/14/12 4:27 PM, Andres Freund wrote: Hi, In response to this you will soon find the 14 patches that currently implement $subject. I'll go over each one after showing off for a bit: Lemme be the first to say, wow. Impressive work. Now the debugging starts ... -- Josh Berkus

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Tom Lane
Craig Ringer cr...@2ndquadrant.com writes: On 11/15/2012 12:46 AM, Tom Lane wrote: Agreed, and there's also the question of passing switches etc to the program, so the string can't be a bare file name anyway. I proposed pipe symbols (|) in the string previously, but if you find that too

Re: [HACKERS] Enabling Checksums

2012-11-14 Thread Robert Haas
On Wed, Nov 14, 2012 at 6:24 PM, Jeff Davis pg...@j-davis.com wrote: Hmm... what if we took this a step further and actually stored the checksums in a separate relation fork? That would make it pretty simple to support enabling/disabling checksums for particular relations. It would also

Re: [HACKERS] Doc patch making firm recommendation for setting the value of commit_delay

2012-11-14 Thread David Rowley
-Original Message- From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers- ow...@postgresql.org] On Behalf Of Peter Geoghegan Sent: 15 November 2012 09:44 To: PG Hackers Subject: [HACKERS] Doc patch making firm recommendation for setting the value of commit_delay Some of

[HACKERS] [PATCH 03/14] Add simple xlogdump tool

2012-11-14 Thread Andres Freund
--- src/bin/Makefile| 2 +- src/bin/xlogdump/Makefile | 25 +++ src/bin/xlogdump/xlogdump.c | 468 3 files changed, 494 insertions(+), 1 deletion(-) create mode 100644 src/bin/xlogdump/Makefile create mode 100644

[HACKERS] [PATCH 07/14] Introduce InvalidCommandId and declare that to be the new maximum for CommandCounterIncrement

2012-11-14 Thread Andres Freund
This is useful to be able to represent a CommandId thats invalid. There was no such value before. This decreases the possible number of subtransactions by one which seems unproblematic. Its also not a problem for pg_upgrade because cmin/cmax are never looked at outside the context of their own

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Craig Ringer
On 11/15/2012 10:19 AM, Tom Lane wrote: I disagree very very strongly with that. If we prevent use of shell syntax, we will lose a lot of functionality, for instance copy ... from program 'foo somefile' copy ... from program 'foo | bar' unless you're imagining that we will

[HACKERS] [PATCH 14/14] design document v2.3 and snapshot building design doc v0.2

2012-11-14 Thread Andres Freund
--- src/backend/replication/logical/DESIGN.txt | 603 + src/backend/replication/logical/Makefile | 6 + .../replication/logical/README.SNAPBUILD.txt | 298 ++ 3 files changed, 907 insertions(+) create mode 100644

Re: [HACKERS] recursive view syntax

2012-11-14 Thread Peter Eisentraut
On Tue, 2012-11-13 at 23:44 -0500, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: I noticed we don't implement the recursive view syntax, even though it's part of the standard SQL feature set for recursive queries. Here is a patch to add that. Can't you simplify that by using

Re: [HACKERS] logical changeset generation v3

2012-11-14 Thread Michael Paquier
Looks like cool stuff @-@ I might be interested in looking at that a bit as I think I will hopefully be hopefully be able to grab some time in the next couple of weeks. Are some of those patches already submitted to a CF? -- Michael Paquier http://michael.otacoo.com

Re: [HACKERS] Materialized views WIP patch

2012-11-14 Thread Alvaro Herrera
Kevin Grittner wrote: Interesting stuff. /* + * SetRelationIsValid + * Set the value of the relation's relisvalid field in pg_class. + * + * NOTE: caller must be holding an appropriate lock on the relation. + * ShareUpdateExclusiveLock is sufficient. + * + * NOTE: an

Re: [HACKERS] Doc patch, put commas in the right place in pg_restore docs

2012-11-14 Thread Peter Eisentraut
On Tue, 2012-10-16 at 21:50 -0500, Karl O. Pinc wrote: Simple punctuation change to pg_restore docs. committed -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] Patch to compute Max LSN of Data Pages

2012-11-14 Thread Amit Kapila
On Wednesday, November 14, 2012 10:19 PM Fujii Masao wrote: On Thu, Nov 15, 2012 at 12:55 AM, Amit Kapila amit.kap...@huawei.com wrote: Now user can use this utility to decide if new-standby has max LSN greater And that situation can occur when new-standby has startpoint LSN greater

Re: [HACKERS] Move postgresql_fdw_validator into dblink

2012-11-14 Thread Shigeru Hanada
Sorry for long absence. On Sat, Oct 20, 2012 at 4:24 PM, Kohei KaiGai kai...@kaigai.gr.jp wrote: IIRC, the reason why postgresql_fdw instead of pgsql_fdw was no other fdw module has shorten naming such as ora_fdw for Oracle. However, I doubt whether it is enough strong reason to force to

[HACKERS] add -Wlogical-op to standard compiler options?

2012-11-14 Thread Peter Eisentraut
I think it might be worth adding -Wlogical-op to the standard warning options (for supported compilers, determined by configure test). `-Wlogical-op' Warn about suspicious uses of logical operators in expressions. This includes using logical operators in contexts where a bit-wise

[HACKERS] feature proposal - triggers by semantics

2012-11-14 Thread Darren Duncan
I have a feature request, which at one level should require little code change, but at another level may require more. Since Postgres 9.3 is going to be doing some significant feature additions for triggers, I'd like to see some more. As they currently exist, triggers always fire based on

Re: [HACKERS] [PATCH] Patch to compute Max LSN of Data Pages

2012-11-14 Thread Amit Kapila
On Tuesday, November 13, 2012 10:17 PM Fujii Masao wrote: On Tue, Nov 13, 2012 at 1:23 PM, Amit kapila amit.kap...@huawei.com wrote: On Monday, November 12, 2012 9:56 PM Alvaro Herrera wrote: Robert Haas escribió: On Tue, Jul 31, 2012 at 8:09 AM, Amit kapila amit.kap...@huawei.com wrote:

[HACKERS] Why does delete from table where not exists (select 1 from ... LIMIT 1) perform badly?

2012-11-14 Thread Palle Girgensohn
Hi, I've read about the reason for this before, but cannot find a reference to it now. How come the planner treats the delete from table where not extists(select 1 from table2 where ... LIMIT 1) so differently, and usually badly, when the LIMIT 1 is there. In older version of postgresql, I

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Etsuro Fujita
copy-popen-20121114.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP patch: add (PRE|POST)PROCESSOR options to COPY

2012-11-14 Thread Simon Riggs
On 13 September 2012 10:13, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: I'd like to add the following options to the SQL COPY command and the psql \copy instruction: * PREPROCESSOR: Specifies the user-supplied program for COPY IN. The data from an input file is preprocessed by the

[HACKERS] [PATCH] binary heap implementation

2012-11-14 Thread Abhijit Menon-Sen
Hi. There are two or three places in the Postgres source that implement heap sort (e.g. tuplesort.c, nodeMergeAppend.c), and it's also needed by the BDR code. It seemed reasonable to factor out the functionality. I've attached a patch (binaryheap.diff) that contains a straightforward