Re: [HACKERS] Page Checksums

2012-01-24 Thread jesper
* Robert Treat: Would it be unfair to assert that people who want checksums but aren't willing to pay the cost of running a filesystem that provides checksums aren't going to be willing to make the cost/benefit trade off that will be asked for? Yes, it is unfair of course, but it's

Re: [HACKERS] Page Checksums

2012-01-24 Thread Florian Weimer
I would chip in and say that I would prefer sticking to well-known proved filesystems like xfs/ext4 and let the application do the checksumming. Yes, that's a different way of putting my concern. If you want a proven file system with checksumming (and an fsck), options are really quite

Re: [HACKERS] WAL Restore process during recovery

2012-01-24 Thread Fujii Masao
On Tue, Jan 24, 2012 at 12:23 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Jan 23, 2012 at 12:23 PM, Fujii Masao masao.fu...@gmail.com wrote: Why does walrestore need to be invoked even when restore_command is not specified? It seems to be useless. We invoke walreceiver only when

Re: [HACKERS] WAL Restore process during recovery

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 9:43 AM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jan 24, 2012 at 12:23 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Jan 23, 2012 at 12:23 PM, Fujii Masao masao.fu...@gmail.com wrote: Why does walrestore need to be invoked even when restore_command is

Re: [HACKERS] Online base backup from the hot-standby

2012-01-24 Thread Fujii Masao
On Mon, Jan 23, 2012 at 10:11 PM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Jan 23, 2012 at 10:29 AM, Fujii Masao masao.fu...@gmail.com wrote: On Fri, Jan 20, 2012 at 11:34 PM, Simon Riggs si...@2ndquadrant.com wrote: On Fri, Jan 20, 2012 at 12:54 PM, Fujii Masao masao.fu...@gmail.com

Re: [HACKERS] WAL Restore process during recovery

2012-01-24 Thread Fujii Masao
On Tue, Jan 24, 2012 at 6:49 PM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Jan 24, 2012 at 9:43 AM, Fujii Masao masao.fu...@gmail.com wrote: On Tue, Jan 24, 2012 at 12:23 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Jan 23, 2012 at 12:23 PM, Fujii Masao masao.fu...@gmail.com

Re: [HACKERS] New replication mode: write

2012-01-24 Thread Fujii Masao
On Mon, Jan 23, 2012 at 9:53 PM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, Jan 23, 2012 at 10:03 AM, Fujii Masao masao.fu...@gmail.com wrote: To make the walreceiver call WaitLatchOrSocket(), we would need to merge it and libpq_select() into one function. But the former is the backend

Re: [HACKERS] Online base backup from the hot-standby

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 9:51 AM, Fujii Masao masao.fu...@gmail.com wrote: I'll proceed to commit for this now. Thanks a lot! Can I just check a few things? You say /* +* Update full_page_writes in shared memory and write an +* XLOG_FPW_CHANGE record before resource manager

Re: [HACKERS] Online base backup from the hot-standby

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 10:54 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Jan 24, 2012 at 9:51 AM, Fujii Masao masao.fu...@gmail.com wrote: I'll proceed to commit for this now. Thanks a lot! Can I just check a few things? Just to clarify, not expecting another patch version, just

Re: [HACKERS] New replication mode: write

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 10:47 AM, Fujii Masao masao.fu...@gmail.com wrote: I'm afraid I could not understand your idea. Could you explain it in more detail? We either tell libpqwalreceiver about the latch, or we tell walreceiver about the socket used by libpqwalreceiver. In either case we

Re: GiST for range types (was Re: [HACKERS] Range Types - typo + NULL string constructor)

2012-01-24 Thread Alexander Korotkov
Hi! New version of patch is attached. On Thu, Dec 22, 2011 at 11:46 AM, Jeff Davis pg...@j-davis.com wrote: A few comments: * In range_gist_picksplit, it would be nice to have a little bit more intuitive description of what's going on with the nonEmptyCount and nonInfCount numbers. For

Re: [HACKERS] basic pgbench runs with various performance-related patches

2012-01-24 Thread Robert Haas
On Tue, Jan 24, 2012 at 1:26 AM, Tatsuo Ishii is...@postgresql.org wrote: ** pgbench, permanent tables, scale factor 100, 300 s ** 1 group-commit-2012-01-21 614.425851 -10.4% 8 group-commit-2012-01-21 4705.129896 +6.3% 16 group-commit-2012-01-21 7962.131701 +2.0% 24 group-commit-2012-01-21

Re: GUC_REPORT for protocol tunables was: Re: [HACKERS] Optimize binary serialization format of arrays with fixed size elements

2012-01-24 Thread Robert Haas
On Mon, Jan 23, 2012 at 5:49 PM, Merlin Moncure mmonc...@gmail.com wrote: I'm not sure that you're getting anything with that user facing complexity.  The only realistic case I can see for explicit control of wire formats chosen is to defend your application from format changes in the server

Re: [HACKERS] Inline Extension

2012-01-24 Thread Robert Haas
On Mon, Jan 23, 2012 at 7:59 PM, Daniel Farina dan...@heroku.com wrote: Things are still a bit ugly in the more complex cases: consider PostGIS's linkage against libproj and other libraries.  In order to cover all cases, I feel that what I need is an optional hook (for the same of argument,

Re: Removing freelist (was Re: [HACKERS] Should I implement DROP INDEX CONCURRENTLY?)

2012-01-24 Thread Jeff Janes
On Sat, Jan 21, 2012 at 2:29 PM, Jim Nasby j...@nasby.net wrote: On Jan 20, 2012, at 11:54 AM, Heikki Linnakangas wrote: So, you're proposing that we remove freelist altogether? Sounds reasonable, but that needs to be performance tested somehow. I'm not sure what exactly the test should

Re: [HACKERS] Page Checksums

2012-01-24 Thread Robert Treat
On Tue, Jan 24, 2012 at 3:02 AM,  jes...@krogh.cc wrote: * Robert Treat: Would it be unfair to assert that people who want checksums but aren't willing to pay the cost of running a filesystem that provides checksums aren't going to be willing to make the cost/benefit trade off that will be

Re: Removing freelist (was Re: [HACKERS] Should I implement DROP INDEX CONCURRENTLY?)

2012-01-24 Thread Jeff Janes
On Mon, Jan 23, 2012 at 5:06 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 23, 2012 at 12:12 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Sat, Jan 21, 2012 at 5:29 PM, Jim Nasby j...@nasby.net wrote: We should also look at having the freelist

Re: Removing freelist (was Re: [HACKERS] Should I implement DROP INDEX CONCURRENTLY?)

2012-01-24 Thread Simon Riggs
On Mon, Jan 23, 2012 at 4:01 PM, Tom Lane t...@sss.pgh.pa.us wrote: The real problem there is that BufFreelistLock is also used to protect the clock sweep pointer. Agreed I think basically we gotta find a way to allow multiple backends to run clock sweeps concurrently.  Or else fix things

Re: [HACKERS] Page Checksums

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 2:49 PM, Robert Treat r...@xzilla.net wrote: And yes, I would for sure turn such functionality on if it were present. That's nice to say, but most people aren't willing to take a 50% performance hit. Not saying what we end up with will be that bad, but I've seen

Re: [HACKERS] Multithread Query Planner

2012-01-24 Thread Robert Haas
On Mon, Jan 23, 2012 at 2:45 PM, Merlin Moncure mmonc...@gmail.com wrote: Yes, but OP is proposing to use multiple threads inside the forked execution process.  That's a completely different beast.  Many other databases support parallel execution of a single query and it might very well be

Re: GUC_REPORT for protocol tunables was: Re: [HACKERS] Optimize binary serialization format of arrays with fixed size elements

2012-01-24 Thread Merlin Moncure
On Tue, Jan 24, 2012 at 8:26 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Jan 23, 2012 at 5:49 PM, Merlin Moncure mmonc...@gmail.com wrote: I'm not sure that you're getting anything with that user facing complexity.  The only realistic case I can see for explicit control of wire

Re: [HACKERS] Measuring relation free space

2012-01-24 Thread Jaime Casanova
On Mon, Jan 23, 2012 at 7:18 PM, Noah Misch n...@leadboat.com wrote: On Mon, Jan 23, 2012 at 04:56:24PM -0300, Alvaro Herrera wrote: Hm.  Leaf pages hold as much tuples as non-leaf pages, no?  I mean for each page element there's a value and a CTID.  In non-leaf those CTIDs point to other

Re: [HACKERS] Multithread Query Planner

2012-01-24 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I doubt it. Almost nothing in the backend is thread-safe. You can't acquire a heavyweight lock, a lightweight lock, or a spinlock. You can't do anything that might elog() or ereport(). None of those things are reentrant. Not to mention palloc,

Re: [HACKERS] Multithread Query Planner

2012-01-24 Thread Robert Haas
On Tue, Jan 24, 2012 at 11:25 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I doubt it.  Almost nothing in the backend is thread-safe.  You can't acquire a heavyweight lock, a lightweight lock, or a spinlock. You can't do anything that might elog() or

Re: [HACKERS] lots of unused variable warnings in assert-free builds

2012-01-24 Thread Robert Haas
On Sat, Jan 21, 2012 at 1:06 PM, Peter Eisentraut pete...@gmx.net wrote: So, here are three patches that could solve this issue. pg-cassert-unused-attribute.patch, the one I already showed, using __attribute__((unused). pg-cassert-unused-ifdef.patch, using only additional #ifdef

Re: [HACKERS] lots of unused variable warnings in assert-free builds

2012-01-24 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Spraying the code with __attribute__((unused)) is somewhat undesirable because it could mask a failure to properly initialize the variable in an assert-enabled build. Ouch. Is it really true that __attribute__((unused)) disables detection of use of

Re: [HACKERS] lots of unused variable warnings in assert-free builds

2012-01-24 Thread Robert Haas
On Tue, Jan 24, 2012 at 12:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: Spraying the code with __attribute__((unused)) is somewhat undesirable because it could mask a failure to properly initialize the variable in an assert-enabled build. Ouch.  Is it

Re: [HACKERS] Page Checksums

2012-01-24 Thread Jim Nasby
On Jan 24, 2012, at 9:15 AM, Simon Riggs wrote: On Tue, Jan 24, 2012 at 2:49 PM, Robert Treat r...@xzilla.net wrote: And yes, I would for sure turn such functionality on if it were present. That's nice to say, but most people aren't willing to take a 50% performance hit. Not saying what we

[HACKERS] PgNext: CFP

2012-01-24 Thread Joshua D. Drake
Hello, The call for papers for PgNext (the old PgWest/PgEast) is now open: January 19th: Talk submission opens April 15th: Talk submission closes April 30th: Speaker notification Submit: https://www.postgresqlconference.org/talk_types Sincerely, JD -- Command Prompt, Inc. -

Re: GUC_REPORT for protocol tunables was: Re: [HACKERS] Optimize binary serialization format of arrays with fixed size elements

2012-01-24 Thread Robert Haas
On Tue, Jan 24, 2012 at 11:16 AM, Merlin Moncure mmonc...@gmail.com wrote: Our current protocol allocates a 2-byte integer for the purposes of specifying the type of each parameter, and another 2-byte integer for the purpose of specifying the result type... but only one bit is really needed at

Re: [HACKERS] lots of unused variable warnings in assert-free builds

2012-01-24 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: On Tue, Jan 24, 2012 at 12:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Ouch. Is it really true that __attribute__((unused)) disables detection of use of uninitialized variables? Oh, I think maybe I am confused. The downsides of disabling *unused*

Re: [HACKERS] lots of unused variable warnings in assert-free builds

2012-01-24 Thread Tom Lane
I wrote: Also, it occurs to me that an intermediate macro PG_USED_FOR_ASSERTS_ONLY would be a good idea, first because it documents *why* you want to mark the variable as possibly unused, and second because changing the macro definition would provide an easy way to check for totally-unused

Re: [HACKERS] lots of unused variable warnings in assert-free builds

2012-01-24 Thread Robert Haas
On Tue, Jan 24, 2012 at 1:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: I wrote: Also, it occurs to me that an intermediate macro PG_USED_FOR_ASSERTS_ONLY would be a good idea, first because it documents *why* you want to mark the variable as possibly unused, and second because changing the macro

Re: [HACKERS] patch : Allow toast tables to be moved to a different tablespace

2012-01-24 Thread Robert Haas
On Sun, Jan 22, 2012 at 11:04 AM, Julien Tachoires jul...@gmail.com wrote: 2011/12/15 Alvaro Herrera alvhe...@commandprompt.com: Uhm, surely you could compare the original toast tablespace to the heap tablespace, and if they differ, handle appropriately when creating the new toast table?  

Re: [HACKERS] lots of unused variable warnings in assert-free builds

2012-01-24 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: Yes, that's what I meant when I suggested it originally. I'm just not sure it's any nicer than adding ifdefs for USE_ASSERT_CHECKING. I'm inclined to think that it probably is nicer, just because of less vertical space used. But again, this opinion is

Re: [HACKERS] PgNext: CFP

2012-01-24 Thread Dave Page
What date venue? On Tuesday, January 24, 2012, Joshua D. Drake j...@commandprompt.com wrote: Hello, The call for papers for PgNext (the old PgWest/PgEast) is now open: January 19th: Talk submission opens April 15th: Talk submission closes April 30th: Speaker notification Submit:

Re: [HACKERS] controlling the location of server-side SSL files

2012-01-24 Thread Peter Eisentraut
On tor, 2012-01-19 at 15:44 -0500, Robert Haas wrote: On Sat, Jan 14, 2012 at 8:40 AM, Peter Eisentraut pete...@gmx.net wrote: On mån, 2012-01-02 at 06:32 +0200, Peter Eisentraut wrote: I think I would like to have a set of GUC parameters to control the location of the server-side SSL

Re: [HACKERS] PgNext: CFP

2012-01-24 Thread Stefan Kaltenbrunner
On 01/24/2012 07:36 PM, Dave Page wrote: What date venue? On Tuesday, January 24, 2012, Joshua D. Drake j...@commandprompt.com wrote: Hello, The call for papers for PgNext (the old PgWest/PgEast) is now open: January 19th: Talk submission opens April 15th: Talk submission closes April

Re: [HACKERS] New replication mode: write

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 11:00 AM, Simon Riggs si...@2ndquadrant.com wrote: Yes, it might be too hard, but lets look. Your committer has timed out ;-) committed write mode only --  Simon Riggs   http://www.2ndQuadrant.com/  PostgreSQL Development, 24x7 Support, Training

[HACKERS] some longer, larger pgbench tests with various performance-related patches

2012-01-24 Thread Robert Haas
Early yesterday morning, I was able to use Nate Boley's test machine do a single 30-minute pgbench run at scale factor 300 using a variety of trees built with various patches, and with the -l option added to track latency on a per-transaction basis. All tests were done using 32 clients and

Re: GUC_REPORT for protocol tunables was: Re: [HACKERS] Optimize binary serialization format of arrays with fixed size elements

2012-01-24 Thread Merlin Moncure
On Tue, Jan 24, 2012 at 11:55 AM, Robert Haas robertmh...@gmail.com wrote: I do wonder whether we are making a mountain out of a mole-hill here, though.  If I properly understand the proposal on the table, which it's possible that I don't, but if I do, the new format is self-identifying: when

Re: [HACKERS] some longer, larger pgbench tests with various performance-related patches

2012-01-24 Thread Simon Riggs
On Tue, Jan 24, 2012 at 8:53 PM, Robert Haas robertmh...@gmail.com wrote: do a single 30-minute pgbench run at scale factor 300 using a variety Nice A minor but necessary point: Repeated testing of the Group commit patch when you have synch commit off is clearly pointless, so publishing

[HACKERS] Different error messages executing CREATE TABLE or ALTER TABLE to create a column xmin

2012-01-24 Thread Vik Reykja
I took my first stab at hacking the sources to fix the bug reported here: http://archives.postgresql.org/pgsql-bugs/2012-01/msg00152.php It's such a simple patch but it took me several hours with Google and IRC and I'm sure I did many things wrong. It seems to work as advertised, though, so I'm

Re: [HACKERS] basic pgbench runs with various performance-related patches

2012-01-24 Thread Tatsuo Ishii
My test was run with synchronous_commit=off, so I didn't expect the group commit patch to have much of an impact. I included it mostly to see whether by chance it helped anyway (since it also helps other WAL flushes, not just commits) or whether it caused any regression. Oh, I see. One

Re: GUC_REPORT for protocol tunables was: Re: [HACKERS] Optimize binary serialization format of arrays with fixed size elements

2012-01-24 Thread Tom Lane
Merlin Moncure mmonc...@gmail.com writes: On Tue, Jan 24, 2012 at 11:55 AM, Robert Haas robertmh...@gmail.com wrote: I do wonder whether we are making a mountain out of a mole-hill here, though. If I properly understand the proposal on the table, which it's possible that I don't, but if I do,

Re: GUC_REPORT for protocol tunables was: Re: [HACKERS] Optimize binary serialization format of arrays with fixed size elements

2012-01-24 Thread Robert Haas
On Tue, Jan 24, 2012 at 8:13 PM, Tom Lane t...@sss.pgh.pa.us wrote: Well, the bytea experience was IMNSHO a complete disaster (It was earlier mentioned that jdbc clients were silently corrupting bytea datums) and should be held up as an example of how *not* to do things; Yeah.  In both cases,

[HACKERS] Fix for pg_upgrade tablespace function usage

2012-01-24 Thread Bruce Momjian
I have applied the attached patch to git head to fix the new SQL tablespace location function usage in pg_upgrade to properly check cluster version numbers, and a fix missing table alias. I found this problem during testing. -- Bruce Momjian br...@momjian.ushttp://momjian.us

Re: [HACKERS] pg_upgrade with plpython is broken

2012-01-24 Thread Bruce Momjian
On Fri, Jan 20, 2012 at 07:01:46AM +0200, Peter Eisentraut wrote: On tor, 2012-01-19 at 17:04 -0500, Bruce Momjian wrote: For that reason, I wonder if I should just hard-code the plpython rename into the pg_upgrade test in check_loadable_libraries(). Yes, I haven't come up with a better

Re: [HACKERS] Group commit, revised

2012-01-24 Thread Simon Riggs
On Fri, Jan 20, 2012 at 10:30 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I spent some time cleaning this up. Details below, but here are the highlights: * Reverted the removal of wal_writer_delay * Removed heuristic on big flushes No contested viewpoints on anything

Re: [HACKERS] some longer, larger pgbench tests with various performance-related patches

2012-01-24 Thread Pavan Deolasee
On Wed, Jan 25, 2012 at 2:23 AM, Robert Haas robertmh...@gmail.com wrote: Early yesterday morning, I was able to use Nate Boley's test machine do a single 30-minute pgbench run at scale factor 300 using a variety of trees built with various patches, and with the -l option added to track

Re: [HACKERS] PgNext: CFP

2012-01-24 Thread Dave Page
On Tuesday, January 24, 2012, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: On 01/24/2012 07:36 PM, Dave Page wrote: What date venue? On Tuesday, January 24, 2012, Joshua D. Drake j...@commandprompt.com wrote: Hello, The call for papers for PgNext (the old PgWest/PgEast) is now

Re: [HACKERS] [COMMITTERS] pgsql: Add new replication mode synchronous_commit = 'write'.

2012-01-24 Thread Fujii Masao
On Wed, Jan 25, 2012 at 5:35 AM, Jaime Casanova ja...@2ndquadrant.com wrote: On Tue, Jan 24, 2012 at 3:22 PM, Simon Riggs si...@2ndquadrant.com wrote: Add new replication mode synchronous_commit = 'write'. Replication occurs only to memory on standby, not to disk, so provides additional

Re: [HACKERS] New replication mode: write

2012-01-24 Thread Fujii Masao
On Wed, Jan 25, 2012 at 5:28 AM, Simon Riggs si...@2ndquadrant.com wrote: On Tue, Jan 24, 2012 at 11:00 AM, Simon Riggs si...@2ndquadrant.com wrote: Yes, it might be too hard, but lets look. Your committer has timed out ;-) committed write mode only Thanks for the commit! The apply