[HACKERS] (auto)vacuum truncate exclusive lock

2013-04-10 Thread Jeff Janes
I guess I'm a couple releases late to review the "autovacuum truncate exclusive lock" patch (a79ae0bc0d454b9f2c95a), but this patch did not only affect autovac, it affects manual vacuum as well (as did the original behavior it is a modification of). So the compiler constants are misnamed, and the

Re: [HACKERS] [GSOC] questions about idea "rewrite pg_dump as library"

2013-04-10 Thread Pavel Golub
Hello, 帅. You wrote: 帅> Hi all, 帅> I'd like to introduce myself to the dev community. I am Shuai 帅> Fan, a student from Dalian University of Technology, DLUT , for 帅> short, China. And I am interested in working with PostgreSQL project in GSOC2013. 帅> I'm interested in the idea "Rewrite

Re: [HACKERS] [DOCS] synchronize_seqscans' description is a bit misleading

2013-04-10 Thread Tom Lane
Gurjeet Singh writes: > On Wed, Apr 10, 2013 at 11:10 PM, Tom Lane wrote: >> The point you're missing is that the synchronization is self-enforcing: > Let's consider a pathological case where a scan is performed by a user > controlled cursor, whose scan speed depends on how fast the user presses

Re: [HACKERS] [DOCS] synchronize_seqscans' description is a bit misleading

2013-04-10 Thread Gurjeet Singh
On Wed, Apr 10, 2013 at 11:10 PM, Tom Lane wrote: > Gurjeet Singh writes: > > If I'm reading the code right [1], this GUC does not actually > *synchronize* > > the scans, but instead just makes sure that a new scan starts from a > block > > that was reported by some other backend performing a sc

[HACKERS] ObjectClass/ObjectType mixup

2013-04-10 Thread Peter Eisentraut
src/backend/catalog/dependency.c:213: EventTriggerSupportsObjectType(getObjectClass(thisobj))) src/backend/commands/event_trigger.c:1014: Assert(EventTriggerSupportsObjectType(getObjectClass(object))); getObjectClass() returns type ObjectClass, but EventTriggerSupportsObjectType()

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Jeff Davis
On Wed, 2013-04-10 at 20:17 +0100, Simon Riggs wrote: > OK, so we have a single combined "calculate a checksum for a block" > function. That uses Jeff's zeroing trick and Ants' bulk-oriented > performance optimization. > > > For buffer checksums we simply calculate for the block. Sounds good.

Re: [HACKERS] [DOCS] synchronize_seqscans' description is a bit misleading

2013-04-10 Thread Tom Lane
Gurjeet Singh writes: > If I'm reading the code right [1], this GUC does not actually *synchronize* > the scans, but instead just makes sure that a new scan starts from a block > that was reported by some other backend performing a scan on the same > relation. Well, that's the only *direct* effec

[HACKERS] synchronize_seqscans' description is a bit misleading

2013-04-10 Thread Gurjeet Singh
If I'm reading the code right [1], this GUC does not actually *synchronize* the scans, but instead just makes sure that a new scan starts from a block that was reported by some other backend performing a scan on the same relation. Since the backends scanning the relation may be processing the rela

Re: [HACKERS] SIGHUP not received by custom bgworkers if postmaster is notified

2013-04-10 Thread Michael Paquier
Thanks for committing the fix! On Thu, Apr 11, 2013 at 4:11 AM, Alvaro Herrera wrote: > Michael Paquier escribió: > > Hi all, > > > > Please find attached a simple example of bgworker that logs a message > each > > time a SIGTERM or SIGHUP signal is received by it: > > - "hello signal: processed

Re: [HACKERS] [sepgsql 3/3] Add db_procedure:execute permission checks

2013-04-10 Thread Alvaro Herrera
Kohei KaiGai wrote: > This patch adds sepgsql support for permission checks almost > equivalent to the existing FUNCTION EXECUTE privilege. While skimming this patch I noticed that you're using getObjectDescription() as the "audit_name" of objects. This may be a bit unstable, for example consider

Re: [HACKERS] corrupt pages detected by enabling checksums

2013-04-10 Thread Robert Haas
On Sat, Apr 6, 2013 at 10:44 AM, Andres Freund wrote: > I feel pretty strongly that we shouldn't add any such complications to > XLogInsert() itself, its complicated enough already and it should be > made simpler, not more complicated. +1, emphatically. XLogInsert is a really nasty scalability b

Re: [HACKERS] [GSOC] questions about idea "rewrite pg_dump as library"

2013-04-10 Thread Tom Lane
Alvaro Herrera writes: > Hannu Krosing wrote: >> Natural solution to this seems to move most of pg_dump functionality >> into backend as functions, so we have pg_dump_xxx() for everything >> we want to dump plus a topological sort function for getting the >> objects in right order. > This idea do

Re: [HACKERS] [GSOC] questions about idea "rewrite pg_dump as library"

2013-04-10 Thread Alvaro Herrera
Hannu Krosing wrote: > On 04/10/2013 11:02 PM, Alvaro Herrera wrote: > >Peter Eisentraut wrote: > > > >>I think the main uses cases mentioned in connection with this idea are > >>usually in the direction of finer-grained control over what gets dumped > >>and how. But making pg_dump into a library

Re: [HACKERS] [GSOC] questions about idea "rewrite pg_dump as library"

2013-04-10 Thread Hannu Krosing
On 04/10/2013 11:02 PM, Alvaro Herrera wrote: Peter Eisentraut wrote: I think the main uses cases mentioned in connection with this idea are usually in the direction of finer-grained control over what gets dumped and how. But making pg_dump into a library would not necessarily address that. T

Re: [HACKERS] [GSOC] questions about idea "rewrite pg_dump as library"

2013-04-10 Thread Alvaro Herrera
Peter Eisentraut wrote: > I think the main uses cases mentioned in connection with this idea are > usually in the direction of finer-grained control over what gets dumped > and how. But making pg_dump into a library would not necessarily > address that. There's also the matter of embedding pg_du

Re: [HACKERS] [GSOC] questions about idea "rewrite pg_dump as library"

2013-04-10 Thread Peter Eisentraut
On 4/10/13 10:54 AM, ˧ wrote: > I'm interested in the idea "Rewrite (add) pg_dump and pg_restore > utilities as libraries (.so, .dll & .dylib)". The pg_dump code is a giant mess, and refactoring it as a library is perhaps not a project for a new hacker. Independent of that, I think the first cons

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Simon Riggs
On 10 April 2013 11:15, Ants Aasma wrote: > > * We might even be able to calculate CRC32 checksum for normal WAL > records, > > and use Ants' checksum for full page writes (only). So checking WAL > checksum > > would then be to confirm header passes CRC32 and then re-check the Ants > > checksum o

Re: [HACKERS] SIGHUP not received by custom bgworkers if postmaster is notified

2013-04-10 Thread Alvaro Herrera
Michael Paquier escribió: > Hi all, > > Please find attached a simple example of bgworker that logs a message each > time a SIGTERM or SIGHUP signal is received by it: > - "hello signal: processed SIGHUP" when SIGHUP is handled by my example > - "hello signal: processed SIGTERM" when SIGTERM is ha

Re: [HACKERS] SIGHUP not received by custom bgworkers if postmaster is notified

2013-04-10 Thread Alvaro Herrera
Michael Paquier escribió: > Hi all, > > While playing with custom background workers, I noticed that postmaster > does not notify its registered bgworkers if it receives SIGHUP, > so you have to send a SIGHUP directly to the bgworker process to notify it. > Signal handling is correctly done for SI

Re: [HACKERS] bgworker sigusr1 handler

2013-04-10 Thread Alvaro Herrera
Robert Haas escribió: > Just for fun, I implemented a toy background worker tonight using the > new bgworker framework. Generally, it went well, and I'm pleased with > the design of the new facility. Thanks. > However, I did notice one oddity. I initialized the worker flags like > this: > >

[HACKERS] Re: [COMMITTERS] pgsql: Get rid of USE_WIDE_UPPER_LOWER dependency in trigram constructi

2013-04-10 Thread Stefan Kaltenbrunner
On 04/08/2013 10:11 AM, Dimitri Fontaine wrote: > Tom Lane writes: >> If there is anybody still using Postgres on machines without wcstombs() or >> towlower(), and they have non-ASCII data indexed by pg_trgm, they'll need >> to REINDEX those indexes after pg_upgrade to 9.3, else searches may fail

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Andres Freund
On 2013-04-10 20:39:25 +0200, Boszormenyi Zoltan wrote: > 2013-04-10 18:46 keltezéssel, Fujii Masao írta: > >On Wed, Apr 10, 2013 at 11:16 PM, Andres Freund > >wrote: > >>On 2013-04-10 10:10:31 -0400, Tom Lane wrote: > >>>Amit Kapila writes: > On Wednesday, April 10, 2013 3:42 PM Samrat Reva

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Boszormenyi Zoltan
2013-04-10 18:46 keltezéssel, Fujii Masao írta: On Wed, Apr 10, 2013 at 11:16 PM, Andres Freund wrote: On 2013-04-10 10:10:31 -0400, Tom Lane wrote: Amit Kapila writes: On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: Sorry, this is incorrect. Streaming replication continuous, ma

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Jeff Davis
On Wed, 2013-04-10 at 11:01 +0300, Ants Aasma wrote: > I think we should first deal with using it for page checksums and if > future versions want to reuse some of the code for WAL checksums then > we can rearrange the code. Sounds good to me, although I expect we at least want any assembly to be

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Tom Lane
Ants Aasma writes: > We already rely on WAL-before-data to ensure correct recovery. What is > proposed here is to slightly redefine it to require WAL to be > replicated before it is considered to be flushed. This ensures that no > data page on disk differs from the WAL that the slave has. The > ma

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Ants Aasma
On Wed, Apr 10, 2013 at 7:44 PM, Shaun Thomas wrote: > On 04/10/2013 11:40 AM, Fujii Masao wrote: > >> Strange. If this is really true, shared disk failover solution is >> fundamentally broken because the standby needs to start up with the >> shared "corrupted" database at the failover. > > > How

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Fujii Masao
On Thu, Apr 11, 2013 at 1:44 AM, Shaun Thomas wrote: > On 04/10/2013 11:40 AM, Fujii Masao wrote: > >> Strange. If this is really true, shared disk failover solution is >> fundamentally broken because the standby needs to start up with the >> shared "corrupted" database at the failover. > > > How

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Fujii Masao
On Wed, Apr 10, 2013 at 11:16 PM, Andres Freund wrote: > On 2013-04-10 10:10:31 -0400, Tom Lane wrote: >> Amit Kapila writes: >> > On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: >> >> Sorry, this is incorrect. Streaming replication continuous, master is not >> >> waiting, whenever th

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Shaun Thomas
On 04/10/2013 11:40 AM, Fujii Masao wrote: Strange. If this is really true, shared disk failover solution is fundamentally broken because the standby needs to start up with the shared "corrupted" database at the failover. How so? Shared disk doesn't use replication. The point I was trying to

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Fujii Masao
On Wed, Apr 10, 2013 at 11:26 PM, Shaun Thomas wrote: > On 04/10/2013 09:10 AM, Tom Lane wrote: > >> IOW, I wouldn't consider skipping the rsync even if I had a feature >> like this. > > > Totally. Out in the field, we consider the "old" database corrupt the moment > we fail over. Strange. If thi

Re: [HACKERS] Behaviour of bgworker with SIGHUP

2013-04-10 Thread Alvaro Herrera
Guillaume Lelarge wrote: > worker_spi.naptime is the naptime between two checks. > worker_spi.total_workers is the number of workers to launch at > postmaster start time. The first one can change with a sighup, the last > one obviously needs a restart. Many thanks. Pushed as http://git.postgres

Re: [HACKERS] Problem with background worker

2013-04-10 Thread Alvaro Herrera
Marc Cousin escribió: > On 20/03/2013 16:33, Alvaro Herrera wrote: > >Ah. The reason for this problem is that the statement start time (which > >also sets the transaction start time, when it's the first statement) is > >set by postgres.c, not the transaction-control functions in xact.c. So > >yo

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Dang Minh Huong
2013/04/10 23:44、Andres Freund のメッセージ: > On 2013-04-10 23:37:44 +0900, Dang Minh Huong wrote: >> Thanks all, >> >> (2013/04/10 22:55), Andres Freund wrote: >>> On 2013-04-10 22:38:07 +0900, Kyotaro HORIGUCHI wrote: Hello, On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong wr

[HACKERS] [GSOC] questions about idea "rewrite pg_dump as library"

2013-04-10 Thread
Hi all, I'd like to introduce myself to the dev community. I am Shuai Fan, a student from Dalian University of Technology, DLUT , for short, China. And I am interested in working with PostgreSQL project in GSOC2013. I'm interested in the idea "Rewrite (add) pg_dump and pg_restore utiliti

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Andres Freund
On 2013-04-10 23:37:44 +0900, Dang Minh Huong wrote: > Thanks all, > > (2013/04/10 22:55), Andres Freund wrote: > >On 2013-04-10 22:38:07 +0900, Kyotaro HORIGUCHI wrote: > >>Hello, > >> > >>On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong > >>wrote: > >>>In 9.3, it sounds replication_timeout is

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Dang Minh Huong
Thanks all, (2013/04/10 22:55), Andres Freund wrote: On 2013-04-10 22:38:07 +0900, Kyotaro HORIGUCHI wrote: Hello, On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong wrote: In 9.3, it sounds replication_timeout is replaced by wal_sender_timeout. So if it is solved in 9.3 i think there is a way

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Shaun Thomas
On 04/10/2013 09:10 AM, Tom Lane wrote: IOW, I wouldn't consider skipping the rsync even if I had a feature like this. Totally. Out in the field, we consider the "old" database corrupt the moment we fail over. There is literally no way to verify the safety of any data along the broken chain,

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Bruce Momjian
On Wed, Apr 10, 2013 at 01:15:12PM +0300, Ants Aasma wrote: > > This work needs to happen now, since once the checksum algorithm is set we > > won't easily be able to change it. > > The page checksum algorithm needs to be decided now, but WAL CRCs and > full page writes can be changed in 9.4 and d

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Andres Freund
On 2013-04-10 10:10:31 -0400, Tom Lane wrote: > Amit Kapila writes: > > On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: > >> Sorry, this is incorrect. Streaming replication continuous, master is not > >> waiting, whenever the master writes the data page it checks that the WAL > >> reco

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Tom Lane
Amit Kapila writes: > On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: >> Sorry, this is incorrect. Streaming replication continuous, master is not >> waiting, whenever the master writes the data page it checks that the WAL >> record is written in standby till that LSN. > I am not sure

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Andres Freund
On 2013-04-10 22:38:07 +0900, Kyotaro HORIGUCHI wrote: > Hello, > > On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong wrote: > > In 9.3, it sounds replication_timeout is replaced by wal_sender_timeout. > > So if it is solved in 9.3 i think there is a way to terminate it. > > I hope it is fixed in

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Kyotaro HORIGUCHI
Hello, On Wed, Apr 10, 2013 at 6:57 PM, Dang Minh Huong wrote: > In 9.3, it sounds replication_timeout is replaced by wal_sender_timeout. > So if it is solved in 9.3 i think there is a way to terminate it. > I hope it is fixed in 9.1 soon Hmm. He said that, > But in my environment the sender pr

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Amit Kapila
On Wednesday, April 10, 2013 3:42 PM Samrat Revagade wrote: >>(5) The master then forces a write of the data page related to this transaction. > Sorry, this is incorrect. Whenever the master writes the data page it checks that the WAL record is written in standby till that LSN.  >> While master i

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Ants Aasma
On Wed, Apr 10, 2013 at 12:25 PM, Simon Riggs wrote: > On 10 April 2013 09:01, Ants Aasma wrote: >> >> >> > Using SIMD for WAL is not a requirement at all; I just thought it might >> > be a nice benefit for non-checksum-enabled users in some later release. >> >> I think we should first deal with

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Samrat Revagade
>(5) *The master then forces a write of the data page related to this transaction.* *Sorry, this is incorrect. Whenever the master writes the data page it checks that the WAL record is written in standby till that LSN. * * * >While master is waiting to force a write (point 5) for this data page,

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Dang Minh Huong
Hi Amit, Thank you for your consideration. My project not allows to use 9.2 or 9.3. In 9.3, it sounds replication_timeout is replaced by wal_sender_timeout. So if it is solved in 9.3 i think there is a way to terminate it. I hope it is fixed in 9.1 soon Regards, 2013/04/10 18:33、Amit Kapila

Re: [HACKERS] Inconsistent DB data in Streaming Replication

2013-04-10 Thread Samrat Revagade
>>it's one of the reasons why a fresh base backup is required when starting old master as new standby? >>If yes, I agree with you. I've often heard the complaints about a backup when restarting new standby. >>That's really big problem. I think Fujii Masao is on the same page. >In case of syncre

Re: [HACKERS] [BUGS] replication_timeout not effective

2013-04-10 Thread Amit Kapila
> Sent: Wednesday, April 10, 2013 1:49 PM Dang Minh Huong wrote: > To: Amit Kapila > Subject: Re: [BUGS] replication_timeout not effective On Wednesday, April 10, 2013 1:49 PM > Hi, > > Thank you for your soon reply. > > I'm trying to set the network timeout related parameters to terminate > it. >

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Simon Riggs
On 10 April 2013 09:01, Ants Aasma wrote: > > > Using SIMD for WAL is not a requirement at all; I just thought it might > > be a nice benefit for non-checksum-enabled users in some later release. > > I think we should first deal with using it for page checksums and if > future versions want to re

Re: [HACKERS] Enabling Checksums

2013-04-10 Thread Ants Aasma
On Wed, Apr 10, 2013 at 4:36 AM, Jeff Davis wrote: > On Tue, 2013-04-09 at 05:35 +0300, Ants Aasma wrote: >> And here you go. I decided to be verbose with the comments as it's >> easier to delete a comment to write one. I also left in a huge jumble >> of macros to calculate the contents of a helpe

Re: [HACKERS] lwlock contention with SSI

2013-04-10 Thread Dan Ports
On Tue, Apr 09, 2013 at 07:49:51PM -0400, Robert Haas wrote: > These locks are all SSI-related and they're all really hot. Lock 28 > is SerializableXactHashLock and lock 29 is > SerializableFinishedListLock; both are acquired an order of magnitude > more often than any non-SSI lock, and cause two

Re: [HACKERS] page 1 of relation global/11787 was uninitialized

2013-04-10 Thread Albe Laurenz
Tom Lane wrote: > Andres Freund writes: >> Afaik we don't have any debugging utility to dump the pg_filenode.map >> contents? > > Hardly need one ... od -t d4 $PGDATA/global/pg_filenode.map > is readable enough, though it does leave you still having to > map the numeric OIDs back to names. The O