Re: [HACKERS] WAL logging problem in 9.4.3?

2017-09-13 Thread Kyotaro HORIGUCHI
At Wed, 13 Sep 2017 17:42:39 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170913.174239.25978735.horiguchi.kyot...@lab.ntt.co.jp> > filterdiff seems to did something wrong.. # to did... The patch is broken by filterdiff so I send a new patch made directly by git format-patch. I c

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-09-13 Thread Kyotaro HORIGUCHI
At Wed, 13 Sep 2017 15:05:31 +1200, Thomas Munro wrote in > It doesn't compile for me, because your patch removed the definition > of HEAP_INSERT_SKIP_WAL but hasn't removed that reference to it. > > I'm not sure what happened. Is it possible that your patch was not > created by diffing again

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-09-13 Thread Alvaro Herrera
Kyotaro HORIGUCHI wrote: > The CF status of this patch turned into "Waiting on Author" by > automated CI checking. I object to automated turning of patches to waiting on author by machinery. Sending occasional reminder messages to authors making them know about outdated patches seems acceptable

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-09-12 Thread Thomas Munro
On Wed, Sep 13, 2017 at 1:04 PM, Kyotaro HORIGUCHI wrote: > The CF status of this patch turned into "Waiting on Author" by > automated CI checking. However, I still don't get any error even > on the current master (69835bc) after make distclean. Also I > don't see any difference between the "probl

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-09-12 Thread Kyotaro HORIGUCHI
Hello, (does this seem to be a top post?) The CF status of this patch turned into "Waiting on Author" by automated CI checking. However, I still don't get any error even on the current master (69835bc) after make distclean. Also I don't see any difference between the "problematic" patch and my wor

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-09-11 Thread Kyotaro HORIGUCHI
Hello, At Fri, 08 Sep 2017 16:30:01 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170908.163001.53230385.horiguchi.kyot...@lab.ntt.co.jp> > > >> 2017-04-13 12:11:27.065 JST [85441] t/102_vacuumdb_stages.pl > > >> STATEMENT: ANALYZE; > > >> 2017-04-13 12:12:25.766 JST [85492] LOG:

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-09-08 Thread Kyotaro HORIGUCHI
Thank you for your notification. At Tue, 5 Sep 2017 12:05:01 +0200, Daniel Gustafsson wrote in > > On 13 Apr 2017, at 11:42, Kyotaro HORIGUCHI > > wrote: > > > > At Thu, 13 Apr 2017 13:52:40 +0900, Michael Paquier > > wrote in > > > >> On Tue, Apr 11, 2017 at 5:38 PM, Kyotaro HORIGUCHI >

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-09-05 Thread Daniel Gustafsson
> On 13 Apr 2017, at 11:42, Kyotaro HORIGUCHI > wrote: > > At Thu, 13 Apr 2017 13:52:40 +0900, Michael Paquier > wrote in > >> On Tue, Apr 11, 2017 at 5:38 PM, Kyotaro HORIGUCHI >> wrote: >>> Sorry, what I have just sent was broken. >> >> You can use PROVE_TESTS when running make check to

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-13 Thread Kyotaro HORIGUCHI
At Thu, 13 Apr 2017 13:52:40 +0900, Michael Paquier wrote in > On Tue, Apr 11, 2017 at 5:38 PM, Kyotaro HORIGUCHI > wrote: > > Sorry, what I have just sent was broken. > > You can use PROVE_TESTS when running make check to select a subset of > tests you want to run. I use that all the time whe

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-12 Thread Kyotaro HORIGUCHI
I'd like to put a supplimentary explanation. At Tue, 11 Apr 2017 17:38:12 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170411.173812.133964522.horiguchi.kyot...@lab.ntt.co.jp> > Sorry, what I have just sent was broken. > > At Tue, 11 Apr 2017 17:33:41 +0900 (Tokyo Standard Time),

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-12 Thread Michael Paquier
On Tue, Apr 11, 2017 at 5:38 PM, Kyotaro HORIGUCHI wrote: > Sorry, what I have just sent was broken. You can use PROVE_TESTS when running make check to select a subset of tests you want to run. I use that all the time when working on patches dedicated to certain code paths. >> - Relation has new

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-11 Thread Kyotaro HORIGUCHI
Sorry, what I have just sent was broken. At Tue, 11 Apr 2017 17:33:41 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170411.173341.257028732.horiguchi.kyot...@lab.ntt.co.jp> > At Tue, 11 Apr 2017 09:56:06 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI > wrote in > <20170411.09560

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-11 Thread Kyotaro HORIGUCHI
At Tue, 11 Apr 2017 09:56:06 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20170411.095606.245908357.horiguchi.kyot...@lab.ntt.co.jp> > Hello, thank you for looking this. > > At Fri, 07 Apr 2017 20:38:35 -0400, Tom Lane wrote in > <27309.1491611...@sss.pgh.pa.us> > > Alvaro Herrera

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-10 Thread Kyotaro HORIGUCHI
Hello, thank you for looking this. At Fri, 07 Apr 2017 20:38:35 -0400, Tom Lane wrote in <27309.1491611...@sss.pgh.pa.us> > Alvaro Herrera writes: > > Interesting. I wonder if it's possible that a relcache invalidation > > would cause these values to get lost for some reason, because that woul

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-07 Thread Tom Lane
Alvaro Herrera writes: > Interesting. I wonder if it's possible that a relcache invalidation > would cause these values to get lost for some reason, because that would > be dangerous. > I suppose the rationale is that this shouldn't happen because any > operation that does things this way must h

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-07 Thread Alvaro Herrera
Alvaro Herrera wrote: > I suppose the rationale is that this shouldn't happen because any > operation that does things this way must hold an exclusive lock on the > relation. But that doesn't guarantee that the relcache entry is > completely stable, does it? If we can get proof of that, then thi

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-07 Thread Alvaro Herrera
I have claimed this patch as committer FWIW. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://ww

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-04-07 Thread Alvaro Herrera
Kyotaro HORIGUCHI wrote: > The attached patch is quiiiccck-and-dirty-hack of Michael's patch > just as a PoC of my proposal quoted above. This also passes the > 006 test. The major changes are the following. > > - Moved sync_above and truncted_to into RelationData. Interesting. I wonder if it

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-17 Thread Amit Kapila
On Fri, Mar 17, 2017 at 12:30 PM, Ashutosh Sharma wrote: > On Fri, Mar 17, 2017 at 9:03 AM, Amit Kapila wrote: >> On Thu, Mar 16, 2017 at 1:15 PM, Ashutosh Sharma >> wrote: >>> Hi, >>> >>> Attached is the patch that allows WAL consistency tool to mask >>> 'LH_PAGE_HAS_DEAD_TUPLES' flag in hash

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-17 Thread Ashutosh Sharma
On Fri, Mar 17, 2017 at 9:03 AM, Amit Kapila wrote: > On Thu, Mar 16, 2017 at 1:15 PM, Ashutosh Sharma > wrote: >> Hi, >> >> Attached is the patch that allows WAL consistency tool to mask >> 'LH_PAGE_HAS_DEAD_TUPLES' flag in hash index. The flag got added as a >> part of 'Microvacuum support for

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-16 Thread Amit Kapila
On Thu, Mar 16, 2017 at 1:15 PM, Ashutosh Sharma wrote: > Hi, > > Attached is the patch that allows WAL consistency tool to mask > 'LH_PAGE_HAS_DEAD_TUPLES' flag in hash index. The flag got added as a > part of 'Microvacuum support for Hash index' patch . I have already > tested it using Kuntal's

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-16 Thread Ashutosh Sharma
Hi, Attached is the patch that allows WAL consistency tool to mask 'LH_PAGE_HAS_DEAD_TUPLES' flag in hash index. The flag got added as a part of 'Microvacuum support for Hash index' patch . I have already tested it using Kuntal's WAL consistency tool and it works fine. -- With Regards, Ashutosh S

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-14 Thread Kuntal Ghosh
On Wed, Mar 15, 2017 at 12:32 AM, Robert Haas wrote: > On Mon, Mar 13, 2017 at 10:36 AM, Ashutosh Sharma > wrote: >> Couple of review comments,, >> >> You may also need to update the documentation as now we are also going >> to support wal consistency check for hash index. The current >> documen

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-14 Thread Robert Haas
On Mon, Mar 13, 2017 at 10:36 AM, Ashutosh Sharma wrote: > Couple of review comments,, > > You may also need to update the documentation as now we are also going > to support wal consistency check for hash index. The current > documentation does not include hash index. > > +only records or

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-13 Thread Ashutosh Sharma
Couple of review comments,, You may also need to update the documentation as now we are also going to support wal consistency check for hash index. The current documentation does not include hash index. +only records originating from those resource managers. Currently, +the suppo

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-08 Thread Kuntal Ghosh
On Fri, Mar 3, 2017 at 9:44 AM, Amit Kapila wrote: > On Tue, Feb 28, 2017 at 11:06 AM, Kuntal Ghosh > wrote: >> Hello everyone, >> >> I've attached a patch which implements WAL consistency checking for >> hash indexes. This feature is going to be useful for developing and >> testing of WAL loggin

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-04 Thread Amit Kapila
On Sat, Mar 4, 2017 at 2:29 PM, Robert Haas wrote: > On Fri, Mar 3, 2017 at 9:44 AM, Amit Kapila wrote: >> On Tue, Feb 28, 2017 at 11:06 AM, Kuntal Ghosh >> wrote: >>> Hello everyone, >>> >>> I've attached a patch which implements WAL consistency checking for >>> hash indexes. This feature is go

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-04 Thread Robert Haas
On Fri, Mar 3, 2017 at 9:44 AM, Amit Kapila wrote: > On Tue, Feb 28, 2017 at 11:06 AM, Kuntal Ghosh > wrote: >> Hello everyone, >> >> I've attached a patch which implements WAL consistency checking for >> hash indexes. This feature is going to be useful for developing and >> testing of WAL loggin

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-02 Thread Amit Kapila
On Tue, Feb 28, 2017 at 11:06 AM, Kuntal Ghosh wrote: > Hello everyone, > > I've attached a patch which implements WAL consistency checking for > hash indexes. This feature is going to be useful for developing and > testing of WAL logging for hash index. > I think it is better if you base your pa

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-03-01 Thread David Steele
On 1/30/17 11:33 PM, Michael Paquier wrote: > On Fri, Dec 2, 2016 at 1:39 PM, Haribabu Kommi > wrote: >> The latest proposed patch still having problems. >> Closed in 2016-11 commitfest with "moved to next CF" status because of a bug >> fix patch. >> Please feel free to update the status once you

[HACKERS] WAL Consistency checking for hash indexes

2017-02-27 Thread Kuntal Ghosh
Hello everyone, I've attached a patch which implements WAL consistency checking for hash indexes. This feature is going to be useful for developing and testing of WAL logging for hash index. Note that, this patch should be applied on top of the following WAL logging for hash index patch: https://

Re: [HACKERS] WAL consistency check facility

2017-02-14 Thread Michael Paquier
On Wed, Feb 15, 2017 at 11:08 AM, Robert Haas wrote: > On Tue, Feb 14, 2017 at 7:12 PM, Tom Lane wrote: >> FWIW, my own habit when creating new PG files is generally to write >> >> * Portions Copyright (c) 1996-2017, PostgreSQL Global Development Group >> * Portions Copyright (c) 1994, Regents

Re: [HACKERS] WAL consistency check facility

2017-02-14 Thread Robert Haas
On Tue, Feb 14, 2017 at 7:12 PM, Tom Lane wrote: > Robert Haas writes: >> On Tue, Feb 14, 2017 at 5:16 PM, Michael Paquier >> wrote: >>> Just for curiosity: does the moment when the code has been written or >>> committed counts? It's no big deal seeing how liberal the Postgres >>> license is, bu

Re: [HACKERS] WAL consistency check facility

2017-02-14 Thread Tom Lane
Robert Haas writes: > On Tue, Feb 14, 2017 at 5:16 PM, Michael Paquier > wrote: >> Just for curiosity: does the moment when the code has been written or >> committed counts? It's no big deal seeing how liberal the Postgres >> license is, but this makes me wonder... > IANAL, but I think if you as

Re: [HACKERS] WAL consistency check facility

2017-02-14 Thread Robert Haas
On Tue, Feb 14, 2017 at 5:16 PM, Michael Paquier wrote: > On Wed, Feb 15, 2017 at 2:43 AM, Robert Haas wrote: >> On Thu, Feb 9, 2017 at 8:17 PM, Michael Paquier >> wrote: >>> Please find attached a patch with those fixes. >> >> Committed, but I changed the copyright dates to 2016-2017 rather tha

Re: [HACKERS] WAL consistency check facility

2017-02-14 Thread Michael Paquier
On Wed, Feb 15, 2017 at 2:43 AM, Robert Haas wrote: > On Thu, Feb 9, 2017 at 8:17 PM, Michael Paquier > wrote: >> Please find attached a patch with those fixes. > > Committed, but I changed the copyright dates to 2016-2017 rather than > just 2017 since surely some of the code was originally writt

Re: [HACKERS] WAL consistency check facility

2017-02-14 Thread Robert Haas
On Thu, Feb 9, 2017 at 8:17 PM, Michael Paquier wrote: > Please find attached a patch with those fixes. Committed, but I changed the copyright dates to 2016-2017 rather than just 2017 since surely some of the code was originally written before 2017. Even that might not really be going back far e

Re: [HACKERS] WAL consistency check facility

2017-02-09 Thread Michael Paquier
On Thu, Feb 9, 2017 at 5:56 AM, Robert Haas wrote: > On Wed, Feb 8, 2017 at 1:25 AM, Kuntal Ghosh > wrote: >> Thank you Robert for the review. Please find the updated patch in the >> attachment. > > I have committed this patch after fairly extensive revisions: Cool. I had finally a look at what

Re: [HACKERS] WAL consistency check facility

2017-02-08 Thread Kuntal Ghosh
On Thu, Feb 9, 2017 at 2:26 AM, Robert Haas wrote: > On Wed, Feb 8, 2017 at 1:25 AM, Kuntal Ghosh > wrote: >> Thank you Robert for the review. Please find the updated patch in the >> attachment. > > I have committed this patch after fairly extensive revisions: > Thank you, Robert, for the above

Re: [HACKERS] WAL consistency check facility

2017-02-08 Thread Robert Haas
On Wed, Feb 8, 2017 at 1:25 AM, Kuntal Ghosh wrote: > Thank you Robert for the review. Please find the updated patch in the > attachment. I have committed this patch after fairly extensive revisions: * Rewrote the documentation to give some idea what the underlying mechanism of operation of the

Re: [HACKERS] WAL consistency check facility

2017-02-07 Thread Kuntal Ghosh
On Tue, Jan 31, 2017 at 9:36 PM, Robert Haas wrote: > On Thu, Jan 5, 2017 at 12:54 AM, Kuntal Ghosh > wrote: >> I've attached the patch with the modified changes. PFA. > > Can this patch check contrib/bloom? > Added support for generic resource manager type. > +/* > + * Mask some

Re: [HACKERS] WAL consistency check facility

2017-02-07 Thread Robert Haas
On Tue, Feb 7, 2017 at 6:32 AM, Amit Kapila wrote: > On Tue, Jan 31, 2017 at 9:36 PM, Robert Haas wrote: >> >> +if (!HeapTupleHeaderXminFrozen(page_htup)) >> +page_htup->t_infomask |= HEAP_XACT_MASK; >> +else >> +page_htup->t_infomask |= HEA

Re: [HACKERS] WAL consistency check facility

2017-02-07 Thread Amit Kapila
On Tue, Jan 31, 2017 at 9:36 PM, Robert Haas wrote: > > +if (!HeapTupleHeaderXminFrozen(page_htup)) > +page_htup->t_infomask |= HEAP_XACT_MASK; > +else > +page_htup->t_infomask |= HEAP_XMAX_COMMITTED | > HEAP_XMAX_INVALID; > > Comment doesn't

Re: [HACKERS] WAL consistency check facility

2017-02-01 Thread Robert Haas
On Tue, Jan 31, 2017 at 9:31 PM, Michael Paquier wrote: > On Wed, Feb 1, 2017 at 1:06 AM, Robert Haas wrote: >> On Thu, Jan 5, 2017 at 12:54 AM, Kuntal Ghosh >> wrote: >>> I've attached the patch with the modified changes. PFA. >> >> Can this patch check contrib/bloom? > > Only full pages are ap

Re: [HACKERS] WAL consistency check facility

2017-01-31 Thread Kuntal Ghosh
On Wed, Feb 1, 2017 at 8:01 AM, Michael Paquier wrote: > On Wed, Feb 1, 2017 at 1:06 AM, Robert Haas wrote: >> +/* >> + * Leave if no masking functions defined, this is possible in the case >> + * resource managers generating just full page writes, comparing an >> + * image to it

Re: [HACKERS] WAL consistency check facility

2017-01-31 Thread Kuntal Ghosh
On Tue, Jan 31, 2017 at 9:36 PM, Robert Haas wrote: > On Thu, Jan 5, 2017 at 12:54 AM, Kuntal Ghosh > wrote: >> I've attached the patch with the modified changes. PFA. Thanks Robert for taking your time for the review. I'll update the patch with the changes suggested by you. -- Thanks & Regar

Re: [HACKERS] WAL consistency check facility

2017-01-31 Thread Michael Paquier
On Wed, Feb 1, 2017 at 1:06 AM, Robert Haas wrote: > On Thu, Jan 5, 2017 at 12:54 AM, Kuntal Ghosh > wrote: >> I've attached the patch with the modified changes. PFA. > > Can this patch check contrib/bloom? Only full pages are applied at redo by the generic WAL facility. So you would finish by c

Re: [HACKERS] WAL consistency check facility

2017-01-31 Thread Robert Haas
On Thu, Jan 5, 2017 at 12:54 AM, Kuntal Ghosh wrote: > I've attached the patch with the modified changes. PFA. Can this patch check contrib/bloom? +/* + * Mask some line pointer bits, particularly those marked as + * used on a master and unused on a standby. + */

Re: [HACKERS] WAL consistency check facility

2017-01-30 Thread Michael Paquier
On Thu, Jan 5, 2017 at 2:54 PM, Kuntal Ghosh wrote: > On Wed, Dec 21, 2016 at 10:53 PM, Robert Haas wrote: > >> On a first read-through of this patch -- I have not studied it in >> detail yet -- this looks pretty good to me. One concern is that this >> patch adds a bit of code to XLogInsert(), w

Re: [HACKERS] WAL logging problem in 9.4.3?

2017-01-30 Thread Michael Paquier
On Fri, Dec 2, 2016 at 1:39 PM, Haribabu Kommi wrote: > The latest proposed patch still having problems. > Closed in 2016-11 commitfest with "moved to next CF" status because of a bug > fix patch. > Please feel free to update the status once you submit the updated patch. And moved to CF 2017-03..

Re: [HACKERS] WAL consistency check facility

2017-01-04 Thread Kuntal Ghosh
On Wed, Dec 21, 2016 at 10:53 PM, Robert Haas wrote: > On a first read-through of this patch -- I have not studied it in > detail yet -- this looks pretty good to me. One concern is that this > patch adds a bit of code to XLogInsert(), which is a very hot piece of > code. Conceivably, that migh

Re: [HACKERS] WAL consistency check facility

2016-12-21 Thread Kuntal Ghosh
On Wed, Dec 21, 2016 at 10:53 PM, Robert Haas wrote: > On Mon, Nov 28, 2016 at 11:31 PM, Michael Paquier > wrote: >> Moved to CF 2017-01, as no committers have showed up yet :( > > Seeing no other volunteers, here I am. > Thanks Robert for looking into the patch. > On a first read-through of thi

Re: [HACKERS] WAL consistency check facility

2016-12-21 Thread Robert Haas
On Mon, Nov 28, 2016 at 11:31 PM, Michael Paquier wrote: > Moved to CF 2017-01, as no committers have showed up yet :( Seeing no other volunteers, here I am. On a first read-through of this patch -- I have not studied it in detail yet -- this looks pretty good to me. One concern is that this pa

Re: [HACKERS] WAL logging problem in 9.4.3?

2016-12-01 Thread Haribabu Kommi
On Wed, Nov 9, 2016 at 5:55 PM, Michael Paquier wrote: > > > On Wed, Nov 9, 2016 at 9:27 AM, Michael Paquier > wrote: > > On Wed, Nov 9, 2016 at 5:39 AM, Robert Haas > wrote: > >> On Thu, Feb 4, 2016 at 7:24 AM, Heikki Linnakangas > wrote: > >>> I dropped the ball on this one back in July, so

Re: [HACKERS] WAL consistency check facility

2016-11-28 Thread Michael Paquier
On Wed, Nov 16, 2016 at 2:15 AM, Michael Paquier wrote: > On Tue, Nov 15, 2016 at 7:50 AM, Kuntal Ghosh > wrote: >> I've modified the guc parameter name as wal_consistency_check (little >> hesitant for a participle in suffix :) ). Also, updated the sgml and >> variable name accordingly. > > The c

Re: [HACKERS] WAL recycle retading based on active sync rep.

2016-11-20 Thread Kyotaro HORIGUCHI
Hello, At Fri, 18 Nov 2016 10:16:22 -0800, Andres Freund wrote in <20161118181622.hklschaizwaxo...@alap3.anarazel.de> > Hi, > > On 2016-11-18 14:12:42 +0900, Kyotaro HORIGUCHI wrote: > > We had too-early WAL recycling during a test we had on a sync > > replication set. This is not a bug and a b

Re: [HACKERS] WAL recycle retading based on active sync rep.

2016-11-20 Thread Kyotaro HORIGUCHI
Thanks for the comment. At Fri, 18 Nov 2016 17:06:55 +0800, Craig Ringer wrote in > > We had too-early WAL recycling during a test we had on a sync > > replication set. This is not a bug and a bit extreme case but is > > contrary to expectation on synchronous replication. > > Isn't this preven

Re: [HACKERS] WAL recycle retading based on active sync rep.

2016-11-18 Thread Andres Freund
Hi, On 2016-11-18 14:12:42 +0900, Kyotaro HORIGUCHI wrote: > We had too-early WAL recycling during a test we had on a sync > replication set. This is not a bug and a bit extreme case but is > contrary to expectation on synchronous replication. I don't think you can expect anything else. > This

Re: [HACKERS] WAL recycle retading based on active sync rep.

2016-11-18 Thread Craig Ringer
On 18 Nov. 2016 13:14, "Kyotaro HORIGUCHI" wrote: > > Hello. > > We had too-early WAL recycling during a test we had on a sync > replication set. This is not a bug and a bit extreme case but is > contrary to expectation on synchronous replication. Isn't this prevented by using a physical replicat

[HACKERS] WAL recycle retading based on active sync rep.

2016-11-17 Thread Kyotaro HORIGUCHI
Hello. We had too-early WAL recycling during a test we had on a sync replication set. This is not a bug and a bit extreme case but is contrary to expectation on synchronous replication. > FATAL: could not receive data from WAL stream: ERROR: requested WAL segment > 00010088 has

Re: [HACKERS] WAL consistency check facility

2016-11-15 Thread Michael Paquier
On Tue, Nov 15, 2016 at 7:50 AM, Kuntal Ghosh wrote: > I've modified the guc parameter name as wal_consistency_check (little > hesitant for a participle in suffix :) ). Also, updated the sgml and > variable name accordingly. The changes look good to me. -- Michael -- Sent via pgsql-hackers ma

Re: [HACKERS] WAL consistency check facility

2016-11-15 Thread Kuntal Ghosh
On Tue, Nov 15, 2016 at 7:23 PM, Robert Haas wrote: > On Sat, Nov 12, 2016 at 10:06 PM, Peter Eisentraut > wrote: >> On 11/9/16 11:55 PM, Michael Paquier wrote: >>> + >>> + wal_consistency (string) >>> + >>> + wal_consistency configuration >>> parameter >>> + >>> +

Re: [HACKERS] WAL consistency check facility

2016-11-15 Thread Robert Haas
On Sat, Nov 12, 2016 at 10:06 PM, Peter Eisentraut wrote: > On 11/9/16 11:55 PM, Michael Paquier wrote: >> + >> + wal_consistency (string) >> + >> + wal_consistency configuration >> parameter >> + >> + >> + >> + >> +This parameter is used to

Re: [HACKERS] WAL consistency check facility

2016-11-12 Thread Michael Paquier
On Sun, Nov 13, 2016 at 12:06 PM, Peter Eisentraut wrote: > Could we name this something like wal_consistency_checking? > > Otherwise it sounds like you use this to select the amount of > consistency in the WAL (similar to, say, wal_level). Or wal_check? Or wal_consistency_check? -- Michael --

Re: [HACKERS] WAL consistency check facility

2016-11-12 Thread Peter Eisentraut
On 11/9/16 11:55 PM, Michael Paquier wrote: > + > + wal_consistency (string) > + > + wal_consistency configuration parameter > + > + > + > + > +This parameter is used to check the consistency of WAL records, i.e, > +whether the WAL reco

Re: [HACKERS] WAL consistency check facility

2016-11-11 Thread Kuntal Ghosh
On Fri, Nov 11, 2016 at 3:36 AM, Michael Paquier wrote: > On Fri, Nov 11, 2016 at 1:36 AM, Robert Haas wrote: >> So, who are all of the people involved in the effort to produce this >> patch, and what's the right way to attribute credit? > > The original idea was from Heikki as he has introduced

Re: [HACKERS] WAL consistency check facility

2016-11-10 Thread Michael Paquier
On Fri, Nov 11, 2016 at 1:36 AM, Robert Haas wrote: > So, who are all of the people involved in the effort to produce this > patch, and what's the right way to attribute credit? The original idea was from Heikki as he has introduced the idea of doing such checks if you look at the original thread

Re: [HACKERS] WAL consistency check facility

2016-11-10 Thread Robert Haas
On Thu, Nov 10, 2016 at 10:02 AM, Kuntal Ghosh wrote: >> With the patch for BRIN applied, I am able to get installcheck-world >> working with wal_consistency = all and a standby doing the consistency >> checks behind. Adding wal_consistency = all in PostgresNode.pm, the >> recovery tests are passi

Re: [HACKERS] WAL consistency check facility

2016-11-10 Thread Kuntal Ghosh
On Thu, Nov 10, 2016 at 10:25 AM, Michael Paquier wrote: > On Wed, Nov 9, 2016 at 11:32 PM, Kuntal Ghosh >> Thanks a lot for reviewing the patch. Based on your review, I've attached the >> I've done a fair amount of testing which includes regression tests >> and manual creation of inconsistencies

Re: [HACKERS] WAL consistency check facility

2016-11-09 Thread Michael Paquier
On Wed, Nov 9, 2016 at 11:32 PM, Kuntal Ghosh wrote: > On Fri, Nov 4, 2016 at 1:32 PM, Michael Paquier > wrote: >> Thank you for the new patch. This will be hopefully the last round of >> reviews, we are getting close to something that has an acceptable >> shape. > > Thanks a lot for reviewing th

Re: [HACKERS] WAL consistency check facility

2016-11-09 Thread Kuntal Ghosh
On Fri, Nov 4, 2016 at 1:32 PM, Michael Paquier wrote: > Thank you for the new patch. This will be hopefully the last round of > reviews, we are getting close to something that has an acceptable > shape. Thanks a lot for reviewing the patch. Based on your review, I've attached the updated patch al

Re: [HACKERS] WAL logging problem in 9.4.3?

2016-11-08 Thread Michael Paquier
On Wed, Nov 9, 2016 at 9:27 AM, Michael Paquier wrote: > On Wed, Nov 9, 2016 at 5:39 AM, Robert Haas wrote: >> On Thu, Feb 4, 2016 at 7:24 AM, Heikki Linnakangas wrote: >>> I dropped the ball on this one back in July, so here's an attempt to revive >>> this thread. >>> >>> I spent some time fixi

Re: [HACKERS] WAL logging problem in 9.4.3?

2016-11-08 Thread Michael Paquier
On Wed, Nov 9, 2016 at 5:39 AM, Robert Haas wrote: > On Thu, Feb 4, 2016 at 7:24 AM, Heikki Linnakangas wrote: >> I dropped the ball on this one back in July, so here's an attempt to revive >> this thread. >> >> I spent some time fixing the remaining issues with the prototype patch I >> posted ea

Re: [HACKERS] WAL logging problem in 9.4.3?

2016-11-08 Thread Robert Haas
On Thu, Feb 4, 2016 at 7:24 AM, Heikki Linnakangas wrote: > I dropped the ball on this one back in July, so here's an attempt to revive > this thread. > > I spent some time fixing the remaining issues with the prototype patch I > posted earlier, and rebased that on top of current git master. See a

Re: [HACKERS] WAL consistency check facility

2016-11-04 Thread Michael Paquier
On Fri, Nov 4, 2016 at 6:02 PM, Michael Paquier wrote: > On Fri, Nov 4, 2016 at 5:02 PM, Michael Paquier > wrote: >> On Thu, Nov 3, 2016 at 11:17 PM, Kuntal Ghosh >> wrote: >>> I've updated the patch for review. >> >> Thank you for the new patch. This will be hopefully the last round of >> revie

Re: [HACKERS] WAL consistency check facility

2016-11-04 Thread Michael Paquier
On Fri, Nov 4, 2016 at 5:02 PM, Michael Paquier wrote: > On Thu, Nov 3, 2016 at 11:17 PM, Kuntal Ghosh > wrote: >> I've updated the patch for review. > > Thank you for the new patch. This will be hopefully the last round of > reviews, we are getting close to something that has an acceptable > sha

Re: [HACKERS] WAL consistency check facility

2016-11-04 Thread Michael Paquier
On Thu, Nov 3, 2016 at 11:17 PM, Kuntal Ghosh wrote: > I've updated the patch for review. Thank you for the new patch. This will be hopefully the last round of reviews, we are getting close to something that has an acceptable shape. + + + + + + Did you try to compile

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Michael Paquier
On Fri, Nov 4, 2016 at 4:16 AM, Kuntal Ghosh wrote: > On Thu, Nov 3, 2016 at 8:24 PM, Robert Haas wrote: >> On Wed, Nov 2, 2016 at 10:30 AM, Kuntal Ghosh >> wrote: >>> - Another suggestion was to remove wal_consistency from PostgresNode.pm >>> because small buildfarm machines may suffer on it. A

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 8:24 PM, Robert Haas wrote: > On Wed, Nov 2, 2016 at 10:30 AM, Kuntal Ghosh > wrote: >> - Another suggestion was to remove wal_consistency from PostgresNode.pm >> because small buildfarm machines may suffer on it. Although I've no >> experience in this matter, I would like

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Robert Haas
On Wed, Nov 2, 2016 at 10:30 AM, Kuntal Ghosh wrote: > - Another suggestion was to remove wal_consistency from PostgresNode.pm > because small buildfarm machines may suffer on it. Although I've no > experience in this matter, I would like to be certain that nothings breaks > in recovery tests afte

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 7:47 PM, Kuntal Ghosh wrote: > I've updated the patch for review. > If an inconsistency is found, it'll just log it for now. Once, the patch is finalized, we can change it to FATAL as before. I was making sure that all regression tests should pass with the patch. It seems th

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
I've updated the patch for review. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com walconsistency_v12.patch Description: application/download -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.post

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Michael Paquier
On Thu, Nov 3, 2016 at 6:48 PM, Kuntal Ghosh wrote: > So, whenever we are required to use bimg_info flag, we should make > sure that has_image > is set. OK, we are taking past each other here. There are two possible patterns: - has_image is set, not apply, meaning that the image block is used for

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 2:52 PM, Michael Paquier wrote: > On Thu, Nov 3, 2016 at 6:15 PM, Kuntal Ghosh > wrote: >> Actually, I just verified that bimg_info is not even valid if >> has_image is not set. >> In DecodeXLogRecord, we initialize bimg_info only when has_image flag >> is set. So, keeping

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Michael Paquier
On Thu, Nov 3, 2016 at 6:15 PM, Kuntal Ghosh wrote: > Actually, I just verified that bimg_info is not even valid if > has_image is not set. > In DecodeXLogRecord, we initialize bimg_info only when has_image flag > is set. So, keeping them > separate doesn't look a good approach to me. If we keep t

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Michael Paquier
On Thu, Nov 3, 2016 at 5:56 PM, Kuntal Ghosh wrote: > I'm not getting why we should introduce a new redo action and return > from the function beforehand. Per my last email, same conclusion from here :) Sorry if I am picky and noisy on many points, I am trying to think about the value of each cha

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 2:27 PM, Michael Paquier wrote: > On Thu, Nov 3, 2016 at 4:04 PM, Michael Paquier > wrote: >> Wouldn't the definition of a new redo action make sense then? Say >> SKIPPED. None of the existing actions match the non-apply case. > > I just took 5 minutes to look at the code a

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Michael Paquier
On Thu, Nov 3, 2016 at 4:04 PM, Michael Paquier wrote: > Wouldn't the definition of a new redo action make sense then? Say > SKIPPED. None of the existing actions match the non-apply case. I just took 5 minutes to look at the code and reason about it, and something like what your patch is doing w

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 12:34 PM, Michael Paquier wrote: > On Thu, Nov 3, 2016 at 3:24 PM, Kuntal Ghosh > wrote: >> On Thu, Nov 3, 2016 at 2:35 AM, Michael Paquier >>> -/* If it's a full-page image, restore it. */ >>> -if (XLogRecHasBlockImage(record, block_id)) >>> +/* If full-page i

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Michael Paquier
On Thu, Nov 3, 2016 at 3:24 PM, Kuntal Ghosh wrote: > On Thu, Nov 3, 2016 at 2:35 AM, Michael Paquier > wrote: >>> - Another suggestion was to remove wal_consistency from PostgresNode.pm >>> because small buildfarm machines may suffer on it. Although I've no >>> experience in this matter, I would

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 2:35 AM, Michael Paquier wrote: >> - Another suggestion was to remove wal_consistency from PostgresNode.pm >> because small buildfarm machines may suffer on it. Although I've no >> experience in this matter, I would like to be certain that nothings breaks >> in recovery test

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Michael Paquier
On Wed, Nov 2, 2016 at 11:30 PM, Kuntal Ghosh wrote: > On Wed, Nov 2, 2016 at 1:11 PM, Kuntal Ghosh > wrote: >> Rest of the suggestions are well-taken. I'll update the patch accordingly. > I've updated the last submitted patch(v10) with the following changes: > - added a block level flag BKPIMAG

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Kuntal Ghosh
On Wed, Nov 2, 2016 at 1:11 PM, Kuntal Ghosh wrote: > Rest of the suggestions are well-taken. I'll update the patch accordingly. I've updated the last submitted patch(v10) with the following changes: - added a block level flag BKPIMAGE_APPLY to distinguish backup image blocks which needs to be res

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Kuntal Ghosh
On Wed, Nov 2, 2016 at 1:34 PM, Michael Paquier wrote: > Hm... Right. That was broken. And actually, while the record-level > flag is useful so as you don't need to rely on checking > wal_consistency when doing WAL redo, the block-level flag is useful to > make a distinction between blocks that ha

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Michael Paquier
On Wed, Nov 2, 2016 at 4:41 PM, Kuntal Ghosh wrote: > On Wed, Nov 2, 2016 at 10:23 AM, Michael Paquier > wrote: >> On Tue, Nov 1, 2016 at 10:31 PM, Robert Haas wrote: >>> IMHO, your rewrite of this patch was a bit heavy-handed. >> >> OK... Sorry for that. >> >>> I haven't >>> scrutinized the cod

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Kuntal Ghosh
On Wed, Nov 2, 2016 at 10:23 AM, Michael Paquier wrote: > On Tue, Nov 1, 2016 at 10:31 PM, Robert Haas wrote: >> IMHO, your rewrite of this patch was a bit heavy-handed. > > OK... Sorry for that. > >> I haven't >> scrutinized the code here so maybe it was a big improvement, and if so >> fine, but

Re: [HACKERS] WAL consistency check facility

2016-11-01 Thread Michael Paquier
On Tue, Nov 1, 2016 at 10:31 PM, Robert Haas wrote: > IMHO, your rewrite of this patch was a bit heavy-handed. OK... Sorry for that. > I haven't > scrutinized the code here so maybe it was a big improvement, and if so > fine, but if not it's better to collaborate with the author than to > take o

Re: [HACKERS] WAL consistency check facility

2016-11-01 Thread Peter Geoghegan
On Mon, Oct 31, 2016 at 5:31 AM, Robert Haas wrote: > On Fri, Oct 28, 2016 at 2:05 AM, Michael Paquier > wrote: >> And here we go. Here is a review as well as a large brush-up for this >> patch. A couple of things: >> - wal_consistency is using a list of RMGRs, at the cost of being >> PGC_POSTMAS

Re: [HACKERS] WAL consistency check facility

2016-11-01 Thread Robert Haas
On Mon, Oct 31, 2016 at 5:51 PM, Michael Paquier wrote: > Hehe, I was expecting you to jump on those lines. While looking at the > patch I have simplified it first to focus on the core engine of the > thing. Adding back this code sounds fine to me as there is a wall of > contestation. I offer to d

Re: [HACKERS] WAL logging problem in 9.4.3?

2016-10-31 Thread Kyotaro HORIGUCHI
Hi, At Sun, 2 Oct 2016 21:43:46 +0900, Michael Paquier wrote in > On Thu, Sep 29, 2016 at 10:02 PM, Kyotaro HORIGUCHI > wrote: > > Hello, > > > > At Thu, 29 Sep 2016 16:59:55 +0900, Michael Paquier > > wrote in > > > >> On Mon, Sep 26, 2016 at 5:03 PM, Kyotaro HORIGUCHI > >> wrote: > >> >

  1   2   3   4   5   6   7   8   9   10   >