Re: [HACKERS] WAL replay bugs

On Thu, Jun 18, 2015 at 3:39 AM, Alvaro Herrera wrote: > Michael Paquier wrote: > >> From 077d675795b4907904fa4e85abed8c4528f4666f Mon Sep 17 00:00:00 2001 >> From: Michael Paquier >> Date: Sat, 19 Jul 2014 10:40:20 +0900 >> Subject: [PATCH 3/3] Buffer capture facility: check WAL replay consisten

Re: [HACKERS] WAL replay bugs

Michael Paquier wrote: > From 077d675795b4907904fa4e85abed8c4528f4666f Mon Sep 17 00:00:00 2001 > From: Michael Paquier > Date: Sat, 19 Jul 2014 10:40:20 +0900 > Subject: [PATCH 3/3] Buffer capture facility: check WAL replay consistency Is there a newer version of this tech? -- Álvaro Herrera

Re: [HACKERS] WAL replay bugs

On Thu, Nov 6, 2014 at 5:41 AM, Peter Eisentraut wrote: > On 11/4/14 10:50 PM, Michael Paquier wrote: > > Except pg_upgrade, are there other tests using bash? > > There are a few obscure things under src/test/. > Oh, I see. There is quite a number here, and each script is doing quite different t

Re: [HACKERS] WAL replay bugs

On 11/4/14 10:50 PM, Michael Paquier wrote: > Except pg_upgrade, are there other tests using bash? There are a few obscure things under src/test/. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] WAL replay bugs

Michael Paquier wrote: > Now, do we really want this feature in-core? That's somewhat a duplicate of > what is mentioned here: > http://www.postgresql.org/message-id/CAB7nPqQMq=4ejak317mxz4has0i+1rslbqu29zx18jwlb2j...@mail.gmail.com > Of course both things do not have the same coverage as the form

Re: [HACKERS] WAL replay bugs

On Wed, Nov 5, 2014 at 6:29 AM, Peter Eisentraut wrote: > On 11/4/14 3:21 PM, Alvaro Herrera wrote: > > FWIW I gave this a trial run and found I needed some tweaks to test.sh > > and the Makefile in order to make it work on VPATH; mainly replace ./ > > with `dirname $0` in a couple test.sh in a c

Re: [HACKERS] WAL replay bugs

Thanks for the tests. On Wed, Nov 5, 2014 at 5:21 AM, Alvaro Herrera wrote: > Michael Paquier wrote: > > On Mon, Jul 14, 2014 at 6:14 PM, Kyotaro HORIGUCHI > > wrote: > > > Although I doubt necessity of the flexibility seeing the current > > > testing framework, I don't have so strong objection

Re: [HACKERS] WAL replay bugs

On 11/4/14 3:21 PM, Alvaro Herrera wrote: > FWIW I gave this a trial run and found I needed some tweaks to test.sh > and the Makefile in order to make it work on VPATH; mainly replace ./ > with `dirname $0` in a couple test.sh in a couple of places, and > something similar in the Makefile. Also yo

Re: [HACKERS] WAL replay bugs

Michael Paquier wrote: > On Mon, Jul 14, 2014 at 6:14 PM, Kyotaro HORIGUCHI > wrote: > > Although I doubt necessity of the flexibility seeing the current > > testing framework, I don't have so strong objection about > > that. Nevertheless, perhaps you are appreciated to put a notice > > on.. READM

Re: [HACKERS] WAL replay bugs

Michael Paquier wrote: > On Mon, Jul 14, 2014 at 6:14 PM, Kyotaro HORIGUCHI > wrote: > > Although I doubt necessity of the flexibility seeing the current > > testing framework, I don't have so strong objection about > > that. Nevertheless, perhaps you are appreciated to put a notice > > on.. READM

Re: [HACKERS] WAL replay bugs

Hi, I'm not so confident whether it's the time to do this... I mark this as 'Ready for Committer' since no additional comment or objection was put by others on this patch. > After all, I have no more comment about this patch. I will mark > this as 'Ready for committer' unless no comment comes fr

Re: [HACKERS] WAL replay bugs

Hello, > > Although I doubt necessity of the flexibility seeing the current > > testing framework, I don't have so strong objection about > > that. Nevertheless, perhaps you are appreciated to put a notice > > on.. README or somewhere. > Hm, well... Fine, I added it in this updated series. Thank

Re: [HACKERS] WAL replay bugs

Hello, Let me apologize for continuing the discussion even though the deadline is approaching. At Fri, 11 Jul 2014 09:49:55 +0900, Michael Paquier wrote in > Updated patches attached. > > On Thu, Jul 10, 2014 at 7:13 PM, Kyotaro HORIGUCHI > wrote: > > > > The usage of this is a bit irregular

Re: [HACKERS] WAL replay bugs

Hello, The new patch looks good for me. The usage of this is a bit irregular as a (extension) module but it is the nature of this 'contrib'. The rearranged page type detection logic seems neater and keeps to work as intended. This logic will be changed after the new page type detection scheme beco

Re: [HACKERS] WAL replay bugs

Hello, At Thu, 3 Jul 2014 14:48:50 +0900, Michael Paquier wrote in > TODO ... > > Each type of page can be confirmed by the following way *if* > > its type is previously *hinted* except for gin. > > > > btree: 32bit magic at pd->opaque > > gin : No magic > > gist : 16-bit

Re: [HACKERS] WAL replay bugs

Hello, thank you for considering my comments, including somewhat impractical ones. I'll have a look at the latest patch sooner. At Fri, 4 Jul 2014 15:29:51 +0900, Michael Paquier wrote in > OK, I have been working more on this patch, improving on-the-fly the > following things on top of what

Re: [HACKERS] WAL replay bugs

Hello, This is the additional comments for other part. I haven't see significant defect in the code so far. = bufcapt.c: - buffer_capture_remember() or so. Pages for unlogged tables are avoided to be written taking advantage that the lsn for such pages stays 0/0. I'd like to see a comm

Re: [HACKERS] WAL replay bugs

On Thu, Jul 3, 2014 at 3:38 PM, Tom Lane wrote: > Michael Paquier writes: >> On Wed, Jul 2, 2014 at 5:32 PM, Kyotaro HORIGUCHI >> wrote: >>> Pehaps this is showing that no tidy or generalized way to tell >>> what a page is used for. Many of the modules which have their >>> own page format has a

Re: [HACKERS] WAL replay bugs

Michael Paquier writes: > On Wed, Jul 2, 2014 at 5:32 PM, Kyotaro HORIGUCHI > wrote: >> Pehaps this is showing that no tidy or generalized way to tell >> what a page is used for. Many of the modules which have their >> own page format has a magic value and the values seem to be >> selected carefu

Re: [HACKERS] WAL replay bugs

TODO On Wed, Jul 2, 2014 at 5:32 PM, Kyotaro HORIGUCHI wrote: > bufcapt.c: 326 memcpy(&tail, &page[BLCKSZ - 2], 2); > > This seems duzzling.. Isn't "*(uint16*)(&page[BLCKSZ - 2])" applicable? Well yes it is. > Pehaps this is showing that no tidy or generalized way to tell > what a page is

Re: [HACKERS] WAL replay bugs

Hello, > Thanks for your comments. Looking forward to seeing some more input. Thank you. bufcapt.c was a poser. bufcapt.c: 326 memcpy(&tail, &page[BLCKSZ - 2], 2); This seems duzzling.. Isn't "*(uint16*)(&page[BLCKSZ - 2])" applicable? bufcapt.c: 331 else if (PageGetSpecial Generally

Re: [HACKERS] WAL replay bugs

Hello, I had a look on this patch. Let me show you some comments about the README, Makefile and buffer_capture_cmp of the second part for the present. A continuation of this comment would be seen later.. - contrib/buffer_capture_cmp/README - 'contains' seems duplicate in the first paragraph.

Re: [HACKERS] WAL replay bugs

On Fri, Jun 27, 2014 at 2:57 PM, Michael Paquier wrote: > Here are rebased patches, their was a conflict with a recent commit in > contrib/pg_upgrade. > I am resending patch 2 as it contained a rebase conflict not correctly resolved (Thanks Alvaro). Regards, -- Michael From d4f0289ffcece54a78e5

Re: [HACKERS] WAL replay bugs

On Tue, Jun 17, 2014 at 5:40 PM, Michael Paquier wrote: > On Wed, Jun 18, 2014 at 1:40 AM, Robert Haas wrote: >> On Mon, Jun 2, 2014 at 8:55 AM, Michael Paquier >> wrote: >> I'm not sure if this is reasonably possible, but one thing that would >> make this tool a whole lot easier to use would be

Re: [HACKERS] WAL replay bugs

On Wed, Jun 18, 2014 at 1:40 AM, Robert Haas wrote: > On Mon, Jun 2, 2014 at 8:55 AM, Michael Paquier > wrote: > I'm not sure if this is reasonably possible, but one thing that would > make this tool a whole lot easier to use would be if you could make > all the magic happen in a single server.

Re: [HACKERS] WAL replay bugs

On Mon, Jun 2, 2014 at 8:55 AM, Michael Paquier wrote: > On Wed, Apr 23, 2014 at 9:43 PM, Heikki Linnakangas > wrote: >> And here is the tool itself. It consists of two parts: >> >> 1. Modifications to the backend to write the page images >> 2. A post-processing tool to compare the logged images

Re: [HACKERS] WAL replay bugs

On Fri, Jun 13, 2014 at 4:50 PM, Michael Paquier wrote: > On Fri, Jun 13, 2014 at 4:48 PM, Heikki Linnakangas > wrote: >> On 06/13/2014 10:14 AM, Michael Paquier wrote: >>> >>> On Mon, Jun 2, 2014 at 9:55 PM, Michael Paquier >>> wrote: On Wed, Apr 23, 2014 at 9:43 PM, Heikki Linnakanga

Re: [HACKERS] WAL replay bugs

On Fri, Jun 13, 2014 at 4:48 PM, Heikki Linnakangas wrote: > On 06/13/2014 10:14 AM, Michael Paquier wrote: >> >> On Mon, Jun 2, 2014 at 9:55 PM, Michael Paquier >> wrote: >>> >>> On Wed, Apr 23, 2014 at 9:43 PM, Heikki Linnakangas >>> wrote: >>> Perhaps there are parts of what is proposed here

Re: [HACKERS] WAL replay bugs

On 06/13/2014 10:14 AM, Michael Paquier wrote: On Mon, Jun 2, 2014 at 9:55 PM, Michael Paquier wrote: On Wed, Apr 23, 2014 at 9:43 PM, Heikki Linnakangas wrote: Perhaps there are parts of what is proposed here that could be made more generalized, like the masking functions. So do not hesitate

Re: [HACKERS] WAL replay bugs

On Wed, Apr 23, 2014 at 9:43 PM, Heikki Linnakangas wrote: > And here is the tool itself. It consists of two parts: > > 1. Modifications to the backend to write the page images > 2. A post-processing tool to compare the logged images between master and > standby. Having that into Postgres at the d

Re: [HACKERS] WAL replay bugs

On 04/17/2014 07:59 PM, Heikki Linnakangas wrote: On 04/08/2014 06:41 AM, Michael Paquier wrote: On Tue, Apr 8, 2014 at 3:16 AM, Heikki Linnakangas wrote: I've been playing with a little hack that records a before and after image of every page modification that is WAL-logged, and writes the i

Re: [HACKERS] WAL replay bugs

On Thu, Apr 17, 2014 at 10:33 AM, Tom Lane wrote: >> Any objections to changing those two? > > Not here. I've always suspected #2 was going to bite us someday anyway. +1 -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscri

Re: [HACKERS] WAL replay bugs

Heikki Linnakangas writes: > Two things that are not bugs, but I'd like to change just to make this > tool easier to maintain, and to generally clean things up: > 1. When creating a sequence, we first use simple_heap_insert() to insert > the sequence tuple, which creates a WAL record. Then we w

Re: [HACKERS] WAL replay bugs

On 04/08/2014 06:41 AM, Michael Paquier wrote: On Tue, Apr 8, 2014 at 3:16 AM, Heikki Linnakangas wrote: I've been playing with a little hack that records a before and after image of every page modification that is WAL-logged, and writes the images to a file along with the LSN of the correspon

Re: [HACKERS] WAL replay bugs

On Thu, Apr 10, 2014 at 6:21 PM, Heikki Linnakangas wrote: > On 04/10/2014 10:52 AM, sachin kotwal wrote: > >> >> I executed given steps many times to produce this bug. >> But still I unable to hit this bug. >> I used attached scripts to produce this bug. >> >> Can I get scripts to produce this

Re: [HACKERS] WAL replay bugs

On 04/10/2014 10:52 AM, sachin kotwal wrote: I executed given steps many times to produce this bug. But still I unable to hit this bug. I used attached scripts to produce this bug. Can I get scripts to produce this bug? wal_replay_bug.sh

Re: [HACKERS] WAL replay bugs

I executed given steps many times to produce this bug. But still I unable to hit this bug. I used attached scripts to produce this bug. Can I get scripts to produce this bug? wal_replay_bug.sh - Thanks and Regar

Re: [HACKERS] WAL replay bugs

On Tue, Apr 8, 2014 at 3:16 AM, Heikki Linnakangas wrote: > > I've been playing with a little hack that records a before and after image > of every page modification that is WAL-logged, and writes the images to a > file along with the LSN of the corresponding WAL record. I set up a > master-standb

Re: [HACKERS] WAL replay bugs

On 04/07/2014 02:16 PM, Heikki Linnakangas wrote: > I've been playing with a little hack that records a before and after > image of every page modification that is WAL-logged, and writes the > images to a file along with the LSN of the corresponding WAL record. I > set up a master-standby replicati