Re: [HACKERS] XLog changes for 9.3

2012-06-12 Thread Bruce Momjian
On Thu, Jun 07, 2012 at 02:52:04PM -0400, Tom Lane wrote: > > So you can pretty much kiss goodbye to the idea of pg_upgrade. > > And that is certainly nonsense. I don't think pg_upgrade even knows > about this, and if it does we can surely fix it. pg_upgrade doesn't know anything about xlog file

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Simon Riggs writes: > On 7 June 2012 19:52, Tom Lane wrote: >> This seems like nonsense to me.  How many external scripts are likely to >> know that we skip the FF page?  There might be some, but not many. > If that is the only change in filenames, then all is forgiven. Oh, now I see what you'r

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Simon Riggs
On 7 June 2012 19:52, Tom Lane wrote: > Simon Riggs writes: >> Anything changing filenames will break every HA config anybody has >> anywhere. > > This seems like nonsense to me.  How many external scripts are likely to > know that we skip the FF page?  There might be some, but not many. If that

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Simon Riggs writes: > Anything changing filenames will break every HA config anybody has > anywhere. This seems like nonsense to me. How many external scripts are likely to know that we skip the FF page? There might be some, but not many. > So you can pretty much kiss goodbye to the idea of pg

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 1:15 PM, Kevin Grittner wrote: > Simon Riggs wrote: > >> Anything changing filenames will break every HA config anybody has >> anywhere. > > It will impact our scripts related to backup and archiving, but I > think we're talking about two or three staff days to cover it in

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 1:40 PM, Kevin Grittner wrote: > Robert Haas wrote: > >> But if you're just using regexp matching against pathnames, your >> tool will be just fine.  Do your tools actually rely on the >> occasional absence of a file in what would otherwise be the usual >> sequence of files

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Kevin Grittner
Robert Haas wrote: > But if you're just using regexp matching against pathnames, your > tool will be just fine. Do your tools actually rely on the > occasional absence of a file in what would otherwise be the usual > sequence of files? To save "snapshot" backups for the long term, we generate

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Kevin Grittner
Simon Riggs wrote: > Anything changing filenames will break every HA config anybody has > anywhere. It will impact our scripts related to backup and archiving, but I think we're talking about two or three staff days to cover it in our shop. We should definitely make sure that this change is

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 07:03:32 PM Simon Riggs wrote: > On 7 June 2012 17:12, Heikki Linnakangas > > wrote: > > On 07.06.2012 18:51, Simon Riggs wrote: > >> On 7 June 2012 14:50, Heikki Linnakangas > >> > >> wrote: > >>> These changes will help the XLogInsert scaling patch > >> > >> ...an

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Simon Riggs
On 7 June 2012 17:12, Heikki Linnakangas wrote: > On 07.06.2012 18:51, Simon Riggs wrote: >> >> On 7 June 2012 14:50, Heikki Linnakangas >>  wrote: >> >>> These changes will help the XLogInsert scaling patch >> >> >> ...and as I'm sure you're aware will junk much of the replication code >> and al

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:53:58 PM Tom Lane wrote: > Andres Freund writes: > > dance. If the record can be smeared over two pages there is no point in > > storing it aligned. > > I think this is not true. The value of requiring alignment is that you > can read the record-length field without

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Andres Freund writes: > dance. If the record can be smeared over two pages there is no point in > storing it aligned. I think this is not true. The value of requiring alignment is that you can read the record-length field without first having to copy it somewhere. In particular, it will get rea

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 05:35:11 PM Heikki Linnakangas wrote: > On 07.06.2012 17:18, Andres Freund wrote: > > On Thursday, June 07, 2012 03:50:35 PM Heikki Linnakangas wrote: > >> 3. Move the only field, xl_rem_len, from the continuation record header > >> straight to the xlog page header, elimi

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Robert Haas
On Thu, Jun 7, 2012 at 11:51 AM, Simon Riggs wrote: > So this is a very large curve ball you're throwing there. This is not exactly unexpected. At least the first two of these items were previous discussed in the context of the XLOG scaling patch, many months ago. It shouldn't come as a surpris

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Heikki Linnakangas
On 07.06.2012 18:51, Simon Riggs wrote: On 7 June 2012 14:50, Heikki Linnakangas wrote: These changes will help the XLogInsert scaling patch ...and as I'm sure you're aware will junk much of the replication code and almost certainly set back the other work that we have brewing for 9.3. So t

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 06:02:12 PM Magnus Hagander wrote: > On Thu, Jun 7, 2012 at 5:56 PM, Andres Freund wrote: > > Hi, > > > > On Thursday, June 07, 2012 05:51:07 PM Simon Riggs wrote: > >> On 7 June 2012 14:50, Heikki Linnakangas > >> > >> wrote: > >> > These changes will help the XLogI

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Magnus Hagander
On Thu, Jun 7, 2012 at 5:56 PM, Andres Freund wrote: > Hi, > > On Thursday, June 07, 2012 05:51:07 PM Simon Riggs wrote: >> On 7 June 2012 14:50, Heikki Linnakangas >> >> wrote: >> > These changes will help the XLogInsert scaling patch >> >> ...and as I'm sure you're aware will junk much of the r

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
Hi, On Thursday, June 07, 2012 05:51:07 PM Simon Riggs wrote: > On 7 June 2012 14:50, Heikki Linnakangas > > wrote: > > These changes will help the XLogInsert scaling patch > > ...and as I'm sure you're aware will junk much of the replication code > and almost certainly set back the other work

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Simon Riggs
On 7 June 2012 14:50, Heikki Linnakangas wrote: > These changes will help the XLogInsert scaling patch ...and as I'm sure you're aware will junk much of the replication code and almost certainly set back the other work that we have brewing for 9.3. So this is a very large curve ball you're throw

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Heikki Linnakangas
On 07.06.2012 17:18, Andres Freund wrote: On Thursday, June 07, 2012 03:50:35 PM Heikki Linnakangas wrote: 3. Move the only field, xl_rem_len, from the continuation record header straight to the xlog page header, eliminating XLogContRecord altogether. This makes it easier to calculate in advance

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Tom Lane
Heikki Linnakangas writes: > When I worked on the XLogInsert scaling patch, it became apparent that > some changes to the WAL format would make it a lot easier. So for 9.3, > I'd like to do some refactoring: > 1. Use a 64-bit integer instead of the two-variable log/seg > representation, for id

Re: [HACKERS] XLog changes for 9.3

2012-06-07 Thread Andres Freund
On Thursday, June 07, 2012 03:50:35 PM Heikki Linnakangas wrote: > When I worked on the XLogInsert scaling patch, it became apparent that > some changes to the WAL format would make it a lot easier. So for 9.3, > I'd like to do some refactoring: > 1. Use a 64-bit integer instead of the two-variabl

[HACKERS] XLog changes for 9.3

2012-06-07 Thread Heikki Linnakangas
When I worked on the XLogInsert scaling patch, it became apparent that some changes to the WAL format would make it a lot easier. So for 9.3, I'd like to do some refactoring: 1. Use a 64-bit integer instead of the two-variable log/seg representation, for identifying a WAL segment. This has no