Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Wed, Mar 28, 2012 at 10:08 AM, Fujii Masao wrote: > On Wed, Mar 28, 2012 at 12:30 AM, Alvaro Herrera > wrote: >> >> Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012: >> >>> Anyway, should I add this patch into the next CF? Or is anyone planning >>> to commit the patch for 9.2? >> >> I think the correct thing to do here is add to next CF, and if some >> committer has enough interest in getting it quickly it can be grabbed >> from there and committed into 9.2. > > Yep! I've added it to next CF. Attached is the revised version of the patch. Regards, -- Fujii Masao pg_controldata_walfilename_v4.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Wed, Mar 28, 2012 at 12:30 AM, Alvaro Herrera wrote: > > Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012: > >> Anyway, should I add this patch into the next CF? Or is anyone planning >> to commit the patch for 9.2? > > I think the correct thing to do here is add to next CF, and if some > committer has enough interest in getting it quickly it can be grabbed > from there and committed into 9.2. Yep! I've added it to next CF. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
Excerpts from Fujii Masao's message of mar mar 27 06:40:34 -0300 2012: > Anyway, should I add this patch into the next CF? Or is anyone planning > to commit the patch for 9.2? I think the correct thing to do here is add to next CF, and if some committer has enough interest in getting it quickly it can be grabbed from there and committed into 9.2. -- Álvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Mon, Mar 26, 2012 at 11:18 PM, Tom Lane wrote:
> Robert Haas writes:
>> On Mon, Mar 26, 2012 at 2:50 AM, Magnus Hagander wrote:
s/segment/file/g?
>
>>> We're already using "file" to mean something different *internally*,
>>> don't we? And since pg_controldata shows fairly internal information,
>>> I'm not sure this is the best idea.
>>>
>>> Maybe compromise and call it "segment file" - that is both easier to
>>> understand than segment, and not actually using a term that means
>>> something else...
>
>> It's also kind of wordy. I think "file" is fine.
>
> +1 for "file". I think the internal usage of "file" to mean "roughly
> 4GB worth of WAL" is going to go away soon anyway, as there won't be
> much reason to worry about the concept once LSN arithmetic is 64-bit.
Agreed. This would mean that the following lots of log messages need to
be changed after 64-bit LSN will have been committed.
errmsg("could not fdatasync log file %u, segment %u: %m",
log, seg)));
Anyway, should I add this patch into the next CF? Or is anyone planning
to commit the patch for 9.2?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
Robert Haas writes: > On Mon, Mar 26, 2012 at 2:50 AM, Magnus Hagander wrote: >>> s/segment/file/g? >> We're already using "file" to mean something different *internally*, >> don't we? And since pg_controldata shows fairly internal information, >> I'm not sure this is the best idea. >> >> Maybe compromise and call it "segment file" - that is both easier to >> understand than segment, and not actually using a term that means >> something else... > It's also kind of wordy. I think "file" is fine. +1 for "file". I think the internal usage of "file" to mean "roughly 4GB worth of WAL" is going to go away soon anyway, as there won't be much reason to worry about the concept once LSN arithmetic is 64-bit. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Mon, Mar 26, 2012 at 2:50 AM, Magnus Hagander wrote: >>> s/segment/file/g? >> >> Yep, "file" might be more intuitive for a user than "segment". Attached is >> the >> "file" version of the patch. > > We're already using "file" to mean something different *internally*, > don't we? And since pg_controldata shows fairly internal information, > I'm not sure this is the best idea. > > Maybe compromise and call it "segment file" - that is both easier to > understand than segment, and not actually using a term that means > something else... It's also kind of wordy. I think "file" is fine. There are a few references to xlogid indicating a "file number", but the actual field name is just xlogid. We also use the term "file" to mean the other thing, as in XLOGfileslop, and I have a hard time believing anyone's really going to get confused about what is meant here. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 18:05, Fujii Masao wrote: > On Sat, Mar 24, 2012 at 1:49 AM, Robert Haas wrote: >> On Fri, Mar 23, 2012 at 12:42 PM, Fujii Masao wrote: >>> On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas wrote: On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao wrote: > On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander > wrote: >> Might it be a good idea to put it on it's own row instead of changing >> the format of an existing row, in order not to break scripts and >> programs that are parsing the previous output? > > Good idea! What row name should we use for the WAL file containing > REDO record? "Latest checkpoint's REDO file"? Sounds good to me. I like the idea, too. The status quo is an unnecessary nuisance, so this will be a nice usability improvement. >>> >>> Attached patch adds new row "Latest checkpoint's REDO WAL segment:" into >>> the result of pg_controldata. I used the term "WAL segment" for the row name >>> instead of "file" because "WAL segment" is used in another row "Bytes per >>> WAL >>> segment:". But better name? >> >> s/segment/file/g? > > Yep, "file" might be more intuitive for a user than "segment". Attached is the > "file" version of the patch. We're already using "file" to mean something different *internally*, don't we? And since pg_controldata shows fairly internal information, I'm not sure this is the best idea. Maybe compromise and call it "segment file" - that is both easier to understand than segment, and not actually using a term that means something else... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Sat, Mar 24, 2012 at 1:49 AM, Robert Haas wrote: > On Fri, Mar 23, 2012 at 12:42 PM, Fujii Masao wrote: >> On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas wrote: >>> On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao wrote: On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander wrote: > Might it be a good idea to put it on it's own row instead of changing > the format of an existing row, in order not to break scripts and > programs that are parsing the previous output? Good idea! What row name should we use for the WAL file containing REDO record? "Latest checkpoint's REDO file"? >>> >>> Sounds good to me. I like the idea, too. The status quo is an >>> unnecessary nuisance, so this will be a nice usability improvement. >> >> Attached patch adds new row "Latest checkpoint's REDO WAL segment:" into >> the result of pg_controldata. I used the term "WAL segment" for the row name >> instead of "file" because "WAL segment" is used in another row "Bytes per WAL >> segment:". But better name? > > s/segment/file/g? Yep, "file" might be more intuitive for a user than "segment". Attached is the "file" version of the patch. Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center pg_controldata_walfilename_v3.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 12:42 PM, Fujii Masao wrote: > On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas wrote: >> On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao wrote: >>> On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander >>> wrote: Might it be a good idea to put it on it's own row instead of changing the format of an existing row, in order not to break scripts and programs that are parsing the previous output? >>> >>> Good idea! What row name should we use for the WAL file containing >>> REDO record? "Latest checkpoint's REDO file"? >> >> Sounds good to me. I like the idea, too. The status quo is an >> unnecessary nuisance, so this will be a nice usability improvement. > > Attached patch adds new row "Latest checkpoint's REDO WAL segment:" into > the result of pg_controldata. I used the term "WAL segment" for the row name > instead of "file" because "WAL segment" is used in another row "Bytes per WAL > segment:". But better name? s/segment/file/g? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 9:41 PM, Robert Haas wrote: > On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao wrote: >> On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander wrote: >>> Might it be a good idea to put it on it's own row instead of changing >>> the format of an existing row, in order not to break scripts and >>> programs that are parsing the previous output? >> >> Good idea! What row name should we use for the WAL file containing >> REDO record? "Latest checkpoint's REDO file"? > > Sounds good to me. I like the idea, too. The status quo is an > unnecessary nuisance, so this will be a nice usability improvement. Attached patch adds new row "Latest checkpoint's REDO WAL segment:" into the result of pg_controldata. I used the term "WAL segment" for the row name instead of "file" because "WAL segment" is used in another row "Bytes per WAL segment:". But better name? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center pg_controldata_walfilename_v2.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 6:13 AM, Fujii Masao wrote: > On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander wrote: >> Might it be a good idea to put it on it's own row instead of changing >> the format of an existing row, in order not to break scripts and >> programs that are parsing the previous output? > > Good idea! What row name should we use for the WAL file containing > REDO record? "Latest checkpoint's REDO file"? Sounds good to me. I like the idea, too. The status quo is an unnecessary nuisance, so this will be a nice usability improvement. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 5:56 PM, Magnus Hagander wrote: > Might it be a good idea to put it on it's own row instead of changing > the format of an existing row, in order not to break scripts and > programs that are parsing the previous output? Good idea! What row name should we use for the WAL file containing REDO record? "Latest checkpoint's REDO file"? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 10:51, Fujii Masao wrote: > On Fri, Mar 23, 2012 at 2:06 PM, Jaime Casanova wrote: >> On Thu, Mar 22, 2012 at 11:06 PM, Fujii Masao wrote: >>> >>> We can use >>> pg_xlogfile_name function to calculate that, but it cannot be executed in >>> the standby. Another problem is that pg_xlogfile_name always uses >>> current timeline for the calculation, so if the reported timeline is not >>> the same as current one, pg_xlogfile_name cannot return the correct WAL >>> file name. Making pg_controldata report that WAL file name gets rid of >>> such a complexity. >>> >> >> i would think that pg_xlogfile_name() is not allowed in the standby >> because ThisTimelineId is not very well defined in recovery but if you >> extend pg_xlogfile_name() to also receive a timelineid as you >> suggested in [1] then that version of the function could be allowed in >> the standby. >> or there is something else i'm missing? >> >> is that enough for you to solve your problem? > > Yes, we can more easily calculate the cutoff point by using that extended > pg_xlogfile_name(). But if pg_controldata reports the WAL file name, we > can calculate the cutoff point without starting the server. So I think that > it's worth changing pg_controldata that way even if we extend > pg_xlogfile_name(). +1 - I think they're both useful things, each on it's own. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 2:06 PM, Jaime Casanova wrote: > On Thu, Mar 22, 2012 at 11:06 PM, Fujii Masao wrote: >> >> We can use >> pg_xlogfile_name function to calculate that, but it cannot be executed in >> the standby. Another problem is that pg_xlogfile_name always uses >> current timeline for the calculation, so if the reported timeline is not >> the same as current one, pg_xlogfile_name cannot return the correct WAL >> file name. Making pg_controldata report that WAL file name gets rid of >> such a complexity. >> > > i would think that pg_xlogfile_name() is not allowed in the standby > because ThisTimelineId is not very well defined in recovery but if you > extend pg_xlogfile_name() to also receive a timelineid as you > suggested in [1] then that version of the function could be allowed in > the standby. > or there is something else i'm missing? > > is that enough for you to solve your problem? Yes, we can more easily calculate the cutoff point by using that extended pg_xlogfile_name(). But if pg_controldata reports the WAL file name, we can calculate the cutoff point without starting the server. So I think that it's worth changing pg_controldata that way even if we extend pg_xlogfile_name(). Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Fri, Mar 23, 2012 at 05:06, Fujii Masao wrote: > Hi, > > I'd like to propose to change pg_controldata so that it reports the name > of WAL file containing the latest checkpoint's REDO record, as follows: > > $ pg_controldata $PGDATA > ... > Latest checkpoint's REDO location: 0/16D6ACC (file > 00010001) > Latest checkpoint's TimeLineID: 1 > ... > > This simplifies very much the way to calculate the archive file cutoff point > because the reported WAL file is just cutoff point itself. If the file name is > not reported, we have to calculate the cutoff point from the reported > location and timeline, which is complicated calculation. We can use > pg_xlogfile_name function to calculate that, but it cannot be executed in > the standby. Another problem is that pg_xlogfile_name always uses > current timeline for the calculation, so if the reported timeline is not > the same as current one, pg_xlogfile_name cannot return the correct WAL > file name. Making pg_controldata report that WAL file name gets rid of > such a complexity. > > You may think that archive_cleanup_command is usable for that purpose. > That's true. But it's not usable simply for the case where there are more > than one standby servers. In this case, the archive file cutoff point needs > to be calculated from each standby's REDO location and timeline. > > Attached patch changes pg_controldata as above. Thought? Might it be a good idea to put it on it's own row instead of changing the format of an existing row, in order not to break scripts and programs that are parsing the previous output? -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
On Thu, Mar 22, 2012 at 11:06 PM, Fujii Masao wrote: > > We can use > pg_xlogfile_name function to calculate that, but it cannot be executed in > the standby. Another problem is that pg_xlogfile_name always uses > current timeline for the calculation, so if the reported timeline is not > the same as current one, pg_xlogfile_name cannot return the correct WAL > file name. Making pg_controldata report that WAL file name gets rid of > such a complexity. > i would think that pg_xlogfile_name() is not allowed in the standby because ThisTimelineId is not very well defined in recovery but if you extend pg_xlogfile_name() to also receive a timelineid as you suggested in [1] then that version of the function could be allowed in the standby. or there is something else i'm missing? is that enough for you to solve your problem? [1] http://archives.postgresql.org/message-id/cahgqgwhwqjgeksmp2oteru8p0sj6x7ypqyh5qqeqcbqsxaa...@mail.gmail.com -- Jaime Casanova www.2ndQuadrant.com Professional PostgreSQL: Soporte 24x7 y capacitación -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Reporting WAL file containing checkpoint's REDO record in pg_controldata's result
Hi,
I'd like to propose to change pg_controldata so that it reports the name
of WAL file containing the latest checkpoint's REDO record, as follows:
$ pg_controldata $PGDATA
...
Latest checkpoint's REDO location:0/16D6ACC (file
00010001)
Latest checkpoint's TimeLineID: 1
...
This simplifies very much the way to calculate the archive file cutoff point
because the reported WAL file is just cutoff point itself. If the file name is
not reported, we have to calculate the cutoff point from the reported
location and timeline, which is complicated calculation. We can use
pg_xlogfile_name function to calculate that, but it cannot be executed in
the standby. Another problem is that pg_xlogfile_name always uses
current timeline for the calculation, so if the reported timeline is not
the same as current one, pg_xlogfile_name cannot return the correct WAL
file name. Making pg_controldata report that WAL file name gets rid of
such a complexity.
You may think that archive_cleanup_command is usable for that purpose.
That's true. But it's not usable simply for the case where there are more
than one standby servers. In this case, the archive file cutoff point needs
to be calculated from each standby's REDO location and timeline.
Attached patch changes pg_controldata as above. Thought?
Regards,
--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center
*** a/src/bin/pg_controldata/pg_controldata.c
--- b/src/bin/pg_controldata/pg_controldata.c
***
*** 24,29
--- 24,30
#include
#include "access/xlog.h"
+ #include "access/xlog_internal.h"
#include "catalog/pg_control.h"
***
*** 101,106 main(int argc, char *argv[])
--- 102,110
char sysident_str[32];
const char *strftime_fmt = "%c";
const char *progname;
+ uint32 log;
+ uint32 seg;
+ char xlogfilename[MAXFNAMELEN];
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_controldata"));
***
*** 177,182 main(int argc, char *argv[])
--- 181,193
localtime(&time_tmp));
/*
+ * Calculate the WAL file name containing the latest checkpoint's REDO
+ * record.
+ */
+ XLByteToSeg(ControlFile.checkPointCopy.redo, log, seg);
+ XLogFileName(xlogfilename, ControlFile.checkPointCopy.ThisTimeLineID, log, seg);
+
+ /*
* Format system_identifier separately to keep platform-dependent format
* code out of the translatable message string.
*/
***
*** 204,212 main(int argc, char *argv[])
printf(_("Prior checkpoint location:%X/%X\n"),
ControlFile.prevCheckPoint.xlogid,
ControlFile.prevCheckPoint.xrecoff);
! printf(_("Latest checkpoint's REDO location:%X/%X\n"),
ControlFile.checkPointCopy.redo.xlogid,
! ControlFile.checkPointCopy.redo.xrecoff);
printf(_("Latest checkpoint's TimeLineID: %u\n"),
ControlFile.checkPointCopy.ThisTimeLineID);
printf(_("Latest checkpoint's full_page_writes: %s\n"),
--- 215,224
printf(_("Prior checkpoint location:%X/%X\n"),
ControlFile.prevCheckPoint.xlogid,
ControlFile.prevCheckPoint.xrecoff);
! printf(_("Latest checkpoint's REDO location:%X/%X (file %s)\n"),
ControlFile.checkPointCopy.redo.xlogid,
! ControlFile.checkPointCopy.redo.xrecoff,
! xlogfilename);
printf(_("Latest checkpoint's TimeLineID: %u\n"),
ControlFile.checkPointCopy.ThisTimeLineID);
printf(_("Latest checkpoint's full_page_writes: %s\n"),
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
