Re: [HACKERS] Re: [ANNOUNCE] Bug-fix and new feature of pg_lesslog is released
Thanks a lot for the comment/advice. Yes, full page backup block considerablly shortens the recovery time. As we discussed about two years ago, I have a solution accelerate the recovery even without full page image. I'd like to submit this solution to the community again. When I evaluated this two years ago, recovery speed was as good as those with full page image, depending upon application and tuning, of course. This is a separate tool and can be used in various scenes. Regards; -- Koichi Suzuki 2010/5/13 Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp: Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Yes, I would love to get this into /contrib for PG 9.1! How much are people really going to care about pg_lesslog now that we've got streaming replication? There might be some small use-case still left, but it's hard to believe that it would be worth carrying it in contrib. I hope pg_lesslog would work as a WAL filter of streaming replication. It might be hard-coded in WAL sender, or be an addon based on a new common filtering infrastructure of WAL streaming. Also, there is a long-standing issue in pg_lesslog; It slows down recovery because we need to read data pages before write in recovery. We're avoiding reading pages for full-page image in 8.3, but pg_lesslog will disable the optimization. Recovery routine in core also needs to be adjusted to use read-ahead, like posix_fadvise(). There was another idea, full-page image logs separated with WAL logging. In theory, full-page images don't have to be written at commit, but only by writing corresponding data pages, I'm not sure whether it is an actually good idea or not, but if we go the direction, we won't need pg_lesslog. Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [ANNOUNCE] Bug-fix and new feature of pg_lesslog is released
WAL streming filter is an interesting idea. We can leave full page backup for local recovery and decrease the amount of WAL to transfer. If SR writes WAL in record by record basis, implementation will be simple. If SR writes WAL in block by block basis, WAL records may be divided from transfer to transfer and it is not simple to handle. Regards; -- Koichi Suzuki 2010/5/13 Takahiro Itagaki itagaki.takah...@oss.ntt.co.jp: Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Yes, I would love to get this into /contrib for PG 9.1! How much are people really going to care about pg_lesslog now that we've got streaming replication? There might be some small use-case still left, but it's hard to believe that it would be worth carrying it in contrib. I hope pg_lesslog would work as a WAL filter of streaming replication. It might be hard-coded in WAL sender, or be an addon based on a new common filtering infrastructure of WAL streaming. Also, there is a long-standing issue in pg_lesslog; It slows down recovery because we need to read data pages before write in recovery. We're avoiding reading pages for full-page image in 8.3, but pg_lesslog will disable the optimization. Recovery routine in core also needs to be adjusted to use read-ahead, like posix_fadvise(). There was another idea, full-page image logs separated with WAL logging. In theory, full-page images don't have to be written at commit, but only by writing corresponding data pages, I'm not sure whether it is an actually good idea or not, but if we go the direction, we won't need pg_lesslog. Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [ANNOUNCE] Bug-fix and new feature of pg_lesslog is released
Koichi Suzuki wrote: It is an announcement that long-waited bug-fix of pg_lesslog is now released. This includes the following. 1. Error in calculation of GiST-related WAL record length was fixed. 2. pg_compresslog now has an option to print WAL segment analysis. 3. New test script is added which analyzes what WAL record was generated and if all the recovery was done successfully. 2 and 3 are added to ensure the tools run correctly. Because 8.4 has additional WAL records, I made release material separate for each PG version. You may visit the project page http://pgfoundry.org/frs/?group_id=1000310 to download materials. I acknowledge PG hackers who gave valuable comments/advices. Especially Oleg Bartunov, Teodor Sigaev and Alvaro Herrera gave very helpful information how to generate specific type of WAL records. As Bruce suggested, this can be included in PostgreSQL test suits and could be included in contrib module if it is ported to PostgreSQL 9.1. [ email to hackers list only ] Yes, I would love to get this into /contrib for PG 9.1! -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [ANNOUNCE] Bug-fix and new feature of pg_lesslog is released
Bruce Momjian br...@momjian.us writes: Yes, I would love to get this into /contrib for PG 9.1! How much are people really going to care about pg_lesslog now that we've got streaming replication? There might be some small use-case still left, but it's hard to believe that it would be worth carrying it in contrib. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [ANNOUNCE] Bug-fix and new feature of pg_lesslog is released
On Wed, May 12, 2010 at 11:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Yes, I would love to get this into /contrib for PG 9.1! How much are people really going to care about pg_lesslog now that we've got streaming replication? There might be some small use-case still left, but it's hard to believe that it would be worth carrying it in contrib. Until we have a tool to use streaming replication to actually archive backups, there's still a fairly large usecase for it... -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [ANNOUNCE] Bug-fix and new feature of pg_lesslog is released
On Wed, 2010-05-12 at 23:09 +0200, Magnus Hagander wrote: On Wed, May 12, 2010 at 11:03 PM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Yes, I would love to get this into /contrib for PG 9.1! How much are people really going to care about pg_lesslog now that we've got streaming replication? There might be some small use-case still left, but it's hard to believe that it would be worth carrying it in contrib. Until we have a tool to use streaming replication to actually archive backups, there's still a fairly large usecase for it... Not to mention, WAN based standby, timed standby etc... Joshua D. Drake -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [ANNOUNCE] Bug-fix and new feature of pg_lesslog is released
Until we have a tool to use streaming replication to actually archive backups, there's still a fairly large usecase for it... yes, pglesslog is for PITR. We have a client who keeps 3 days of log files a time for forensic reasons, and I doubt they're the only one. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Re: [ANNOUNCE] Bug-fix and new feature of pg_lesslog is released
Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Yes, I would love to get this into /contrib for PG 9.1! How much are people really going to care about pg_lesslog now that we've got streaming replication? There might be some small use-case still left, but it's hard to believe that it would be worth carrying it in contrib. I hope pg_lesslog would work as a WAL filter of streaming replication. It might be hard-coded in WAL sender, or be an addon based on a new common filtering infrastructure of WAL streaming. Also, there is a long-standing issue in pg_lesslog; It slows down recovery because we need to read data pages before write in recovery. We're avoiding reading pages for full-page image in 8.3, but pg_lesslog will disable the optimization. Recovery routine in core also needs to be adjusted to use read-ahead, like posix_fadvise(). There was another idea, full-page image logs separated with WAL logging. In theory, full-page images don't have to be written at commit, but only by writing corresponding data pages, I'm not sure whether it is an actually good idea or not, but if we go the direction, we won't need pg_lesslog. Regards, --- Takahiro Itagaki NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
