Re: [HACKERS] Freeze avoidance of very large table.

2016-03-11 Thread Joshua D. Drake
On 03/11/2016 09:48 AM, Masahiko Sawada wrote: Thank you so much! What I wanted deal with in thread is almost done. I'm going to more test the feature for 9.6 releasing. Nicely done! Regards, -- Masahiko Sawada -- Command Prompt, Inc. http://the.postgres.company/

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-11 Thread Masahiko Sawada
On Sat, Mar 12, 2016 at 2:37 AM, Robert Haas wrote: > On Thu, Mar 10, 2016 at 10:47 PM, Masahiko Sawada > wrote: >>> Thanks. I adopted some of your suggested, rejected others, fixed a >>> few minor things that I missed previously, and committed this. If you >>> think any of the changes that I

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-11 Thread Robert Haas
On Thu, Mar 10, 2016 at 10:47 PM, Masahiko Sawada wrote: >> Thanks. I adopted some of your suggested, rejected others, fixed a >> few minor things that I missed previously, and committed this. If you >> think any of the changes that I rejected still have merit, please >> resubmit those changes a

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Masahiko Sawada
On Fri, Mar 11, 2016 at 6:16 AM, Robert Haas wrote: > On Thu, Mar 10, 2016 at 1:41 PM, Masahiko Sawada > wrote: >> On Fri, Mar 11, 2016 at 1:03 AM, Robert Haas wrote: >>> This 001 patch looks so little like what I was expecting that I >>> decided to start over from scratch. The new version I w

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Robert Haas
On Thu, Mar 10, 2016 at 1:41 PM, Masahiko Sawada wrote: > On Fri, Mar 11, 2016 at 1:03 AM, Robert Haas wrote: >> This 001 patch looks so little like what I was expecting that I >> decided to start over from scratch. The new version I wrote is >> attached here. I don't understand why your versio

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Robert Haas
On Thu, Mar 10, 2016 at 1:41 PM, Masahiko Sawada wrote: > On Fri, Mar 11, 2016 at 1:03 AM, Robert Haas wrote: >> This 001 patch looks so little like what I was expecting that I >> decided to start over from scratch. The new version I wrote is >> attached here. I don't understand why your versio

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Masahiko Sawada
On Fri, Mar 11, 2016 at 1:03 AM, Robert Haas wrote: > This 001 patch looks so little like what I was expecting that I > decided to start over from scratch. The new version I wrote is > attached here. I don't understand why your version tinkers with the > logic for setting the all-frozen bit; I t

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Robert Haas
On Thu, Mar 10, 2016 at 8:51 AM, Masahiko Sawada wrote: > After some further thought, I thought that it's better to add check > logic for result of rewriting visibility map to upgrading logic rather > than regression test in order to ensure that rewriting visibility map > has been successfully don

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Masahiko Sawada
On Thu, Mar 10, 2016 at 3:27 PM, Masahiko Sawada wrote: > Thank you for reviewing! > Attached updated patch. > > > On Thu, Mar 10, 2016 at 3:37 AM, Robert Haas wrote: >> On Wed, Mar 9, 2016 at 9:09 AM, Masahiko Sawada >> wrote: Attached latest 2 patches. >>> * 000 patch : Incorporated the review

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-10 Thread Robert Haas
On Wed, Mar 9, 2016 at 9:09 AM, Masahiko Sawada wrote: > * 001 patch : Incorporated the documentation suggestions and updated > logic a little. This 001 patch looks so little like what I was expecting that I decided to start over from scratch. The new version I wrote is attached here. I don't u

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-09 Thread Masahiko Sawada
Thank you for reviewing! Attached updated patch. On Thu, Mar 10, 2016 at 3:37 AM, Robert Haas wrote: > On Wed, Mar 9, 2016 at 9:09 AM, Masahiko Sawada > wrote: Attached latest 2 patches. >> * 000 patch : Incorporated the review comments and made rewriting >> logic more clearly. > > That's bette

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-09 Thread Robert Haas
On Wed, Mar 9, 2016 at 9:09 AM, Masahiko Sawada wrote: Attached latest 2 patches. > * 000 patch : Incorporated the review comments and made rewriting > logic more clearly. That's better, thanks. But your comments don't survive pgindent. After running pgindent, I get this: + /* +

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-09 Thread Masahiko Sawada
On Wed, Mar 9, 2016 at 3:38 AM, Robert Haas wrote: > On Tue, Mar 8, 2016 at 12:59 PM, Masahiko Sawada > wrote: >>> How about instead changing things so that we specifically reject >>> indexes? And maybe some kind of a check that will reject anything >>> that lacks a relfilnode? That seems like

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Robert Haas
On Tue, Mar 8, 2016 at 12:59 PM, Masahiko Sawada wrote: >> How about instead changing things so that we specifically reject >> indexes? And maybe some kind of a check that will reject anything >> that lacks a relfilnode? That seems like it would be more on point. > > I agree, I don't have strong

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Robert Haas
On Tue, Mar 8, 2016 at 12:49 PM, Tom Lane wrote: >> However, after some further thought, I think we might actually be OK. >> If a page goes from all-frozen to not-all-frozen while VACUUM is >> running, any new XID added to the page must be newer than the >> oldestXmin value computed by vacuum_set_

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Masahiko Sawada
On Wed, Mar 9, 2016 at 1:23 AM, Jeff Janes wrote: > On Tue, Mar 8, 2016 at 5:30 AM, Robert Haas wrote: > I left out the relkind check from the final commit because, for one > thing, the check you added isn't actually right: toast relations can > also have a visibility map. And also, I'm sort of

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Tom Lane
Robert Haas writes: > The patch makes some attempt to update the comment mechanically, but > that's not nearly enough. That comment is explaining that you *can't* > rely on the visibility map to tell you *for sure* that a page does not > require vacuuming. For current uses, that's OK, because if

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Robert Haas
On Mon, Mar 7, 2016 at 12:41 PM, Masahiko Sawada wrote: > Attached latest version optimisation patch. > I'm still consider regarding pg_upgrade regression test code, so I > will submit that patch later. I just spent some time looking at this and I'm a bit worried about the following (existing) co

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Jeff Janes
On Tue, Mar 8, 2016 at 5:30 AM, Robert Haas wrote: > On Tue, Mar 8, 2016 at 7:26 AM, Masahiko Sawada wrote: >> Regarding pg_visibility module, I'd like to share some bugs and >> propose to add a relation type condition to each functions. > > OK, thanks. > >> Including it, I've attached remaining

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Robert Haas
On Tue, Mar 8, 2016 at 8:30 AM, Robert Haas wrote: > On Tue, Mar 8, 2016 at 7:26 AM, Masahiko Sawada wrote: >> Regarding pg_visibility module, I'd like to share some bugs and >> propose to add a relation type condition to each functions. > > OK, thanks. I left out the relkind check from the fina

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Robert Haas
On Tue, Mar 8, 2016 at 7:26 AM, Masahiko Sawada wrote: > Regarding pg_visibility module, I'd like to share some bugs and > propose to add a relation type condition to each functions. OK, thanks. > Including it, I've attached remaining 2 patches; one is removing page > conversion code from pg_up

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-08 Thread Masahiko Sawada
On Tue, Mar 8, 2016 at 1:20 PM, Kyotaro HORIGUCHI wrote: > Hello, thank you for updating this tool. > > At Mon, 7 Mar 2016 14:03:08 -0500, Robert Haas wrote > in >> On Mon, Mar 7, 2016 at 12:41 PM, Masahiko Sawada >> wrote: >> > Attached latest version optimisation patch. >> > I'm still consi

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-07 Thread Kyotaro HORIGUCHI
Hello, thank you for updating this tool. At Mon, 7 Mar 2016 14:03:08 -0500, Robert Haas wrote in > On Mon, Mar 7, 2016 at 12:41 PM, Masahiko Sawada > wrote: > > Attached latest version optimisation patch. > > I'm still consider regarding pg_upgrade regression test code, so I > > will submit t

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-07 Thread Peter Geoghegan
On Mon, Mar 7, 2016 at 4:50 PM, Robert Haas wrote: > Here's an updated patch with an API that I think is much more > reasonable to expose to users, and documentation! It assumes that the > patch I posted a few hours ago to remove PD_ALL_FROZEN will be > accepted; if that falls apart for some reas

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-07 Thread Robert Haas
On Sat, Mar 5, 2016 at 9:25 AM, Masahiko Sawada wrote: >> I actually think end-users might well want to use it. Also, I created >> it by hacking up pg_freespacemap, so it may make sense to have it in >> the same place. >> I would also be tempted to add an additional C functions that scan the >> e

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-07 Thread Robert Haas
On Mon, Mar 7, 2016 at 12:41 PM, Masahiko Sawada wrote: > Attached latest version optimisation patch. > I'm still consider regarding pg_upgrade regression test code, so I > will submit that patch later. I was thinking more about this today and I think that we don't actually need the PD_ALL_FROZEN

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-07 Thread Masahiko Sawada
On Sat, Mar 5, 2016 at 11:25 PM, Masahiko Sawada wrote: > On Sat, Mar 5, 2016 at 1:25 AM, Robert Haas wrote: >> On Wed, Mar 2, 2016 at 6:41 PM, Tom Lane wrote: >>> Jim Nasby writes: On 3/2/16 4:21 PM, Peter Geoghegan wrote: > I think you should commit this. The chances of anyone other

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-05 Thread Masahiko Sawada
On Sat, Mar 5, 2016 at 1:25 AM, Robert Haas wrote: > On Wed, Mar 2, 2016 at 6:41 PM, Tom Lane wrote: >> Jim Nasby writes: >>> On 3/2/16 4:21 PM, Peter Geoghegan wrote: I think you should commit this. The chances of anyone other than you and Masahiko recalling that you developed this to

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-04 Thread Robert Haas
On Wed, Mar 2, 2016 at 6:41 PM, Tom Lane wrote: > Jim Nasby writes: >> On 3/2/16 4:21 PM, Peter Geoghegan wrote: >>> I think you should commit this. The chances of anyone other than you >>> and Masahiko recalling that you developed this tool in 3 years is >>> essentially nil. I think that the cos

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-02 Thread Kyotaro HORIGUCHI
At Wed, 2 Mar 2016 17:57:27 -0600, Jim Nasby wrote in <56d77de7.7080...@bluetreble.com> > On 3/2/16 5:41 PM, Tom Lane wrote: > > Jim Nasby writes: > >> On 3/2/16 4:21 PM, Peter Geoghegan wrote: > >>> I think you should commit this. The chances of anyone other than you > >>> and Masahiko recallin

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-02 Thread Jim Nasby
On 3/2/16 5:41 PM, Tom Lane wrote: Jim Nasby writes: On 3/2/16 4:21 PM, Peter Geoghegan wrote: I think you should commit this. The chances of anyone other than you and Masahiko recalling that you developed this tool in 3 years is essentially nil. I think that the cost of committing a developer

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-02 Thread Tom Lane
Jim Nasby writes: > On 3/2/16 4:21 PM, Peter Geoghegan wrote: >> I think you should commit this. The chances of anyone other than you >> and Masahiko recalling that you developed this tool in 3 years is >> essentially nil. I think that the cost of committing a developer-level >> debugging tool lik

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-02 Thread Jim Nasby
On 3/2/16 4:21 PM, Peter Geoghegan wrote: I think you should commit this. The chances of anyone other than you and Masahiko recalling that you developed this tool in 3 years is essentially nil. I think that the cost of committing a developer-level debugging tool like this is very low. Modules lik

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-02 Thread Peter Geoghegan
On Tue, Mar 1, 2016 at 6:51 PM, Robert Haas wrote: > I removed the pgstat stuff. I'm not sure we want that stuff in that > form; it doesn't seem to fit with the rest of what's in that view, and > it wasn't reliable in my testing. I did however throw together a > little contrib module for testing

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-02 Thread Kyotaro HORIGUCHI
Thank you for revising and commiting this. At Tue, 1 Mar 2016 21:51:55 -0500, Robert Haas wrote in > On Thu, Feb 18, 2016 at 3:45 AM, Masahiko Sawada > wrote: > > Attached updated 5 patches. > > I would like to explain these patch shortly again here to make > > reviewing more easier. > > > >

Re: [HACKERS] Freeze avoidance of very large table.

2016-03-01 Thread Robert Haas
On Thu, Feb 18, 2016 at 3:45 AM, Masahiko Sawada wrote: > Attached updated 5 patches. > I would like to explain these patch shortly again here to make > reviewing more easier. > > We can divided these patches into 2 purposes. > > 1. Freeze map > 000_ patch adds additional frozen bit into visibilit

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Masahiko Sawada
On Wed, Feb 17, 2016 at 4:29 PM, Masahiko Sawada wrote: > On Wed, Feb 17, 2016 at 4:08 AM, Bruce Momjian wrote: >> On Tue, Feb 16, 2016 at 03:57:01PM -0300, Alvaro Herrera wrote: >>> Masahiko Sawada wrote: >>> > On Wed, Feb 17, 2016 at 12:02 AM, Bruce Momjian wrote: >>> > > On Tue, Feb 16, 2016

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Masahiko Sawada
On Wed, Feb 17, 2016 at 4:08 AM, Bruce Momjian wrote: > On Tue, Feb 16, 2016 at 03:57:01PM -0300, Alvaro Herrera wrote: >> Masahiko Sawada wrote: >> > On Wed, Feb 17, 2016 at 12:02 AM, Bruce Momjian wrote: >> > > On Tue, Feb 16, 2016 at 11:56:25PM +0900, Masahiko Sawada wrote: >> > >> > I agreed

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Bruce Momjian
On Tue, Feb 16, 2016 at 03:57:01PM -0300, Alvaro Herrera wrote: > Masahiko Sawada wrote: > > On Wed, Feb 17, 2016 at 12:02 AM, Bruce Momjian wrote: > > > On Tue, Feb 16, 2016 at 11:56:25PM +0900, Masahiko Sawada wrote: > > >> > I agreed on ripping out the converter plugin ability of pg_upgrade. >

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Alvaro Herrera
Masahiko Sawada wrote: > On Wed, Feb 17, 2016 at 12:02 AM, Bruce Momjian wrote: > > On Tue, Feb 16, 2016 at 11:56:25PM +0900, Masahiko Sawada wrote: > >> > I agreed on ripping out the converter plugin ability of pg_upgrade. > >> > Remember pg_upgrade was originally written by EnterpriseDB staff, a

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Masahiko Sawada
On Wed, Feb 17, 2016 at 12:02 AM, Bruce Momjian wrote: > On Tue, Feb 16, 2016 at 11:56:25PM +0900, Masahiko Sawada wrote: >> > I agreed on ripping out the converter plugin ability of pg_upgrade. >> > Remember pg_upgrade was originally written by EnterpriseDB staff, and I >> > think they expected t

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Bruce Momjian
On Tue, Feb 16, 2016 at 11:56:25PM +0900, Masahiko Sawada wrote: > > I agreed on ripping out the converter plugin ability of pg_upgrade. > > Remember pg_upgrade was originally written by EnterpriseDB staff, and I > > think they expected their closed-source fork of Postgres might need a > > custom p

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-16 Thread Masahiko Sawada
On Tue, Feb 16, 2016 at 6:13 AM, Bruce Momjian wrote: > On Wed, Feb 10, 2016 at 04:39:15PM +0900, Kyotaro HORIGUCHI wrote: >> > I still agree with this plugin approach, but I felt it's still >> > complicated a bit, and I'm concerned that patch size has been >> > increased. >> > Please give me feed

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-15 Thread Bruce Momjian
On Wed, Feb 10, 2016 at 04:39:15PM +0900, Kyotaro HORIGUCHI wrote: > > I still agree with this plugin approach, but I felt it's still > > complicated a bit, and I'm concerned that patch size has been > > increased. > > Please give me feedbacks. > > Yeah, I feel the same. What make it worse, the pl

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-14 Thread Robert Haas
On Sun, Feb 14, 2016 at 12:19 AM, Masahiko Sawada wrote: > Thank you for reviewing this patch. > I've divided 000 patch into two patches, and attached latest 4 patches in > total. Thank you! I'll go through this again as soon as I have a free moment. -- Robert Haas EnterpriseDB: http://www.en

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-11 Thread Robert Haas
On Wed, Feb 3, 2016 at 12:32 PM, Masahiko Sawada wrote: > I've divided the main patch into two patches; add frozen bit patch and > pg_upgrade support patch. > 000 patch is almost same as previous code. (includes small fix) > 001 patch provides rewriting visibility map as a pageConverter routine.

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-11 Thread Masahiko Sawada
Thank you for reviewing this patch! On Wed, Feb 10, 2016 at 4:39 PM, Kyotaro HORIGUCHI wrote: > Hello, > > At Thu, 4 Feb 2016 02:32:29 +0900, Masahiko Sawada > wrote in >> On Tue, Feb 2, 2016 at 7:22 PM, Alvaro Herrera >> wrote: >> > Masahiko Sawada wrote: >> >> I think we have two options.

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-09 Thread Kyotaro HORIGUCHI
Hello, At Thu, 4 Feb 2016 02:32:29 +0900, Masahiko Sawada wrote in > On Tue, Feb 2, 2016 at 7:22 PM, Alvaro Herrera > wrote: > > Masahiko Sawada wrote: > >> I think we have two options. > >> > >> 1. Change page layout(PG_PAGE_LAYOUT_VERSION) to 5. pg_upgrade detects > >> it and then converts

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-03 Thread Masahiko Sawada
On Tue, Feb 2, 2016 at 7:22 PM, Alvaro Herrera wrote: > Masahiko Sawada wrote: > >> I misunderstood. Sorry for noise. >> I agree with adding conversion method as a pageConverter routine. > > \o/ > >> This patch doesn't change page layout actually, but pageConverter >> routine checks only the page

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-02 Thread Masahiko Sawada
On Tue, Feb 2, 2016 at 10:05 PM, Alvaro Herrera wrote: > This patch has gotten its fair share of feedback in this fest. I moved > it to the next commitfest. Please do keep working on it and reviewers > that have additional comments are welcome. Thanks! On Tue, Feb 2, 2016 at 8:59 PM, Kyotaro H

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-02 Thread Alvaro Herrera
This patch has gotten its fair share of feedback in this fest. I moved it to the next commitfest. Please do keep working on it and reviewers that have additional comments are welcome. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA,

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-02 Thread Kyotaro HORIGUCHI
Hello, At Tue, 2 Feb 2016 20:25:23 +0900, Masahiko Sawada wrote in > On Tue, Feb 2, 2016 at 7:22 PM, Alvaro Herrera > wrote: > > Masahiko Sawada wrote: > > > >> I misunderstood. Sorry for noise. > >> I agree with adding conversion method as a pageConverter routine. > > > > \o/ > > > >> This p

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-02 Thread Masahiko Sawada
On Tue, Feb 2, 2016 at 7:22 PM, Alvaro Herrera wrote: > Masahiko Sawada wrote: > >> I misunderstood. Sorry for noise. >> I agree with adding conversion method as a pageConverter routine. > > \o/ > >> This patch doesn't change page layout actually, but pageConverter >> routine checks only the page

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-02 Thread Alvaro Herrera
Masahiko Sawada wrote: > I misunderstood. Sorry for noise. > I agree with adding conversion method as a pageConverter routine. \o/ > This patch doesn't change page layout actually, but pageConverter > routine checks only the page layout. > And we have to plugin named convertLayout_X_to_Y. > > I

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-02 Thread Masahiko Sawada
On Tue, Feb 2, 2016 at 11:42 AM, Masahiko Sawada wrote: > On Tue, Feb 2, 2016 at 10:15 AM, Jim Nasby wrote: >> On 2/1/16 4:59 PM, Alvaro Herrera wrote: >>> >>> Masahiko Sawada wrote: >>> Attached updated version patch. Please review it. >>> >>> >>> In pg_upgrade, the "page convert" func

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-01 Thread Masahiko Sawada
On Tue, Feb 2, 2016 at 10:15 AM, Jim Nasby wrote: > On 2/1/16 4:59 PM, Alvaro Herrera wrote: >> >> Masahiko Sawada wrote: >> >>> Attached updated version patch. >>> Please review it. >> >> >> In pg_upgrade, the "page convert" functionality is there to abstract >> rewrites of pages being copied; yo

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-01 Thread Jim Nasby
On 2/1/16 4:59 PM, Alvaro Herrera wrote: Masahiko Sawada wrote: Attached updated version patch. Please review it. In pg_upgrade, the "page convert" functionality is there to abstract rewrites of pages being copied; your patch is circumventing it and AFAICS it makes the interface more complica

Re: [HACKERS] Freeze avoidance of very large table.

2016-02-01 Thread Alvaro Herrera
Masahiko Sawada wrote: > Attached updated version patch. > Please review it. In pg_upgrade, the "page convert" functionality is there to abstract rewrites of pages being copied; your patch is circumventing it and AFAICS it makes the interface more complicated for no good reason. I think the real

Re: [HACKERS] Freeze avoidance of very large table.

2016-01-17 Thread Masahiko Sawada
On Wed, Jan 13, 2016 at 12:16 AM, Masahiko Sawada wrote: > On Mon, Dec 28, 2015 at 6:38 PM, Masahiko Sawada > wrote: >> On Mon, Dec 21, 2015 at 11:54 PM, Robert Haas wrote: >>> On Mon, Dec 21, 2015 at 3:27 AM, Kyotaro HORIGUCHI >>> wrote: Hello, At Fri, 18 Dec 2015 12:09:43 -050

Re: [HACKERS] Freeze avoidance of very large table.

2016-01-12 Thread Masahiko Sawada
On Mon, Dec 28, 2015 at 6:38 PM, Masahiko Sawada wrote: > On Mon, Dec 21, 2015 at 11:54 PM, Robert Haas wrote: >> On Mon, Dec 21, 2015 at 3:27 AM, Kyotaro HORIGUCHI >> wrote: >>> Hello, >>> >>> At Fri, 18 Dec 2015 12:09:43 -0500, Robert Haas >>> wrote in >>> On Thu, Dec 17, 2015 at 1:17

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-28 Thread Masahiko Sawada
On Mon, Dec 21, 2015 at 11:54 PM, Robert Haas wrote: > On Mon, Dec 21, 2015 at 3:27 AM, Kyotaro HORIGUCHI > wrote: >> Hello, >> >> At Fri, 18 Dec 2015 12:09:43 -0500, Robert Haas >> wrote in >>> On Thu, Dec 17, 2015 at 1:17 AM, Michael Paquier >>> wrote: >>> > I am not really getting the mean

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-22 Thread Bruce Momjian
On Thu, Dec 17, 2015 at 06:44:46AM +, Simon Riggs wrote: > >> Thank you for having a look. > > > > I would not bother mentioning this detail in the pg_upgrade manual page: > > > > +   Since the format of visibility map has been changed in version 9.6, > > +   pg_upgrade

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-21 Thread Robert Haas
On Mon, Dec 21, 2015 at 3:27 AM, Kyotaro HORIGUCHI wrote: > Hello, > > At Fri, 18 Dec 2015 12:09:43 -0500, Robert Haas wrote > in >> On Thu, Dec 17, 2015 at 1:17 AM, Michael Paquier >> wrote: >> > I am not really getting the meaning of this sentence. Shouldn't this >> > be reworded something l

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-21 Thread Kyotaro HORIGUCHI
Hello, At Fri, 18 Dec 2015 12:09:43 -0500, Robert Haas wrote in > On Thu, Dec 17, 2015 at 1:17 AM, Michael Paquier > wrote: > > I am not really getting the meaning of this sentence. Shouldn't this > > be reworded something like: > > "Freezing occurs on the whole table once all pages of this re

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-18 Thread Robert Haas
On Thu, Dec 17, 2015 at 2:26 AM, Andres Freund wrote: > On 2015-12-17 16:22:24 +0900, Michael Paquier wrote: >> On Thu, Dec 17, 2015 at 4:10 PM, Andres Freund wrote: >> > On 2015-12-17 15:56:35 +0900, Michael Paquier wrote: >> >> On Thu, Dec 17, 2015 at 3:44 PM, Simon Riggs >> >> wrote: >> >> >

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-18 Thread Robert Haas
On Thu, Dec 17, 2015 at 1:17 AM, Michael Paquier wrote: > I am not really getting the meaning of this sentence. Shouldn't this > be reworded something like: > "Freezing occurs on the whole table once all pages of this relation require > it." That statement isn't remotely true, and I don't think

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-17 Thread Michael Paquier
On Fri, Dec 18, 2015 at 3:17 AM, Masahiko Sawada wrote: > On Thu, Dec 17, 2015 at 11:47 AM, Michael Paquier > wrote: >> make check with pg_upgrade is failing for me: >> Visibility map rewriting test failed >> make: *** [check] Error 1 > > make check with pg_upgrade is done successfully on my env

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-17 Thread Masahiko Sawada
On Thu, Dec 17, 2015 at 11:47 AM, Michael Paquier wrote: > On Thu, Dec 10, 2015 at 3:31 AM, Robert Haas wrote: >> On Mon, Nov 30, 2015 at 12:58 PM, Bruce Momjian wrote: >>> On Mon, Nov 30, 2015 at 10:48:04PM +0530, Masahiko Sawada wrote: On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: >

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-16 Thread Andres Freund
On 2015-12-17 16:22:24 +0900, Michael Paquier wrote: > On Thu, Dec 17, 2015 at 4:10 PM, Andres Freund wrote: > > On 2015-12-17 15:56:35 +0900, Michael Paquier wrote: > >> On Thu, Dec 17, 2015 at 3:44 PM, Simon Riggs wrote: > >> > For me, rewriting the visibility map is a new data loss bug waiting

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-16 Thread Michael Paquier
On Thu, Dec 17, 2015 at 4:10 PM, Andres Freund wrote: > On 2015-12-17 15:56:35 +0900, Michael Paquier wrote: >> On Thu, Dec 17, 2015 at 3:44 PM, Simon Riggs wrote: >> > For me, rewriting the visibility map is a new data loss bug waiting to >> > happen. I am worried that the group is not taking se

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-16 Thread Andres Freund
On 2015-12-17 15:56:35 +0900, Michael Paquier wrote: > On Thu, Dec 17, 2015 at 3:44 PM, Simon Riggs wrote: > > For me, rewriting the visibility map is a new data loss bug waiting to > > happen. I am worried that the group is not taking seriously the potential > > for catastrophe here. > > FWIW, I

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-16 Thread Michael Paquier
On Thu, Dec 17, 2015 at 3:44 PM, Simon Riggs wrote: > For me, rewriting the visibility map is a new data loss bug waiting to > happen. I am worried that the group is not taking seriously the potential > for catastrophe here. FWIW, I'm following this line and merging the vm file into a single unit

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-16 Thread Simon Riggs
On 9 December 2015 at 18:31, Robert Haas wrote: > On Mon, Nov 30, 2015 at 12:58 PM, Bruce Momjian wrote: > > On Mon, Nov 30, 2015 at 10:48:04PM +0530, Masahiko Sawada wrote: > >> On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes > wrote: > >> > On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada < > saw

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-16 Thread Michael Paquier
On Thu, Dec 10, 2015 at 3:31 AM, Robert Haas wrote: > On Mon, Nov 30, 2015 at 12:58 PM, Bruce Momjian wrote: >> On Mon, Nov 30, 2015 at 10:48:04PM +0530, Masahiko Sawada wrote: >>> On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: >>> > On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada >>> > w

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-09 Thread Robert Haas
On Mon, Nov 30, 2015 at 12:58 PM, Bruce Momjian wrote: > On Mon, Nov 30, 2015 at 10:48:04PM +0530, Masahiko Sawada wrote: >> On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: >> > On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada >> > wrote: >> >> >> >> Yeah, we need to consider to compute chec

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-04 Thread Masahiko Sawada
On Fri, Dec 4, 2015 at 9:51 PM, Jeff Janes wrote: > On Tue, Dec 1, 2015 at 10:40 AM, Masahiko Sawada > wrote: >> On Tue, Dec 1, 2015 at 3:04 AM, Jeff Janes wrote: >>> On Mon, Nov 30, 2015 at 9:18 AM, Masahiko Sawada >>> wrote: On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: > On

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-04 Thread Jeff Janes
On Tue, Dec 1, 2015 at 10:40 AM, Masahiko Sawada wrote: > On Tue, Dec 1, 2015 at 3:04 AM, Jeff Janes wrote: >> On Mon, Nov 30, 2015 at 9:18 AM, Masahiko Sawada >> wrote: >>> On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada wrote: >

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-02 Thread Masahiko Sawada
On Wed, Dec 2, 2015 at 9:30 AM, Kyotaro HORIGUCHI wrote: > Hello, > >> You're right, it's not necessary. >> Attached latest v29 patch which removes the mention in pg_upgrade >> documentation. > > The changes looks to be correct but I haven't tested. > And I have some additional random comments. >

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-01 Thread Kyotaro HORIGUCHI
Hello, > You're right, it's not necessary. > Attached latest v29 patch which removes the mention in pg_upgrade > documentation. The changes looks to be correct but I haven't tested. And I have some additional random comments. visibilitymap.c: In visibilitymap_set, the followint lines.

Re: [HACKERS] Freeze avoidance of very large table.

2015-12-01 Thread Masahiko Sawada
On Tue, Dec 1, 2015 at 3:04 AM, Jeff Janes wrote: > On Mon, Nov 30, 2015 at 9:18 AM, Masahiko Sawada > wrote: >> On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: >>> On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada >>> wrote: Yeah, we need to consider to compute checksum if enable

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-30 Thread Jeff Janes
On Mon, Nov 30, 2015 at 9:18 AM, Masahiko Sawada wrote: > On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: >> On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada >> wrote: >>> >>> Yeah, we need to consider to compute checksum if enabled. >>> I've changed the patch, and attached. >>> Please revie

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-30 Thread Bruce Momjian
On Mon, Nov 30, 2015 at 07:05:21PM +0100, Andres Freund wrote: > On 2015-11-30 12:58:43 -0500, Bruce Momjian wrote: > > I would not bother mentioning this detail in the pg_upgrade manual page: > > > > + Since the format of visibility map has been changed in version 9.6, > > + pg_upgrade create

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-30 Thread Andres Freund
On 2015-11-30 12:58:43 -0500, Bruce Momjian wrote: > I would not bother mentioning this detail in the pg_upgrade manual page: > > + Since the format of visibility map has been changed in version 9.6, > + pg_upgrade creates and rewrite new > '_vm' > + file even if upgrading from 9.5 or befor

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-30 Thread Bruce Momjian
On Mon, Nov 30, 2015 at 10:48:04PM +0530, Masahiko Sawada wrote: > On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: > > On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada > > wrote: > >> > >> Yeah, we need to consider to compute checksum if enabled. > >> I've changed the patch, and attached. > >

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-30 Thread Masahiko Sawada
On Sun, Nov 29, 2015 at 2:21 AM, Jeff Janes wrote: > On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada > wrote: >> >> Yeah, we need to consider to compute checksum if enabled. >> I've changed the patch, and attached. >> Please review it. > > Thanks for the update. This now conflicts with the upd

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-28 Thread Jeff Janes
On Tue, Nov 24, 2015 at 3:13 PM, Masahiko Sawada wrote: > > Yeah, we need to consider to compute checksum if enabled. > I've changed the patch, and attached. > Please review it. Thanks for the update. This now conflicts with the updates doesn to fix pg_upgrade out-of-space issue on Windows. I've

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-24 Thread Masahiko Sawada
On Mon, Nov 23, 2015 at 6:27 AM, Jeff Janes wrote: > On Sun, Nov 22, 2015 at 8:16 AM, Masahiko Sawada > wrote: > >> Thank you for taking the time to review this patch! >> The updated version patch is attached. > > I am skeptical about just copying the old page header to be two new > page headers

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-22 Thread Jeff Janes
On Sun, Nov 22, 2015 at 8:16 AM, Masahiko Sawada wrote: > Thank you for taking the time to review this patch! > The updated version patch is attached. I am skeptical about just copying the old page header to be two new page headers. I don't know what the implications for this will be on pd_lsn.

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-22 Thread Masahiko Sawada
On Sat, Nov 21, 2015 at 6:50 AM, Jeff Janes wrote: > On Thu, Nov 19, 2015 at 6:44 AM, Masahiko Sawada > wrote: >> On Thu, Nov 19, 2015 at 5:54 AM, Jeff Janes wrote: >>> On Wed, Nov 18, 2015 at 11:18 AM, Jeff Janes wrote: I get an error when running pg_upgrade from 9.4 to 9.6-this >>>

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-20 Thread Jeff Janes
On Thu, Nov 19, 2015 at 6:44 AM, Masahiko Sawada wrote: > On Thu, Nov 19, 2015 at 5:54 AM, Jeff Janes wrote: >> On Wed, Nov 18, 2015 at 11:18 AM, Jeff Janes wrote: >>> >>> I get an error when running pg_upgrade from 9.4 to 9.6-this >>> >>> error while copying relation "mediawiki.archive" >>> ("/

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-19 Thread Masahiko Sawada
On Thu, Nov 19, 2015 at 5:54 AM, Jeff Janes wrote: > On Wed, Nov 18, 2015 at 11:18 AM, Jeff Janes wrote: >> >> I get an error when running pg_upgrade from 9.4 to 9.6-this >> >> error while copying relation "mediawiki.archive" >> ("/tmp/data/base/16414/21043_vm" to >> "/tmp/data_fm/base/16400/2104

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-18 Thread Jeff Janes
On Wed, Nov 18, 2015 at 11:18 AM, Jeff Janes wrote: > > I get an error when running pg_upgrade from 9.4 to 9.6-this > > error while copying relation "mediawiki.archive" > ("/tmp/data/base/16414/21043_vm" to > "/tmp/data_fm/base/16400/21043_vm"): No such file or directory OK, so the problem seems

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-18 Thread Jeff Janes
On Tue, Nov 17, 2015 at 10:32 PM, Masahiko Sawada wrote: > > Attached latest v24 patch. > I've changed patch so that just adding frozen bit into visibility map. > So the size of patch is almost half of previous one. > Should there be an Assert in visibilitymap_get_status (or elsewhere) against th

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-17 Thread Masahiko Sawada
On Tue, Nov 17, 2015 at 7:29 PM, Masahiko Sawada wrote: > > > On Tue, Nov 17, 2015 at 10:45 AM, Robert Haas wrote: >> On Sun, Nov 15, 2015 at 1:47 AM, Amit Kapila >> wrote: >>> On Sat, Nov 14, 2015 at 1:19 AM, Andres Freund >>> wrote: On 2015-10-31 11:02:12 +0530, Amit Kapila wrote: >

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-17 Thread Masahiko Sawada
On Wed, Nov 18, 2015 at 12:56 AM, Thom Brown wrote: > On 17 November 2015 at 15:43, Jim Nasby wrote: >> On 11/17/15 4:41 AM, Thom Brown wrote: >>> >>> Could someone post a TL;DR summary of what the current plan looks >>> like? I can see there is a huge amount of discussion to trawl back >>> thro

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-17 Thread Thom Brown
On 17 November 2015 at 15:43, Jim Nasby wrote: > On 11/17/15 4:41 AM, Thom Brown wrote: >> >> Could someone post a TL;DR summary of what the current plan looks >> like? I can see there is a huge amount of discussion to trawl back >> through. I can see it's something to do with the visibility map

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-17 Thread Jim Nasby
On 11/17/15 4:41 AM, Thom Brown wrote: Could someone post a TL;DR summary of what the current plan looks like? I can see there is a huge amount of discussion to trawl back through. I can see it's something to do with the visibility map. And does it avoid freezing very large tables like the tit

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-17 Thread Thom Brown
On 17 November 2015 at 10:29, Masahiko Sawada wrote: > > > On Tue, Nov 17, 2015 at 10:45 AM, Robert Haas wrote: >> On Sun, Nov 15, 2015 at 1:47 AM, Amit Kapila >> wrote: >>> On Sat, Nov 14, 2015 at 1:19 AM, Andres Freund >>> wrote: On 2015-10-31 11:02:12 +0530, Amit Kapila wrote: > On

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-17 Thread Masahiko Sawada
On Tue, Nov 17, 2015 at 10:45 AM, Robert Haas > wrote: > On Sun, Nov 15, 2015 at 1:47 AM, Amit Kapila > wrote: >> On Sat, Nov 14, 2015 at 1:19 AM, Andres Freund > wrote: >>> On 2015-10-31 11:02:12 +0530, Amit Kapila wrote: >>> > On Thu, Oct 8, 2015 at 11:05 PM, Simon Riggs > >>> > wrote: >>> > > >>

Re: [HACKERS] Freeze avoidance of very large table.

2015-11-16 Thread Robert Haas
On Sun, Nov 15, 2015 at 1:47 AM, Amit Kapila wrote: > On Sat, Nov 14, 2015 at 1:19 AM, Andres Freund wrote: >> On 2015-10-31 11:02:12 +0530, Amit Kapila wrote: >> > On Thu, Oct 8, 2015 at 11:05 PM, Simon Riggs >> > wrote: >> > > >> > > On 1 October 2015 at 23:30, Josh Berkus wrote: >> > >> >> >

  1   2   3   >