Re: [HACKERS] 64-bit XIDs again

2015-08-11 Thread Simon Riggs
On 30 July 2015 at 14:26, Alexander Korotkov wrote: > As I mentioned in CSN thread, it would be nice to replace XID with CSN > when setting hint bits for tuple. In this case when hint bits are set we > don't need any additional lookups to check visibility. > > http://www.postgresql.org/message-i

Re: [HACKERS] 64-bit XIDs again

2015-08-01 Thread Simon Riggs
On 31 July 2015 at 22:46, Heikki Linnakangas wrote: > On 07/31/2015 12:29 AM, Josh Berkus wrote: > >> On 07/30/2015 07:24 AM, Heikki Linnakangas wrote: >> >>> >>> You'd never be forced to do anti-wraparound >>> vacuums, you could just let the clog grow arbitrarily large >>> >> >> When I introduce

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Tom Lane
Josh Berkus writes: > On 07/31/2015 02:46 PM, Heikki Linnakangas wrote: >> Well, sure, if you don't want the clog to grow arbitrarily large, then >> you need to freeze. And most people would want to freeze regularly, to >> keep the clog size in check. The point is that you wouldn't *have* to do >>

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Josh Berkus
On 07/31/2015 02:46 PM, Heikki Linnakangas wrote: > On 07/31/2015 12:29 AM, Josh Berkus wrote: >> On 07/30/2015 07:24 AM, Heikki Linnakangas wrote: >>> >>> I think we should move to 64-bit XIDs in in-memory structs snapshots, >>> proc array etc. And expand clog to handle 64-bit XIDs. But keep the >

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Heikki Linnakangas
On 07/31/2015 12:29 AM, Josh Berkus wrote: On 07/30/2015 07:24 AM, Heikki Linnakangas wrote: I think we should move to 64-bit XIDs in in-memory structs snapshots, proc array etc. And expand clog to handle 64-bit XIDs. But keep the xmin/xmax fields on heap pages at 32-bits, and add an epoch-like

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Robert Haas
On Thu, Jul 30, 2015 at 5:23 PM, Arthur Silva wrote: > In the other hand PG tuple overhead is already the largest among the > alternatives. > Even if storage keeps getting faster and cheaper stuff you can't ignore the > overhead of adding yet another 8bytes to each tuple. +1, very much. -- Robe

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Simon Riggs
On 31 July 2015 at 11:00, Alexander Korotkov wrote: > > If use upgrade database cluster with pg_upgrade, he would stop old > postmaster, pg_upgrade, start new postmaster. That means we start from the > point when there is no running transactions. Thus, between tuples of old > format there are two

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Alexander Korotkov
On Fri, Jul 31, 2015 at 12:23 AM, Tom Lane wrote: > But the elephant in the room is on-disk compatibility. There is > absolutely no way that we can just change xmin/xmax to 64 bits without a > disk format break. That seems problematic. But I'm not yet convinced that there is absolutely no way

Re: [HACKERS] 64-bit XIDs again

2015-07-31 Thread Alexander Korotkov
On Fri, Jul 31, 2015 at 1:27 AM, Petr Jelinek wrote: > On 2015-07-30 23:23, Tom Lane wrote: > >> But the elephant in the room is on-disk compatibility. There is >> absolutely no way that we can just change xmin/xmax to 64 bits without a >> disk format break. However, if we do something like wha

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Heikki Linnakangas
On 07/31/2015 09:22 AM, Gurjeet Singh wrote: On Jul 30, 2015 2:23 PM, "Tom Lane" wrote: But the elephant in the room is on-disk compatibility. There is absolutely no way that we can just change xmin/xmax to 64 bits without a disk format break. However, if we do something like what Heikki is s

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Gurjeet Singh
On Jul 30, 2015 2:23 PM, "Tom Lane" wrote: > > Gavin Flower writes: > > On 31/07/15 02:24, Heikki Linnakangas wrote: > >> There is a big downside to expanding xmin/xmax to 64 bits: it takes > >> space. More space means more memory needed for caching, more memory > >> bandwidth, more I/O, etc. > >

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Petr Jelinek
On 2015-07-30 23:23, Tom Lane wrote: Gavin Flower writes: On 31/07/15 02:24, Heikki Linnakangas wrote: There is a big downside to expanding xmin/xmax to 64 bits: it takes space. More space means more memory needed for caching, more memory bandwidth, more I/O, etc. I think having a special c

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Josh Berkus
On 07/30/2015 07:24 AM, Heikki Linnakangas wrote: > > I think we should move to 64-bit XIDs in in-memory structs snapshots, > proc array etc. And expand clog to handle 64-bit XIDs. But keep the > xmin/xmax fields on heap pages at 32-bits, and add an epoch-like field > to the page header so that lo

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Arthur Silva
On Thu, Jul 30, 2015 at 5:31 PM, Gavin Flower wrote: > On 31/07/15 02:24, Heikki Linnakangas wrote: > >> On 07/30/2015 04:26 PM, Alexander Korotkov wrote: >> >>> Also, I think it's possible to migrate to 64-bit XIDs without breaking >>> pg_upgrade. Old tuples can be leaved with 32-bit XIDs while

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Tom Lane
Gavin Flower writes: > On 31/07/15 02:24, Heikki Linnakangas wrote: >> There is a big downside to expanding xmin/xmax to 64 bits: it takes >> space. More space means more memory needed for caching, more memory >> bandwidth, more I/O, etc. > I think having a special case to save 32 bits per tupl

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Gavin Flower
On 31/07/15 02:24, Heikki Linnakangas wrote: On 07/30/2015 04:26 PM, Alexander Korotkov wrote: Also, I think it's possible to migrate to 64-bit XIDs without breaking pg_upgrade. Old tuples can be leaved with 32-bit XIDs while new tuples would be created with 64-bit XIDs. We can use free bits in

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Joshua D. Drake
On 07/30/2015 08:04 AM, Simon Riggs wrote: There is a big downside to expanding xmin/xmax to 64 bits: it takes space. More space means more memory needed for caching, more memory bandwidth, more I/O, etc. My feeling is that the overhead will recede in time. Having a nice, simple c

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Simon Riggs
On 30 July 2015 at 15:24, Heikki Linnakangas wrote: > On 07/30/2015 04:26 PM, Alexander Korotkov wrote: > >> Also, I think it's possible to migrate to 64-bit XIDs without breaking >> pg_upgrade. Old tuples can be leaved with 32-bit XIDs while new tuples >> would be created with 64-bit XIDs. We ca

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Heikki Linnakangas
On 07/30/2015 05:57 PM, Alexander Korotkov wrote: On Thu, Jul 30, 2015 at 5:24 PM, Heikki Linnakangas wrote: I think we should move to 64-bit XIDs in in-memory structs snapshots, proc array etc. And expand clog to handle 64-bit XIDs. But keep the xmin/xmax fields on heap pages at 32-bits, and

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Alexander Korotkov
On Thu, Jul 30, 2015 at 5:24 PM, Heikki Linnakangas wrote: > On 07/30/2015 04:26 PM, Alexander Korotkov wrote: > >> Also, I think it's possible to migrate to 64-bit XIDs without breaking >> pg_upgrade. Old tuples can be leaved with 32-bit XIDs while new tuples >> would be created with 64-bit XIDs

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 07/30/2015 07:14 AM, Simon Riggs wrote: > On 30 July 2015 at 14:26, Alexander Korotkov > > Any thoughts? Do you think 64-bit XIDs worth it? > > The problem of freezing is painful, but not impossible, which is > why we have held out so long. > >

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Heikki Linnakangas
On 07/30/2015 04:26 PM, Alexander Korotkov wrote: Also, I think it's possible to migrate to 64-bit XIDs without breaking pg_upgrade. Old tuples can be leaved with 32-bit XIDs while new tuples would be created with 64-bit XIDs. We can use free bits in t_infomask2 to distinguish old and new formats

Re: [HACKERS] 64-bit XIDs again

2015-07-30 Thread Simon Riggs
On 30 July 2015 at 14:26, Alexander Korotkov wrote: > Any thoughts? Do you think 64-bit XIDs worth it? > The problem of freezing is painful, but not impossible, which is why we have held out so long. The problem of very long lived snapshots is coming closer at the same speed as freezing; there

[HACKERS] 64-bit XIDs again

2015-07-30 Thread Alexander Korotkov
Hackers, I know there were already couple of threads about 64bit XIDs. http://www.postgresql.org/message-id/42cccfe9.9040...@intellilink.co.jp http://www.postgresql.org/message-id/4f6c0e13.3080...@wiesinger.com I read them carefully, but I didn't find all the arguments for 64bit XIDs mentioned. Th