Re: GiST VACUUM

2019-07-24 Thread Peter Geoghegan
On Wed, Jul 24, 2019 at 11:33 AM Heikki Linnakangas wrote: > That's probably how it's going to go, but hey, doesn't hurt to ask :-). I think that it would be fine to be conservative with nbtree, and only target the master branch. The problem is annoying, certainly, but it's not likely to make a h

Re: GiST VACUUM

2019-07-24 Thread Heikki Linnakangas
On 24/07/2019 21:02, Peter Geoghegan wrote: On Wed, Jul 24, 2019 at 10:30 AM Heikki Linnakangas wrote: Pushed this now, to master and REL_12_STABLE. Now, B-tree indexes still have the same problem, in all versions. Any volunteers to write a similar fix for B-trees? I was hoping that you'd wo

Re: GiST VACUUM

2019-07-24 Thread Peter Geoghegan
On Wed, Jul 24, 2019 at 10:30 AM Heikki Linnakangas wrote: > Pushed this now, to master and REL_12_STABLE. > > Now, B-tree indexes still have the same problem, in all versions. Any > volunteers to write a similar fix for B-trees? I was hoping that you'd work on it. :-) Any reason to think that

Re: GiST VACUUM

2019-07-24 Thread Heikki Linnakangas
On 22/07/2019 16:09, Heikki Linnakangas wrote: Unless something comes up, I'll commit this tomorrow. Pushed this now, to master and REL_12_STABLE. Now, B-tree indexes still have the same problem, in all versions. Any volunteers to write a similar fix for B-trees? - Heikki

Re: GiST VACUUM

2019-07-22 Thread Heikki Linnakangas
On 28/06/2019 01:02, Thomas Munro wrote: On Thu, Jun 27, 2019 at 11:38 PM Heikki Linnakangas wrote: * I moved the logic to extend a 32-bit XID to 64-bits to a new helper function in varsup.c. I'm a bit uneasy about making this into a generally available function for reuse. I think we should

Re: GiST VACUUM

2019-07-16 Thread Amit Kapila
On Fri, Jun 28, 2019 at 3:32 AM Thomas Munro wrote: > > On Thu, Jun 27, 2019 at 11:38 PM Heikki Linnakangas wrote: > > * I moved the logic to extend a 32-bit XID to 64-bits to a new helper > > function in varsup.c. > > I'm a bit uneasy about making this into a generally available function > for r

Re: GiST VACUUM

2019-07-03 Thread Peter Geoghegan
On Thu, Apr 4, 2019 at 8:15 AM Heikki Linnakangas wrote: > I think we should fix this in a similar manner in B-tree, too, but that > can be done separately. For B-tree, we need to worry about > backwards-compatibility, but that seems simple enough; we just need to > continue to understand old dele

Re: GiST VACUUM

2019-06-27 Thread Thomas Munro
On Thu, Jun 27, 2019 at 11:38 PM Heikki Linnakangas wrote: > * I moved the logic to extend a 32-bit XID to 64-bits to a new helper > function in varsup.c. I'm a bit uneasy about making this into a generally available function for reuse. I think we should instead come up with a very small number

Re: GiST VACUUM

2019-06-27 Thread Heikki Linnakangas
On 27/06/2019 20:15, Andrey Borodin wrote: But I have stupid question again, about this code: https://github.com/x4m/postgres_g/commit/096d5586537d29ff316ca8ce074bbe1b325879ee#diff-754126824470cb8e68fd5e32af6d3bcaR417 nextFullXid = ReadNextFullTransactionId();

Re: GiST VACUUM

2019-06-27 Thread Andrey Borodin
> 27 июня 2019 г., в 16:38, Heikki Linnakangas написал(а): > > I haven't done any testing on this yet. Andrey, would you happen to have an > environment ready to test this? Patches do not deadlock and delete pages on "rescan test" - setup that we used to detect "left jumps" in during develo

Re: GiST VACUUM

2019-06-27 Thread Andrey Borodin
> 27 июня 2019 г., в 16:38, Heikki Linnakangas написал(а): > > I haven't done any testing on this yet. Andrey, would you happen to have an > environment ready to test this? Thanks! I will do some testing this evening (UTC+5). But I'm not sure I can reliably test wraparound of xids... I wil

Re: GiST VACUUM

2019-06-27 Thread Heikki Linnakangas
On 26/06/2019 06:07, Thomas Munro wrote: On Wed, Jun 26, 2019 at 2:33 PM Michael Paquier wrote: On Tue, Jun 25, 2019 at 02:38:43PM +0500, Andrey Borodin wrote: I feel a little uncomfortable with number 0x7fff right in code. PG_INT32_MAX... MaxTransactionId / 2? Yeah, that makes sense

Re: GiST VACUUM

2019-06-25 Thread Thomas Munro
On Wed, Jun 26, 2019 at 2:33 PM Michael Paquier wrote: > On Tue, Jun 25, 2019 at 02:38:43PM +0500, Andrey Borodin wrote: > > I feel a little uncomfortable with number 0x7fff right in code. > > PG_INT32_MAX... MaxTransactionId / 2? -- Thomas Munro https://enterprisedb.com

Re: GiST VACUUM

2019-06-25 Thread Michael Paquier
On Tue, Jun 25, 2019 at 02:38:43PM +0500, Andrey Borodin wrote: > I feel a little uncomfortable with number 0x7fff right in code. PG_INT32_MAX... -- Michael signature.asc Description: PGP signature

Re: GiST VACUUM

2019-06-25 Thread Andrey Borodin
Hi! Thanks for clarification, now I understand these patches better. > 25 июня 2019 г., в 13:10, Heikki Linnakangas написал(а): > >> Also, I did not understand this optimization: >> +/* >> + * We can skip this if the page was deleted so long ago, that no scan >> can possibly >> + *

Re: GiST VACUUM

2019-06-25 Thread Heikki Linnakangas
(Thanks for the reminder on this, Michael!) On 05/04/2019 08:39, Andrey Borodin wrote: 4 апр. 2019 г., в 20:15, Heikki Linnakangas написал(а): I suggest that we do the attached. It fixes this for GiST. The patch changes expands the "deletion XID" to 64-bits, and changes where it's stored. Inst

Re: GiST VACUUM

2019-06-24 Thread Michael Paquier
Heikki, On Thu, Apr 04, 2019 at 06:15:21PM +0300, Heikki Linnakangas wrote: > I think we should fix this in a similar manner in B-tree, too, but that can > be done separately. For B-tree, we need to worry about > backwards-compatibility, but that seems simple enough; we just need to > continue to

Re: GiST VACUUM

2019-04-04 Thread Andrey Borodin
Hi! > 4 апр. 2019 г., в 20:15, Heikki Linnakangas написал(а): > > On 25/03/2019 15:20, Heikki Linnakangas wrote: >> On 24/03/2019 18:50, Andrey Borodin wrote: >>> I was working on new version of gist check in amcheck and understand one >>> more thing: >>> >>> /* Can this page be recycled yet?

Re: GiST VACUUM

2019-04-04 Thread Heikki Linnakangas
On 25/03/2019 15:20, Heikki Linnakangas wrote: On 24/03/2019 18:50, Andrey Borodin wrote: I was working on new version of gist check in amcheck and understand one more thing: /* Can this page be recycled yet? */ bool gistPageRecyclable(Page page) { return PageIsNew(page) || (Gi

Re: GiST VACUUM

2019-03-25 Thread Heikki Linnakangas
On 24/03/2019 18:50, Andrey Borodin wrote: I was working on new version of gist check in amcheck and understand one more thing: /* Can this page be recycled yet? */ bool gistPageRecyclable(Page page) { return PageIsNew(page) || (GistPageIsDeleted(page) && TransactionIdPr

Re: GiST VACUUM

2019-03-24 Thread Andrey Borodin
> 22 марта 2019 г., в 17:03, Heikki Linnakangas написал(а): > I was working on new version of gist check in amcheck and understand one more thing: /* Can this page be recycled yet? */ bool gistPageRecyclable(Page page) { return PageIsNew(page) || (GistPageIsDeleted(page) &&

Re: GiST VACUUM

2019-03-22 Thread Andrey Borodin
> 22 марта 2019 г., в 19:37, Heikki Linnakangas написал(а): > > On 21/03/2019 19:04, Heikki Linnakangas wrote: >> Attached is the latest patch version, to be applied on top of the >> IntegerSet patch. > > And committed. Thanks Andrey! > > - Heikki Cool! Thank you very much! At the beginning

Re: GiST VACUUM

2019-03-22 Thread Heikki Linnakangas
On 22/03/2019 13:37, Heikki Linnakangas wrote: On 21/03/2019 19:04, Heikki Linnakangas wrote: Attached is the latest patch version, to be applied on top of the IntegerSet patch. And committed. Thanks Andrey! This caused the buildfarm to go pink... I was able to reproduce it, by running the

Re: GiST VACUUM

2019-03-22 Thread Heikki Linnakangas
On 21/03/2019 19:04, Heikki Linnakangas wrote: Attached is the latest patch version, to be applied on top of the IntegerSet patch. And committed. Thanks Andrey! - Heikki

Re: GiST VACUUM

2019-03-22 Thread Heikki Linnakangas
On 22/03/2019 10:00, Andrey Borodin wrote: 22 марта 2019 г., в 1:04, Heikki Linnakangas написал(а): PS. for Gist, we could almost use the LSN / NSN mechanism to detect the case that a deleted page is reused: Add a new field to the GiST page header, to store a new "deleteNSN" field. When a page

Re: GiST VACUUM

2019-03-22 Thread Andrey Borodin
> 22 марта 2019 г., в 1:04, Heikki Linnakangas написал(а): > ... > When I started testing this, I quickly noticed that empty pages were not > being deleted nearly as much as I expected. I tracked it to this check in > gistdeletepage: > >> + if (GistFollowRight(leafPage) >> +

Re: GiST VACUUM

2019-03-21 Thread Heikki Linnakangas
On 21/03/2019 18:06, Andrey Borodin wrote: 21 марта 2019 г., в 2:30, Heikki Linnakangas написал(а): one remaining issue that needs to be fixed: During Hot Standby, the B-tree code writes a WAL reord, when a deleted page is recycled, to prevent the deletion from being replayed too early in the h

Re: GiST VACUUM

2019-03-21 Thread Andrey Borodin
> 21 марта 2019 г., в 2:30, Heikki Linnakangas написал(а): > one remaining issue that needs to be fixed: > > During Hot Standby, the B-tree code writes a WAL reord, when a deleted > page is recycled, to prevent the deletion from being replayed too early > in the hot standby. See _bt_getbuf()

Re: GiST VACUUM

2019-03-20 Thread Heikki Linnakangas
On 15/03/2019 20:25, Andrey Borodin wrote: 11 марта 2019 г., в 20:03, Heikki Linnakangas написал(а): On 10/03/2019 18:40, Andrey Borodin wrote: One thing still bothers me. Let's assume that we have internal page with 2 deletable leaves. We lock these leaves in order of items on internal page.

Re: GiST VACUUM

2019-03-20 Thread Heikki Linnakangas
On 15/03/2019 20:25, Andrey Borodin wrote: 11 марта 2019 г., в 20:03, Heikki Linnakangas написал(а): On 10/03/2019 18:40, Andrey Borodin wrote: One thing still bothers me. Let's assume that we have internal page with 2 deletable leaves. We lock these leaves in order of items on internal page.

Re: GiST VACUUM

2019-03-15 Thread Jeff Janes
On Tue, Mar 5, 2019 at 8:21 AM Heikki Linnakangas wrote: > On 05/03/2019 02:26, Andrey Borodin wrote: > >> I also tried your amcheck tool with this. It did not report any > >> errors. > >> > >> Attached is also latest version of the patch itself. It is the > >> same as your latest patch v19, exce

Re: GiST VACUUM

2019-03-15 Thread Andrey Borodin
Hi! > 11 марта 2019 г., в 20:03, Heikki Linnakangas написал(а): > > On 10/03/2019 18:40, Andrey Borodin wrote: >> Here's new version of the patch for actual page deletion. >> Changes: >> 1. Fixed possible concurrency issue: >> We were locking child page while holding lock on internal page. Notes

Re: GiST VACUUM

2019-03-11 Thread Heikki Linnakangas
On 10/03/2019 18:40, Andrey Borodin wrote: Here's new version of the patch for actual page deletion. Changes: 1. Fixed possible concurrency issue: We were locking child page while holding lock on internal page. Notes in GiST README recommend locking child before parent. Thus now we unlock inter

Re: GiST VACUUM

2019-03-10 Thread Andrey Borodin
> 5 марта 2019 г., в 18:21, Heikki Linnakangas написал(а): > > On 05/03/2019 02:26, Andrey Borodin wrote: >> >> That's cool! I'll work on 2nd step of these patchset to make >> blockset data structure prettier and less hacky. > > Committed the first patch. Thanks for the patch! That's cool! Th

Re: GiST VACUUM

2019-03-05 Thread Heikki Linnakangas
On 05/03/2019 02:26, Andrey Borodin wrote: I also tried your amcheck tool with this. It did not report any errors. Attached is also latest version of the patch itself. It is the same as your latest patch v19, except for some tiny comment kibitzing. I'll mark this as Ready for Committer in the c

Re: GiST VACUUM

2019-03-04 Thread Andrey Borodin
Hi! Thanks for fixing gist amcheck! The idea of using these two patches together seems so obvious now, but never actually visited my mind before. > 4 марта 2019 г., в 18:04, Heikki Linnakangas написал(а): > Thanks! As I noted at > https://www.postgresql.org/message-id/2ff57b1f-01b4-eacf-36a2-4

Re: GiST VACUUM

2019-03-04 Thread Heikki Linnakangas
On 04/01/2019 02:47, Andrey Borodin wrote: 2 янв. 2019 г., в 20:35, Heikki Linnakangas написал(а): In patch #1, to do the vacuum scan in physical order: ... I think this is ready to be committed, except that I didn't do any testing. We discussed the need for testing earlier. Did you write some

Re: GiST VACUUM

2019-03-04 Thread Heikki Linnakangas
On 04/01/2019 21:26, Andrey Borodin wrote: 3 янв. 2019 г., в 23:47, Andrey Borodin написал(а): * Bitmapset stores 32 bit signed integers, but BlockNumber is unsigned. So this will fail with an index larger than 2^31 blocks. That's perhaps academical, I don't think anyone will try to create a

Re: GiST VACUUM

2019-02-03 Thread Michael Paquier
On Fri, Jan 04, 2019 at 06:26:18PM +0500, Andrey Borodin wrote: > Heikki, how do you think, is implementing our own radix tree for > this is viable solution? > I've written working implementation with 4-level statically typed > tree. If we follow this route, probably, there must be tests for > thi

Re: GiST VACUUM

2019-01-04 Thread Andrey Borodin
3 янв. 2019 г., в 23:47, Andrey Borodin написал(а): > >> * Bitmapset stores 32 bit signed integers, but BlockNumber is unsigned. So >> this will fail with an index larger than 2^31 blocks. That's perhaps >> academical, I don't think anyone will try to create a 16 TB GiST index any >> time soon

Re: GiST VACUUM

2019-01-03 Thread Andrey Borodin
Cool, thanks! > 2 янв. 2019 г., в 20:35, Heikki Linnakangas написал(а): > > In patch #1, to do the vacuum scan in physical order: > ... > I think this is ready to be committed, except that I didn't do any testing. > We discussed the need for testing earlier. Did you write some test scripts > f

Re: GiST VACUUM

2019-01-02 Thread Heikki Linnakangas
On 02/01/2019 17:35, Heikki Linnakangas wrote: On 28/10/2018 19:32, Andrey Borodin wrote: Hi everyone! 2 окт. 2018 г., в 6:14, Michael Paquier написал(а): Andrey, your latest patch does not apply. I am moving this to the next CF, waiting for your input. I'm doing preps for CF. Here's rebas

Re: GiST VACUUM

2019-01-02 Thread Heikki Linnakangas
On 28/10/2018 19:32, Andrey Borodin wrote: Hi everyone! 2 окт. 2018 г., в 6:14, Michael Paquier написал(а): Andrey, your latest patch does not apply. I am moving this to the next CF, waiting for your input. I'm doing preps for CF. Here's rebased version. Thanks, I had another look at thes

Re: GiST VACUUM

2018-11-30 Thread Dmitry Dolgov
> On Sun, Oct 28, 2018 at 6:32 PM Andrey Borodin wrote: > > Hi everyone! > > > 2 окт. 2018 г., в 6:14, Michael Paquier написал(а): > > Andrey, your latest patch does not apply. I am moving this to the next > > CF, waiting for your input. > > I'm doing preps for CF. > Here's rebased version. Loo

Re: GiST VACUUM

2018-10-28 Thread Andrey Borodin
Hi everyone! > 2 окт. 2018 г., в 6:14, Michael Paquier написал(а): > Andrey, your latest patch does not apply. I am moving this to the next > CF, waiting for your input. I'm doing preps for CF. Here's rebased version. Best regards, Andrey Borodin. 0002-Delete-pages-during-GiST-VACUUM-v17.pat

Re: GiST VACUUM

2018-10-01 Thread Michael Paquier
On Mon, Aug 06, 2018 at 11:12:00PM +0500, Andrey Borodin wrote: > Done. Added function bms_make_empty(int size) Andrey, your latest patch does not apply. I am moving this to the next CF, waiting for your input. -- Michael signature.asc Description: PGP signature

Re: GiST VACUUM

2018-08-06 Thread Andrey Borodin
Hi! PFA v16. > 5 авг. 2018 г., в 21:45, Andrey Borodin написал(а): >> 5 авг. 2018 г., в 16:18, Heikki Linnakangas написал(а): >> >> Hmm. A ListCell is 16 bytes, plus the AllocChunk header, 16 bytes. 32 >> bytes per internal page in total, while a bitmap consumes one bit per page, >> leaf or i

Re: GiST VACUUM

2018-08-05 Thread Andrey Borodin
Hi! > 5 авг. 2018 г., в 16:18, Heikki Linnakangas написал(а): > > On 31/07/18 23:06, Andrey Borodin wrote: >>> On a typical GiST index, what's the ratio of leaf vs. internal >>> pages? Perhaps an array would indeed be better. > > >> Typical GiST has around 200 tuples per internal page. I've swit

Re: GiST VACUUM

2018-08-05 Thread Heikki Linnakangas
On 31/07/18 23:06, Andrey Borodin wrote: On a typical GiST index, what's the ratio of leaf vs. internal pages? Perhaps an array would indeed be better. > Typical GiST has around 200 tuples per internal page. I've switched to List since it's more efficient than bitmap. Hmm. A ListCell is 16 by

Re: GiST VACUUM

2018-07-31 Thread Andrey Borodin
Hi! Thanks for looking into the patch! > 30 июля 2018 г., в 18:39, Heikki Linnakangas написал(а): > > On 29/07/18 14:47, Andrey Borodin wrote: >> Fixed both problems. PFA v14. > > Thanks, took a really quick look at this. > > The text being added to README is outdated for these latest changes.

Re: GiST VACUUM

2018-07-30 Thread Heikki Linnakangas
On 29/07/18 14:47, Andrey Borodin wrote: Fixed both problems. PFA v14. Thanks, took a really quick look at this. The text being added to README is outdated for these latest changes. In second step I still use paloc's memory, but only to store two bitmaps: bitmap of internal pages and bitmap

Re: GiST VACUUM

2018-07-29 Thread Andrey Borodin
Hi! Thank you! > 29 июля 2018 г., в 14:04, Thomas Munro > написал(а): > > On Tue, Jul 24, 2018 at 6:04 AM, Andrey Borodin wrote: >>> 21 июля 2018 г., в 17:11, Andrey Borodin написал(а): >>> <0001-Physical-GiST-scan-in-VACUUM-v13.patch> >> >> Just in case, here's second part of patch series

Re: GiST VACUUM

2018-07-29 Thread Thomas Munro
On Tue, Jul 24, 2018 at 6:04 AM, Andrey Borodin wrote: >> 21 июля 2018 г., в 17:11, Andrey Borodin написал(а): >> <0001-Physical-GiST-scan-in-VACUUM-v13.patch> > > Just in case, here's second part of patch series with actual page deletion. Hi Andrey, Cfbot says: https://ci.appveyor.com/project

Re: GiST VACUUM

2018-07-23 Thread Andrey Borodin
Hi! > 21 июля 2018 г., в 17:11, Andrey Borodin написал(а): > > <0001-Physical-GiST-scan-in-VACUUM-v13.patch> Just in case, here's second part of patch series with actual page deletion. I was considering further decreasing memory footprint by using bloom filters instead of bitmap, but it will

Re: GiST VACUUM

2018-07-21 Thread Andrey Borodin
Hi! > 19 июля 2018 г., в 23:26, Andrey Borodin написал(а): > > I'm working on triggering left split during vacuum. Will get back when done. > Thanks! Here's patch including some messy hacks to trigger NSN and FollowRight jumps during VACUUM. To trigger FollowRight GiST sometimes forget to cl

Re: GiST VACUUM

2018-07-19 Thread Andrey Borodin
> 19 июля 2018 г., в 16:28, Heikki Linnakangas написал(а): > Hmm. So, while we are scanning the right sibling, which was moved to > lower-numbered block because of a concurrent split, the original page is > split again? That's OK, we've already scanned all the tuples on the original > page,

Re: GiST VACUUM

2018-07-19 Thread Heikki Linnakangas
On 19/07/18 14:42, Andrey Borodin wrote: 19.07.2018, 15:20, "Heikki Linnakangas" : On 19/07/18 13:52, Andrey Borodin wrote: Hi! 19 июля 2018 г., в 1:12, Heikki Linnakangas mailto:hlinn...@iki.fi>> написал(а): Yeah, please, I think this is the way to go.

Re: GiST VACUUM

2018-07-19 Thread Andrey Borodin
19.07.2018, 15:20, "Heikki Linnakangas" :On 19/07/18 13:52, Andrey Borodin wrote: Hi! 19 июля 2018 г., в 1:12, Heikki Linnakangas  написал(а): Yeah, please, I think this is the way to go. Here's v11 divided into proposed steps.Thanks, one quick question: /*

Re: GiST VACUUM

2018-07-19 Thread Heikki Linnakangas
On 19/07/18 13:52, Andrey Borodin wrote: Hi! 19 июля 2018 г., в 1:12, Heikki Linnakangas написал(а): Yeah, please, I think this is the way to go. Here's v11 divided into proposed steps. Thanks, one quick question: /* We should not unlock buffer if we are going to

Re: GiST VACUUM

2018-07-19 Thread Andrey Borodin
Hi! > 19 июля 2018 г., в 1:12, Heikki Linnakangas написал(а): > > Yeah, please, I think this is the way to go. Here's v11 divided into proposed steps. In second step I still use paloc's memory, but only to store two bitmaps: bitmap of internal pages and bitmap of empty leafs. Second physical

Re: GiST VACUUM

2018-07-18 Thread Heikki Linnakangas
On 18/07/18 21:27, Andrey Borodin wrote: Hi! 18 июля 2018 г., в 16:02, Heikki Linnakangas написал(а): , but I think it would be better to split this into two patches as follows: 1st patch: Scan the index in physical rather than logical order. No attempt at deleting empty pages yet. 2nd patc

Re: GiST VACUUM

2018-07-18 Thread Andrey Borodin
Hi! > 18 июля 2018 г., в 16:02, Heikki Linnakangas написал(а): > > In the corresponding B-tree code, we use don't do actual recursion, but a > hand-optimized "tail recursion", to avoid stack overflow if there are a lot > of splits. I think we need to do something like tha there, too. I don't t

Re: GiST VACUUM

2018-07-18 Thread Heikki Linnakangas
On 17/07/18 21:41, Andrey Borodin wrote: I was checking WAL replay of new scheme to log page deletes and found a bug there (incorrect value of deleted downlink in WAL record). Here's fixed patch v10. Also I've added support to WAL identification for new record, done some improvements to comments

Re: GiST VACUUM

2018-07-17 Thread Andrey Borodin
Hi! > 16 июля 2018 г., в 21:24, Andrey Borodin написал(а): > I was checking WAL replay of new scheme to log page deletes and found a bug there (incorrect value of deleted downlink in WAL record). Here's fixed patch v10. Also I've added support to WAL identification for new record, done some

Re: GiST VACUUM

2018-07-16 Thread Andrey Borodin
> 16 июля 2018 г., в 18:58, Robert Haas написал(а): > > On Fri, Jul 13, 2018 at 10:10 AM, Heikki Linnakangas wrote: >> I'm still a bit scared about using pd_prune_xid to store the XID that >> prevents recycling the page too early. Can we use some field in >> GISTPageOpaqueData for that, simila

Re: GiST VACUUM

2018-07-16 Thread Robert Haas
On Fri, Jul 13, 2018 at 10:10 AM, Heikki Linnakangas wrote: > I'm still a bit scared about using pd_prune_xid to store the XID that > prevents recycling the page too early. Can we use some field in > GISTPageOpaqueData for that, similar to how the B-tree stores it in > BTPageOpaqueData? What's yo

Re: GiST VACUUM

2018-07-14 Thread Andrey Borodin
> 14 июля 2018 г., в 14:39, Heikki Linnakangas написал(а): > > On 14/07/18 10:26, Andrey Borodin wrote: >> This is tradeoff between complex concurrency feature and possibility >> of few dead tuples left after VACUUM. I want to understand: is it >> something dangerous in this dead tuples? > Yeah

Re: GiST VACUUM

2018-07-14 Thread Heikki Linnakangas
On 14/07/18 10:26, Andrey Borodin wrote: This is tradeoff between complex concurrency feature and possibility of few dead tuples left after VACUUM. I want to understand: is it something dangerous in this dead tuples? Yeah, that's bad. When a new heap tuple is inserted in the same location, the o

Re: GiST VACUUM

2018-07-14 Thread Andrey Borodin
> 14 июля 2018 г., в 0:28, Heikki Linnakangas написал(а): > > On 13/07/18 21:28, Andrey Borodin wrote: >>> 13 июля 2018 г., в 18:25, Heikki Linnakangas >>> написал(а): >>> Looking at the second patch, to scan the GiST index in physical >>> order, that seems totally unsafe, if there are any con

Re: GiST VACUUM

2018-07-13 Thread Heikki Linnakangas
On 13/07/18 21:28, Andrey Borodin wrote: 13 июля 2018 г., в 18:25, Heikki Linnakangas написал(а): Looking at the second patch, to scan the GiST index in physical order, that seems totally unsafe, if there are any concurrent page splits. In the logical scan, pushStackIfSplited() deals with that,

Re: GiST VACUUM

2018-07-13 Thread Andrey Borodin
> 13 июля 2018 г., в 18:10, Heikki Linnakangas написал(а): > But the situation in gistdoinsert(), where you encounter a deleted leaf page, could happen during normal operation, if vacuum runs concurrently with an insert. Insertion locks only one page at a time, as it descends

Re: GiST VACUUM

2018-07-13 Thread Heikki Linnakangas
Looking at the second patch, to scan the GiST index in physical order, that seems totally unsafe, if there are any concurrent page splits. In the logical scan, pushStackIfSplited() deals with that, by comparing the page's NSN with the parent's LSN. But I don't see anything like that in the phys

Re: GiST VACUUM

2018-07-13 Thread Heikki Linnakangas
On 13/07/18 16:41, Andrey Borodin wrote: 12 июля 2018 г., в 21:07, Andrey Borodin > написал(а): 12 июля 2018 г., в 20:40, Heikki Linnakangas > написал(а): Actually, now that I think about it more, I'm not happy with leaving orphaned pages like

Re: GiST VACUUM

2018-07-13 Thread Andrey Borodin
Hi!PFA v6 of the patch.12 июля 2018 г., в 21:07, Andrey Borodin написал(а):12 июля 2018 г., в 20:40, Heikki Linnakangas написал(а):Actually, now that I think about it more, I'm not happy with leaving orphaned pages like that behind. Let's WAL-log the removal

Re: GiST VACUUM

2018-07-12 Thread Andrey Borodin
> 12 июля 2018 г., в 20:40, Heikki Linnakangas написал(а): > > On 12/07/18 19:06, Andrey Borodin wrote: >>> 11 июля 2018 г., в 0:07, Heikki Linnakangas >>> написал(а): >>> This seems misplaced. This code deals with internal pages, and as >>> far as I can see, this patch never marks internal p

Re: GiST VACUUM

2018-07-12 Thread Heikki Linnakangas
On 12/07/18 19:06, Andrey Borodin wrote: 11 июля 2018 г., в 0:07, Heikki Linnakangas написал(а): This seems misplaced. This code deals with internal pages, and as far as I can see, this patch never marks internal pages as deleted, only leaf pages. However, we should have something like this in

Re: GiST VACUUM

2018-07-12 Thread Andrey Borodin
Hi! PFA v5 of the patch series. > 11 июля 2018 г., в 0:07, Heikki Linnakangas написал(а): > > This seems misplaced. This code deals with internal pages, and as far as I > can see, this patch never marks internal pages as deleted, only leaf pages. > However, we should have something like this

Re: GiST VACUUM

2018-07-11 Thread Andrey Borodin
Hi, Heikki! Thanks for looking into the patch! > 11 июля 2018 г., в 0:07, Heikki Linnakangas написал(а): > > I'm now looking at the first patch in this series, to allow completely empty > GiST pages to be recycled. I've got some questions: > >> --- a/src/backend/access/gist/gist.c >> +++ b/sr

Re: GiST VACUUM

2018-07-10 Thread Heikki Linnakangas
I'm now looking at the first patch in this series, to allow completely empty GiST pages to be recycled. I've got some questions: --- a/src/backend/access/gist/gist.c +++ b/src/backend/access/gist/gist.c @@ -700,6 +700,13 @@ gistdoinsert(Relation r, IndexTuple itup, Size freespace, GISTSTATE *g

Re: GiST VACUUM

2018-05-16 Thread Andrey Borodin
Hi, Thomas! > 17 мая 2018 г., в 9:40, Thomas Munro > написал(а): > > On Wed, Mar 7, 2018 at 7:50 PM, Andrey Borodin wrote: >> Here's new version of GiST VACUUM patch set aimed at CF 2018-09. > > Hi Andrey, > > FYI Windows doesn't like this patch[1]. > > + uint16_t flags; > > I think th

Re: GiST VACUUM

2018-05-16 Thread Thomas Munro
On Wed, Mar 7, 2018 at 7:50 PM, Andrey Borodin wrote: > Here's new version of GiST VACUUM patch set aimed at CF 2018-09. Hi Andrey, FYI Windows doesn't like this patch[1]. + uint16_t flags; I think that needs to be uint16? [1] https://ci.appveyor.com/project/postgresql-cfbot/postgresql/bu