Re: GIN pending list pages not recycled promptly (was Re: [HACKERS] GIN improvements part 1: additional information)

On Wed, Jan 22, 2014 at 9:12 PM, Heikki Linnakangas wrote: > On 01/22/2014 03:39 AM, Tomas Vondra wrote: >> >> What annoys me a bit is the huge size difference between the index >> updated incrementally (by a sequence of INSERT commands), and the index >> rebuilt from scratch using VACUUM FULL. It

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Jan 24, 2014 at 1:19 PM, Heikki Linnakangas wrote: > On 01/24/2014 10:53 AM, Alexander Korotkov wrote: > >> OK. What about previous fix in assert? >> > > Ah right, fixed that too now. Good, now my test-suite passed. Results are so. Time of operations event | period

Re: [HACKERS] GIN improvements part 1: additional information

On 01/24/2014 10:53 AM, Alexander Korotkov wrote: OK. What about previous fix in assert? Ah right, fixed that too now. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Jan 24, 2014 at 12:50 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 01/24/2014 10:03 AM, Alexander Korotkov wrote: > >> ITSM I found this bug. ginVacuumPostingTreeLeaf re-encodes only some >> segments. Others are not even re-palloced. They are moved left >> in dataPlaceToPa

Re: [HACKERS] GIN improvements part 1: additional information

On 01/24/2014 10:03 AM, Alexander Korotkov wrote: ITSM I found this bug. ginVacuumPostingTreeLeaf re-encodes only some segments. Others are not even re-palloced. They are moved left in dataPlaceToPageLeafRecompress by memcpy. But it's incorrect to with memcpy, proper solution is memmove. Using me

Re: [HACKERS] GIN improvements part 1: additional information

On Thu, Jan 23, 2014 at 9:27 AM, Alexander Korotkov wrote: > On Wed, Jan 22, 2014 at 9:28 PM, Heikki Linnakangas < > hlinnakan...@vmware.com> wrote: > >> On 01/22/2014 02:17 PM, Alexander Korotkov wrote: >> >>> We already spent a lot of time with compression. Now we need to figure >>> out >>> the

Re: [HACKERS] GIN improvements part 1: additional information

On Wed, Jan 22, 2014 at 9:28 PM, Heikki Linnakangas wrote: > On 01/22/2014 02:17 PM, Alexander Korotkov wrote: > >> We already spent a lot of time with compression. Now we need to figure out >> the result we want see. I spent quite long time debugging varbyte encoding >> without segments. Finally

Re: [HACKERS] GIN improvements part 1: additional information

On 01/22/2014 02:17 PM, Alexander Korotkov wrote: We already spent a lot of time with compression. Now we need to figure out the result we want see. I spent quite long time debugging varbyte encoding without segments. Finally, it passed very many tests without any problems. Now, it is just piece

Re: GIN pending list pages not recycled promptly (was Re: [HACKERS] GIN improvements part 1: additional information)

Heikki Linnakangas wrote: > I wrote a little utility that scans all pages in a gin index, and > prints out the flags indicating what kind of a page it is. The > distribution looks like this: > > 19 DATA >7420 DATA LEAF > 24701 DELETED > 1 LEAF > 1 META Hah. > I think we n

Re: [HACKERS] GIN improvements part 1: additional information

On Wed, Jan 22, 2014 at 12:30 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 01/22/2014 09:25 AM, Alexander Korotkov wrote: > >> On Wed, Jan 22, 2014 at 1:21 AM, Heikki Linnakangas < >> hlinnakan...@vmware.com >> >>> wrote: >>> >> >> On 01/21/2014 11:35 AM, Heikki Linnakangas wrote

GIN pending list pages not recycled promptly (was Re: [HACKERS] GIN improvements part 1: additional information)

On 01/22/2014 03:39 AM, Tomas Vondra wrote: What annoys me a bit is the huge size difference between the index updated incrementally (by a sequence of INSERT commands), and the index rebuilt from scratch using VACUUM FULL. It's a bit better with the patch (2288 vs. 2035 MB), but is there a chance

Re: [HACKERS] GIN improvements part 1: additional information

On 01/22/2014 09:25 AM, Alexander Korotkov wrote: On Wed, Jan 22, 2014 at 1:21 AM, Heikki Linnakangas wrote: On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: Oh, I see what's going on. I had assumed that there cannot be duplicate insertions into the posting tree, but that's dead wrong. The

Re: [HACKERS] GIN improvements part 1: additional information

On Wed, Jan 22, 2014 at 1:21 AM, Heikki Linnakangas wrote: > On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: > >> On 01/21/2014 04:02 AM, Tomas Vondra wrote: >> >>> On 20.1.2014 19:30, Heikki Linnakangas wrote: >>> Attached is a yet another version, with more bugs fixed and more

Re: [HACKERS] GIN improvements part 1: additional information

On 21.1.2014 22:21, Heikki Linnakangas wrote: > On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: >> On 01/21/2014 04:02 AM, Tomas Vondra wrote: >>> On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more comments added and updat

Re: [HACKERS] GIN improvements part 1: additional information

On 01/21/2014 11:35 AM, Heikki Linnakangas wrote: On 01/21/2014 04:02 AM, Tomas Vondra wrote: On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more comments added and updated. I would appreciate some heavy-testing of this patch now. If y

Re: [HACKERS] GIN improvements part 1: additional information

On Tue, Jan 21, 2014 at 4:28 PM, Alexander Korotkov wrote: > I noticed that the gin vacuum redo routine is dead code, except for the >> data-leaf page handling, because we never remove entries or internal nodes >> (page deletion is a separate wal record type). And the data-leaf case is >> function

Re: [HACKERS] GIN improvements part 1: additional information

On Mon, Jan 20, 2014 at 10:30 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 01/17/2014 08:49 PM, Alexander Korotkov wrote: > >> On Fri, Jan 17, 2014 at 10:38 PM, Heikki Linnakangas < >> hlinnakan...@vmware.com> wrote: >> >> On 01/17/2014 01:05 PM, Alexander Korotkov wrote: >>> >>>

Re: [HACKERS] GIN improvements part 1: additional information

On 01/21/2014 04:02 AM, Tomas Vondra wrote: On 20.1.2014 19:30, Heikki Linnakangas wrote: Attached is a yet another version, with more bugs fixed and more comments added and updated. I would appreciate some heavy-testing of this patch now. If you could re-run the tests you've been using, that c

Re: [HACKERS] GIN improvements part 1: additional information

On 20.1.2014 19:30, Heikki Linnakangas wrote: > > Attached is a yet another version, with more bugs fixed and more > comments added and updated. I would appreciate some heavy-testing of > this patch now. If you could re-run the tests you've been using, > that could be great. I've tested the WAL

Re: [HACKERS] GIN improvements part 1: additional information

On 01/17/2014 08:49 PM, Alexander Korotkov wrote: On Fri, Jan 17, 2014 at 10:38 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: On 01/17/2014 01:05 PM, Alexander Korotkov wrote: Seems to be fixed in the attached version of patch. Another improvement in this version: only left-most se

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Jan 17, 2014 at 10:38 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 01/17/2014 01:05 PM, Alexander Korotkov wrote: > >> Seems to be fixed in the attached version of patch. >> Another improvement in this version: only left-most segments and further >> are re-encoded. Left pa

Re: [HACKERS] GIN improvements part 1: additional information

On 01/17/2014 01:05 PM, Alexander Korotkov wrote: Seems to be fixed in the attached version of patch. Another improvement in this version: only left-most segments and further are re-encoded. Left part of page are left untouched. I'm looking into this now. A few quick notes: * Even when you don

Re: [HACKERS] GIN improvements part 1: additional information

On Wed, Jan 15, 2014 at 10:47 AM, Alexander Korotkov wrote: > On Wed, Jan 15, 2014 at 5:17 AM, Tomas Vondra wrote: > >> On 14.1.2014 00:38, Tomas Vondra wrote: >> > On 13.1.2014 18:07, Alexander Korotkov wrote: >> >> On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra > >> > wrote

Re: [HACKERS] GIN improvements part 1: additional information

On Wed, Jan 15, 2014 at 5:17 AM, Tomas Vondra wrote: > On 14.1.2014 00:38, Tomas Vondra wrote: > > On 13.1.2014 18:07, Alexander Korotkov wrote: > >> On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra >> > wrote: > >> > >> On 8.1.2014 22:58, Alexander Korotkov wrote: > >>

Re: [HACKERS] GIN improvements part 1: additional information

On 14.1.2014 00:38, Tomas Vondra wrote: > On 13.1.2014 18:07, Alexander Korotkov wrote: >> On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra > > wrote: >> >> On 8.1.2014 22:58, Alexander Korotkov wrote: >> > Thanks for reporting. Fixed version is attached. >> >> I've t

Re: [HACKERS] GIN improvements part 1: additional information

On Tue, Jan 14, 2014 at 12:34 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 01/13/2014 07:07 PM, Alexander Korotkov wrote: > >> I've fixed this bug and many other bug. Now patch passes test suite that >> I've used earlier. The results are so: >> >> Operations time: >> eve

Re: [HACKERS] GIN improvements part 1: additional information

On 01/13/2014 07:07 PM, Alexander Korotkov wrote: I've fixed this bug and many other bug. Now patch passes test suite that I've used earlier. The results are so: Operations time: event | period ---+- index_build | 00:01:47.539

Re: [HACKERS] GIN improvements part 1: additional information

Peter Eisentraut wrote: > On 12/10/13, 2:44 PM, Alexander Korotkov wrote: > > However, patch didn't apply to head. Corrected version is attached. > Update the pgstattuple regression test results. The latest version of the patch still doesn't pass the test. I'll look at the patch in further detai

Re: [HACKERS] GIN improvements part 1: additional information

On 13.1.2014 18:07, Alexander Korotkov wrote: > On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra > wrote: > > On 8.1.2014 22:58, Alexander Korotkov wrote: > > Thanks for reporting. Fixed version is attached. > > I've tried to rerun the 'archie' benchmark with the c

Re: [HACKERS] GIN improvements part 1: additional information

On Sat, Jan 11, 2014 at 6:15 AM, Tomas Vondra wrote: > On 8.1.2014 22:58, Alexander Korotkov wrote: > > Thanks for reporting. Fixed version is attached. > > I've tried to rerun the 'archie' benchmark with the current patch, and > once again I got > >PANIC: could not split GIN page, didn't fi

Re: [HACKERS] GIN improvements part 1: additional information

On 8.1.2014 22:58, Alexander Korotkov wrote: > Thanks for reporting. Fixed version is attached. I've tried to rerun the 'archie' benchmark with the current patch, and once again I got PANIC: could not split GIN page, didn't fit I reran it with '--enable-cassert' and with that I got TRAP: Fa

Re: [HACKERS] GIN improvements part 1: additional information

On Mon, Jan 6, 2014 at 12:35 PM, Amit Langote wrote: > On Sat, Dec 21, 2013 at 4:36 AM, Heikki Linnakangas > wrote: > > > > Yet another version. The encoding/decoding code is now quite isolated in > > ginpostinglist.c, so it's easy to experiment with different encodings. > This > > patch uses var

Re: [HACKERS] GIN improvements part 1: additional information

On Sat, Dec 21, 2013 at 4:36 AM, Heikki Linnakangas wrote: > > Yet another version. The encoding/decoding code is now quite isolated in > ginpostinglist.c, so it's easy to experiment with different encodings. This > patch uses varbyte encoding again. > > I got a bit carried away, experimented with

Re: [HACKERS] GIN improvements part 1: additional information

On 12/10/13, 2:44 PM, Alexander Korotkov wrote: > However, patch didn't apply to head. Corrected version is attached. Update the pgstattuple regression test results. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresq

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Dec 20, 2013 at 11:36 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 12/19/2013 03:33 PM, Heikki Linnakangas wrote: > >> On 12/17/2013 12:49 AM, Heikki Linnakangas wrote: >> >>> On 12/17/2013 12:22 AM, Alexander Korotkov wrote: >>> On Mon, Dec 16, 2013 at 3:30 PM, Heikk

Re: [HACKERS] GIN improvements part 1: additional information

Alexander Korotkov escribió: > On Fri, Dec 20, 2013 at 11:43 PM, Alvaro Herrera > wrote: > > > Heikki Linnakangas escribió: > > > > > I believe that eliminates all encodings in the Simple family, as > > > well as PForDelta, and surprisingly also Rice encoding. For example, > > > if you have three

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Dec 20, 2013 at 11:43 PM, Alvaro Herrera wrote: > Heikki Linnakangas escribió: > > > I believe that eliminates all encodings in the Simple family, as > > well as PForDelta, and surprisingly also Rice encoding. For example, > > if you have three items in consecutive offsets, the differences

Re: [HACKERS] GIN improvements part 1: additional information

Heikki Linnakangas escribió: > I believe that eliminates all encodings in the Simple family, as > well as PForDelta, and surprisingly also Rice encoding. For example, > if you have three items in consecutive offsets, the differences > between them are encoded as 11 in rice encoding. If you remove

Re: [HACKERS] GIN improvements part 1: additional information

On 12/19/2013 03:33 PM, Heikki Linnakangas wrote: On 12/17/2013 12:49 AM, Heikki Linnakangas wrote: On 12/17/2013 12:22 AM, Alexander Korotkov wrote: On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas wrote: On 12/12/2013 06:44 PM, Alexander Korotkov wrote: When values are packed into s

Re: [HACKERS] GIN improvements part 1: additional information

On 12/19/2013 10:44 AM, Heikki Linnakangas wrote: On 12/19/2013 08:37 AM, Oleg Bartunov wrote: Guys, before digging deep into the art of comp/decomp world I'd like to know if you familiar with results of http://wwwconference.org/www2008/papers/pdf/p387-zhangA.pdf paper and some newer research ?

Re: [HACKERS] GIN improvements part 1: additional information

On 12/17/2013 12:49 AM, Heikki Linnakangas wrote: On 12/17/2013 12:22 AM, Alexander Korotkov wrote: On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas wrote: On 12/12/2013 06:44 PM, Alexander Korotkov wrote: When values are packed into small groups, we have to either insert inefficientl

Re: [HACKERS] GIN improvements part 1: additional information

On 12/19/2013 08:37 AM, Oleg Bartunov wrote: Guys, before digging deep into the art of comp/decomp world I'd like to know if you familiar with results of http://wwwconference.org/www2008/papers/pdf/p387-zhangA.pdf paper and some newer research ? Yeah, I saw that paper. Do we agree in what we

Re: [HACKERS] GIN improvements part 1: additional information

Guys, before digging deep into the art of comp/decomp world I'd like to know if you familiar with results of http://wwwconference.org/www2008/papers/pdf/p387-zhangA.pdf paper and some newer research ? Do we agree in what we really want ? Basically, there are three main features: size, compression

Re: [HACKERS] GIN improvements part 1: additional information

On 12/18/2013 01:45 PM, Alexander Korotkov wrote: On Tue, Dec 17, 2013 at 2:49 AM, Heikki Linnakangas wrote: On 12/17/2013 12:22 AM, Alexander Korotkov wrote: 2) Storage would be easily extendable to hold additional information as well. Better compression shouldn't block more serious impro

Re: [HACKERS] GIN improvements part 1: additional information

On Tue, Dec 17, 2013 at 2:49 AM, Heikki Linnakangas wrote: > On 12/17/2013 12:22 AM, Alexander Korotkov wrote: > >> On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas < >> hlinnakan...@vmware.com >> >>> wrote: >>> >> >> On 12/12/2013 06:44 PM, Alexander Korotkov wrote: >>> >>> When values are

Re: [HACKERS] GIN improvements part 1: additional information

On 12/17/2013 12:22 AM, Alexander Korotkov wrote: On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas wrote: On 12/12/2013 06:44 PM, Alexander Korotkov wrote: When values are packed into small groups, we have to either insert inefficiently encoded value or re-encode whole right part of va

Re: [HACKERS] GIN improvements part 1: additional information

On Mon, Dec 16, 2013 at 3:30 PM, Heikki Linnakangas wrote: > On 12/12/2013 06:44 PM, Alexander Korotkov wrote: > >> I've thought about different algorithms little more. General problem I see >> is online update. We need it while it is typically not covered by >> researches at all. We already have

Re: [HACKERS] GIN improvements part 1: additional information

On 12/12/2013 06:44 PM, Alexander Korotkov wrote: I've thought about different algorithms little more. General problem I see is online update. We need it while it is typically not covered by researches at all. We already have to invent small index in the end of page. Different encoding methods ad

Re: [HACKERS] GIN improvements part 1: additional information

On Tue, Dec 10, 2013 at 12:26 AM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 12/09/2013 11:34 AM, Alexander Korotkov wrote: > >> On Mon, Dec 9, 2013 at 1:18 PM, Heikki Linnakangas >> wrote: >> >> Even if we use varbyte encoding, I wonder if it would be better to treat >>> block + o

Re: [HACKERS] GIN improvements part 1: additional information

On Tue, Dec 10, 2013 at 12:26 AM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 12/09/2013 11:34 AM, Alexander Korotkov wrote: > >> On Mon, Dec 9, 2013 at 1:18 PM, Heikki Linnakangas >> wrote: >> >> Even if we use varbyte encoding, I wonder if it would be better to treat >>> block + o

Re: [HACKERS] GIN improvements part 1: additional information

On 12/09/2013 11:34 AM, Alexander Korotkov wrote: On Mon, Dec 9, 2013 at 1:18 PM, Heikki Linnakangas wrote: Even if we use varbyte encoding, I wonder if it would be better to treat block + offset number as a single 48-bit integer, rather than encode them separately. That would allow the delta o

Re: [HACKERS] GIN improvements part 1: additional information

On Mon, Dec 9, 2013 at 1:18 PM, Heikki Linnakangas wrote: > On 12/08/2013 09:56 PM, Alexander Korotkov wrote: > >> On Fri, Nov 29, 2013 at 11:17 PM, Heikki Linnakangas < >> hlinnakan...@vmware.com> wrote: >> >> I'll continue reviewing next week.. >>> >> > Got dragged into other things and didn't

Re: [HACKERS] GIN improvements part 1: additional information

On 12/08/2013 09:56 PM, Alexander Korotkov wrote: On Fri, Nov 29, 2013 at 11:17 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: I'll continue reviewing next week.. Got dragged into other things and didn't make any progress on this last week. I'm trying again now. Good. Thanks for

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Nov 29, 2013 at 11:17 PM, Heikki Linnakangas < hlinnakan...@vmware.com> wrote: > On 11/29/2013 11:41 AM, Heikki Linnakangas wrote: > >> On 11/28/2013 09:19 AM, Alexander Korotkov wrote: >> >>> On Wed, Nov 27, 2013 at 1:14 AM, Heikki Linnakangas >>> >> wrote: >>> >>> On 11/26/13

Re: [HACKERS] GIN improvements part 1: additional information

On 11/29/2013 11:41 AM, Heikki Linnakangas wrote: On 11/28/2013 09:19 AM, Alexander Korotkov wrote: On Wed, Nov 27, 2013 at 1:14 AM, Heikki Linnakangas wrote: On 11/26/13 15:34, Alexander Korotkov wrote: What's your plans about GIN now? I tried to rebase packed posting lists with head. But

Re: [HACKERS] GIN improvements part 1: additional information

On Wed, Nov 27, 2013 at 1:14 AM, Heikki Linnakangas wrote: > On 11/26/13 15:34, Alexander Korotkov wrote: > >> What's your plans about GIN now? I tried to rebase packed posting lists >> with head. But I found that you've changed interface of placeToPage >> function. That's conflicts with packed p

Re: [HACKERS] GIN improvements part 1: additional information

This patch needs to be rebased. > > -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] GIN improvements part 1: additional information

On 11/26/13 15:34, Alexander Korotkov wrote: What's your plans about GIN now? I tried to rebase packed posting lists with head. But I found that you've changed interface of placeToPage function. That's conflicts with packed posting lists, because dataPlaceToPageLeaf needs not only offset number t

Re: [HACKERS] GIN improvements part 1: additional information

On Tue, Nov 26, 2013 at 5:34 PM, Alexander Korotkov wrote: > On Wed, Nov 20, 2013 at 9:02 PM, Heikki Linnakangas < > hlinnakan...@vmware.com> wrote: > >> On 06.11.2013 17:36, Alvaro Herrera wrote: >> >>> Just for my own illumination, can someone explain this bit? >>> >>> + If a posting list is too

Re: [HACKERS] GIN improvements part 1: additional information

Hi! On Wed, Nov 20, 2013 at 9:02 PM, Heikki Linnakangas wrote: > On 06.11.2013 17:36, Alvaro Herrera wrote: > >> Just for my own illumination, can someone explain this bit? >> >> + If a posting list is too large to store in-line in a key entry, a >> posting tree >> + is created. A posting tree i

Re: [HACKERS] GIN improvements part 1: additional information

On 06.11.2013 17:36, Alvaro Herrera wrote: Just for my own illumination, can someone explain this bit? + If a posting list is too large to store in-line in a key entry, a posting tree + is created. A posting tree is a B-tree structure, where the ItemPointer is + used as the key. At the leaf-leve

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Nov 15, 2013 at 9:56 PM, Peter Eisentraut wrote: > On 11/15/13, 12:24 PM, Alexander Korotkov wrote: > > On Fri, Nov 15, 2013 at 8:58 PM, Peter Eisentraut > > wrote: > > > > On 11/14/13, 6:00 AM, Alexander Korotkov wrote: > > > Sorry, I posted buggy version

Re: [HACKERS] GIN improvements part 1: additional information

On 11/15/13, 12:24 PM, Alexander Korotkov wrote: > On Fri, Nov 15, 2013 at 8:58 PM, Peter Eisentraut > wrote: > > On 11/14/13, 6:00 AM, Alexander Korotkov wrote: > > Sorry, I posted buggy version of patch. Attached version is correct. > > This patch crashes th

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Nov 15, 2013 at 8:58 PM, Peter Eisentraut wrote: > On 11/14/13, 6:00 AM, Alexander Korotkov wrote: > > Sorry, I posted buggy version of patch. Attached version is correct. > > This patch crashes the hstore the pg_trgm regression tests. > What exact version did you try 14 or 16? If it was

Re: [HACKERS] GIN improvements part 1: additional information

On 11/14/13, 6:00 AM, Alexander Korotkov wrote: > Sorry, I posted buggy version of patch. Attached version is correct. This patch crashes the hstore the pg_trgm regression tests. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://

Re: [HACKERS] GIN improvements part 1: additional information

On Thu, Nov 14, 2013 at 3:00 PM, Alexander Korotkov wrote: > On Thu, Nov 14, 2013 at 2:17 PM, Alexander Korotkov > wrote: > >> On Tue, Nov 5, 2013 at 9:49 PM, Heikki Linnakangas < >> hlinnakan...@vmware.com> wrote: >> >>> On 04.11.2013 23:44, Alexander Korotkov wrote: >>> On Mon, Oct 21, 201

Re: [HACKERS] GIN improvements part 1: additional information

On Thu, Nov 14, 2013 at 2:17 PM, Alexander Korotkov wrote: > On Tue, Nov 5, 2013 at 9:49 PM, Heikki Linnakangas < > hlinnakan...@vmware.com> wrote: > >> On 04.11.2013 23:44, Alexander Korotkov wrote: >> >>> On Mon, Oct 21, 2013 at 11:12 PM, Alexander Korotkov >>> wrote: >>> >>> Attached version o

Re: [HACKERS] GIN improvements part 1: additional information

On Tue, Nov 5, 2013 at 9:49 PM, Heikki Linnakangas wrote: > On 04.11.2013 23:44, Alexander Korotkov wrote: > >> On Mon, Oct 21, 2013 at 11:12 PM, Alexander Korotkov >> wrote: >> >> Attached version of patch is debugged. I would like to note that number >>> of >>> bugs was low and it wasn't very h

Re: [HACKERS] GIN improvements part 1: additional information

Heikki Linnakangas escribió: > On 04.11.2013 23:44, Alexander Korotkov wrote: > >Attached version of patch has support of old page format. Patch still needs > >more documentation and probably refactoring, but I believe idea is clear > >and it can be reviewed. In the patch I have to revert change of

Re: [HACKERS] GIN improvements part 1: additional information

On 04.11.2013 23:44, Alexander Korotkov wrote: Attached version of patch has support of old page format. Patch still needs more documentation and probably refactoring, but I believe idea is clear and it can be reviewed. In the patch I have to revert change of null category placement for compatibi

Re: [HACKERS] GIN improvements part 1: additional information

On 04.11.2013 23:44, Alexander Korotkov wrote: On Mon, Oct 21, 2013 at 11:12 PM, Alexander Korotkov wrote: Attached version of patch is debugged. I would like to note that number of bugs was low and it wasn't very hard to debug. I've rerun tests on it. You can see that numbers are improved as t

Re: [HACKERS] GIN improvements part 1: additional information

On Mon, Oct 21, 2013 at 11:12 PM, Alexander Korotkov wrote: > Attached version of patch is debugged. I would like to note that number of > bugs was low and it wasn't very hard to debug. I've rerun tests on it. You > can see that numbers are improved as the result of your refactoring. > >

Re: [HACKERS] GIN improvements part 1: additional information

On Thu, Oct 10, 2013 at 3:57 PM, Heikki Linnakangas wrote: > On 09.10.2013 02:04, Tomas Vondra wrote: > >> On 8.10.2013 21:59, Heikki Linnakangas wrote: >> >>> On 08.10.2013 17:47, Alexander Korotkov wrote: >>> Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondra wrote: >

Re: [HACKERS] GIN improvements part 1: additional information

On Sat, Oct 19, 2013 at 05:11:55PM -0400, Bruce Momjian wrote: > I am in Moscow with Alexander and we were discussing GIN pg_upgrade > issues. One option we have already discussed was to take the old GIN > index code and put it in a separate directory, and call the new GIN > index something differ

Re: [HACKERS] GIN improvements part 1: additional information

On Thu, Oct 3, 2013 at 05:24:49PM -0400, Bruce Momjian wrote: > On Thu, Oct 3, 2013 at 02:48:20PM -0400, Robert Haas wrote: > > On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas > > wrote: > > > It seems we've all but decided that we'll require reindexing GIN indexes > > > in > > > 9.4. > > >

Re: [HACKERS] GIN improvements part 1: additional information

On 12.10.2013 12:11, Alexander Korotkov wrote: > On Sat, Oct 12, 2013 at 1:55 AM, Tomas Vondra > wrote: > > Yup, this version fixed the issues. I haven't been able to do any > benchmarks yet, all I have is some basic stats > >| HEAD | patche

Re: [HACKERS] GIN improvements part 1: additional information

On Sat, Oct 12, 2013 at 1:55 AM, Tomas Vondra wrote: > On 10.10.2013 13:57, Heikki Linnakangas wrote: > > On 09.10.2013 02:04, Tomas Vondra wrote: > >> On 8.10.2013 21:59, Heikki Linnakangas wrote: > >>> On 08.10.2013 17:47, Alexander Korotkov wrote: > Hi, Tomas! > > On Sun, Oct 6,

Re: [HACKERS] GIN improvements part 1: additional information

On 10.10.2013 13:57, Heikki Linnakangas wrote: > On 09.10.2013 02:04, Tomas Vondra wrote: >> On 8.10.2013 21:59, Heikki Linnakangas wrote: >>> On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondra wrote: > I've attempted to

Re: [HACKERS] GIN improvements part 1: additional information

You linked to this email from the commitfest entry, but there is no patch here. You probably meant a different email. Check please. On Tue, 2013-10-08 at 21:48 +0300, Heikki Linnakangas wrote: > On 04.10.2013 14:13, Alexander Korotkov wrote: > > On Fri, Oct 4, 2013 at 12:28 PM, Heikki Linnakang

Re: [HACKERS] GIN improvements part 1: additional information

On 09.10.2013 02:04, Tomas Vondra wrote: On 8.10.2013 21:59, Heikki Linnakangas wrote: On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondra wrote: I've attempted to rerun the benchmarks tests I did a few weeks ago, but I got repeated cra

Re: [HACKERS] GIN improvements part 1: additional information

On 8.10.2013 21:59, Heikki Linnakangas wrote: > On 08.10.2013 17:47, Alexander Korotkov wrote: >> Hi, Tomas! >> >> On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondra wrote: >> >>> I've attempted to rerun the benchmarks tests I did a few weeks ago, but >>> I got repeated crashes when loading the data (

Re: [HACKERS] GIN improvements part 1: additional information

On 08.10.2013 17:47, Alexander Korotkov wrote: Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondra wrote: I've attempted to rerun the benchmarks tests I did a few weeks ago, but I got repeated crashes when loading the data (into a table with tsvector+gin index). Right before a crash, t

Re: [HACKERS] GIN improvements part 1: additional information

On 04.10.2013 14:13, Alexander Korotkov wrote: On Fri, Oct 4, 2013 at 12:28 PM, Heikki Linnakangas wrote: In the attached patch, I in fact already did that for data leaf pages, but didn't change the format of non-leaf pages yet. If we want to support pg_upgrade, we might want to refrain from c

Re: [HACKERS] GIN improvements part 1: additional information

Hi, Tomas! On Sun, Oct 6, 2013 at 3:58 AM, Tomas Vondra wrote: > I've attempted to rerun the benchmarks tests I did a few weeks ago, but > I got repeated crashes when loading the data (into a table with > tsvector+gin index). > > Right before a crash, theres this message in the log: > >PANI

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Oct 4, 2013 at 12:28 PM, Heikki Linnakangas wrote: > Aside from the pg_upgrade discussion, here's an updated version of the > patch, rebased over master. It also contains some other misc refactoring > I've done while reading through the patch. I haven't tested this much, I > may well have

Re: [HACKERS] GIN improvements part 1: additional information

Aside from the pg_upgrade discussion, here's an updated version of the patch, rebased over master. It also contains some other misc refactoring I've done while reading through the patch. I haven't tested this much, I may well have also broken something, but I wanted to post an update before the

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Oct 4, 2013 at 02:23:33AM +0400, Alexander Korotkov wrote: > I came to idea that I like option #4 more than option #2. > If we try to make new GIN work with old page formats we have to maintain 3 use > cases: > 1) old GIN with old page format (because of old releases) > 2) new GIN with old

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Oct 4, 2013 at 12:37 AM, Alexander Korotkov wrote: > On Thu, Oct 3, 2013 at 10:48 PM, Robert Haas wrote: > >> On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas >> wrote: >> > It seems we've all but decided that we'll require reindexing GIN >> indexes in >> > 9.4. >> >> I thought the cons

Re: [HACKERS] GIN improvements part 1: additional information

Bruce Momjian escribió: > Agreed. I was stating only that this is easy for pg_upgrade. One cool > thing is that the upgrades completes, and the indexes are there, but > just marked as invalid until the REINDEX. > > One other point Alexander made is that the new GIN indexes will be > smaller so

Re: [HACKERS] GIN improvements part 1: additional information

On 03.10.2013 23:54, Alexander Korotkov wrote: ItemPointers compression reduce occupied space in all normal cases. It's not very realistic, but it could increase space in worst case. That would lead to page split after conversion. Are we going to support such case? Hmm, that's probably rare eno

Re: [HACKERS] GIN improvements part 1: additional information

On Thu, Oct 3, 2013 at 02:48:20PM -0400, Robert Haas wrote: > On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas > wrote: > > It seems we've all but decided that we'll require reindexing GIN indexes in > > 9.4. > > I thought the consensus in Ottawa was strongly against that. I'm not > aware tha

Re: [HACKERS] GIN improvements part 1: additional information

On Fri, Oct 4, 2013 at 12:41 AM, Heikki Linnakangas wrote: > On 03.10.2013 23:37, Alexander Korotkov wrote: > >> 2) Insert kluges into GIN to support both old and new formats. So, kluges >> are kluges :) I don't see elegant way to do it for now, because formats >> are >> very different. >> > > Hm

Re: [HACKERS] GIN improvements part 1: additional information

On 03.10.2013 23:37, Alexander Korotkov wrote: 2) Insert kluges into GIN to support both old and new formats. So, kluges are kluges :) I don't see elegant way to do it for now, because formats are very different. Hmm. All you need is some code to read the old format, and a function to convert

Re: [HACKERS] GIN improvements part 1: additional information

On Thu, Oct 3, 2013 at 10:48 PM, Robert Haas wrote: > On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas > wrote: > > It seems we've all but decided that we'll require reindexing GIN indexes > in > > 9.4. > > I thought the consensus in Ottawa was strongly against that. I'm not > aware that anyo

Re: [HACKERS] GIN improvements part 1: additional information

On Thu, Oct 3, 2013 at 2:43 PM, Heikki Linnakangas wrote: > It seems we've all but decided that we'll require reindexing GIN indexes in > 9.4. I thought the consensus in Ottawa was strongly against that. I'm not aware that anyone has subsequently changed their position on the topic. Bruce is ri

Re: [HACKERS] GIN improvements part 1: additional information

On 23.09.2013 18:35, Bruce Momjian wrote: On Sun, Sep 15, 2013 at 01:14:45PM +0400, Alexander Korotkov wrote: On Sat, Jun 29, 2013 at 12:56 PM, Heikki Linnakangas wrote: There's a few open questions: 1. How are we going to handle pg_upgrade? It would be nice to be able to read t

Re: [HACKERS] GIN improvements part 1: additional information

On Wed, Sep 25, 2013 at 5:22 PM, Peter Eisentraut wrote: > On 9/23/13 5:36 PM, Alexander Korotkov wrote: > > In the attached version of patch double finding of ItemPointer during > > insert is avoided. Overhead becomes lower as expected. > > Fails cpluspluscheck: > > ./src/include/access/gin_priv

Re: [HACKERS] GIN improvements part 1: additional information

On 9/23/13 5:36 PM, Alexander Korotkov wrote: > In the attached version of patch double finding of ItemPointer during > insert is avoided. Overhead becomes lower as expected. Fails cpluspluscheck: ./src/include/access/gin_private.h: In function ‘char* ginDataPageLeafReadItemPointer(char*, ItemPoi

Re: [HACKERS] GIN improvements part 1: additional information

On Mon, Sep 23, 2013 at 12:47 AM, Alexander Korotkov wrote: > It's probably an option to select 64 entries instead of 32. > There is still some regression in update speed. However, there is also > room for improvement patch. It searches item index entries 2 times on > insert: in dataLocateLeafItem

Re: [HACKERS] GIN improvements part 1: additional information

On Sun, Sep 15, 2013 at 01:14:45PM +0400, Alexander Korotkov wrote: > On Sat, Jun 29, 2013 at 12:56 PM, Heikki Linnakangas > wrote: > > There's a few open questions: > > 1. How are we going to handle pg_upgrade? It would be nice to be able to > read the old page format, or convert on

  1   2   >