Re: [GENERAL] still gin index creation takes forever

2008-11-18 Thread Ivan Sergio Borgonovo
On Thu, 13 Nov 2008 09:11:05 -0500 Tom Lane <[EMAIL PROTECTED]> wrote: > Teodor Sigaev <[EMAIL PROTECTED]> writes: > >> Yeah, I'm not convinced either. Still, Teodor's theory should > >> be easily testable: set synchronize_seqscans to FALSE and see > >> if the problem goes away. > > > Test suit

Re: [GENERAL] still gin index creation takes forever

2008-11-13 Thread Teodor Sigaev
changing it; I've applied a patch for that. I'm still not quite convinced that Ivan isn't seeing some other issue though. Thank you In the meantime, I noticed something odd while experimenting with your test case: when running with default maintenance_work_mem = 16MB, there is a slowdown of 3

Re: [GENERAL] still gin index creation takes forever

2008-11-13 Thread Tom Lane
Teodor Sigaev <[EMAIL PROTECTED]> writes: >> We could extend IndexBuildHeapScan's API to support that, but I'm >> not quite convinced that this is the issue. > That extension might be useful for bitmap index too to simplify index > creation > process. Maybe, but in any case the measurable GIN s

Re: [GENERAL] still gin index creation takes forever

2008-11-13 Thread Ivan Sergio Borgonovo
On Thu, 13 Nov 2008 09:11:05 -0500 Tom Lane <[EMAIL PROTECTED]> wrote: > Teodor Sigaev <[EMAIL PROTECTED]> writes: > >> Yeah, I'm not convinced either. Still, Teodor's theory should > >> be easily testable: set synchronize_seqscans to FALSE and see > >> if the problem goes away. > > > Test suit

Re: [GENERAL] still gin index creation takes forever

2008-11-13 Thread Tom Lane
Teodor Sigaev <[EMAIL PROTECTED]> writes: >> Yeah, I'm not convinced either. Still, Teodor's theory should be easily >> testable: set synchronize_seqscans to FALSE and see if the problem goes >> away. > Test suit to reproduce the problem: I don't doubt that you're describing a real effect, I'm j

Re: [GENERAL] still gin index creation takes forever

2008-11-13 Thread Teodor Sigaev
We could extend IndexBuildHeapScan's API to support that, but I'm not quite convinced that this is the issue. That extension might be useful for bitmap index too to simplify index creation process. -- Teodor Sigaev E-mail: [EMAIL PROTECTED]

Re: [GENERAL] still gin index creation takes forever

2008-11-13 Thread Teodor Sigaev
Yeah, I'm not convinced either. Still, Teodor's theory should be easily testable: set synchronize_seqscans to FALSE and see if the problem goes away. Test suit to reproduce the problem: DROP TABLE IF EXISTS foo; DROP TABLE IF EXISTS footmp; CREATE OR REPLACE FUNCTION ge

Re: [GENERAL] still gin index creation takes forever

2008-11-13 Thread Tom Lane
Ivan Sergio Borgonovo <[EMAIL PROTECTED]> writes: > Tom Lane <[EMAIL PROTECTED]> wrote: >> I see. So this could explain Ivan's issue if his table contains >> large numbers of repeated GIN keys. Ivan, is that what your data >> looks like? > Well if by GIN keys you mean lexemes it could be. But I

Re: [GENERAL] still gin index creation takes forever

2008-11-13 Thread Ivan Sergio Borgonovo
On Wed, 12 Nov 2008 15:18:05 -0500 Tom Lane <[EMAIL PROTECTED]> wrote: > > So, in that case process can insert about 1000 ItemPointers per > > one data tree lookup, in opposite case it does 1000 lookups in > > data tree. > I see. So this could explain Ivan's issue if his table contains > large n

Re: [GENERAL] still gin index creation takes forever

2008-11-12 Thread Tom Lane
Teodor Sigaev <[EMAIL PROTECTED]> writes: >> I'm not following. Rightmost page of what --- it can't be the whole >> index, can it, or the case would hardly ever apply? > GIN's index contains btree over keys (entry tree) and for each key it > contains list of ItemPointers (posting list) or btree

Re: [GENERAL] still gin index creation takes forever

2008-11-12 Thread Teodor Sigaev
GIN's build algorithm could use bulk insert of ItemPointers if and only if they should be inserted on rightmost page (exact piece of code - dataPlaceToPage() in gindatapage.c, lines 407-427) I'm not following. Rightmost page of what --- it can't be the whole index, can it, or the case would ha

Re: [GENERAL] still gin index creation takes forever

2008-11-12 Thread Tom Lane
Teodor Sigaev <[EMAIL PROTECTED]> writes: > GIN's build algorithm could use bulk insert of ItemPointers if and only if > they > should be inserted on rightmost page (exact piece of code - dataPlaceToPage() > in > gindatapage.c, lines 407-427) I'm not following. Rightmost page of what --- it c

Re: [GENERAL] still gin index creation takes forever

2008-11-12 Thread Teodor Sigaev
Any suggestion about how to track down the problem? What you are describing sounds rather like a use-of-uninitialized-memory problem, wherein the behavior depends on what happened to be in that memory previously. If so, using a debug/cassert-enabled build of Postgres might help to make the beha

Re: [GENERAL] still gin index creation takes forever

2008-11-12 Thread Ivan Sergio Borgonovo
On Tue, 11 Nov 2008 22:02:17 -0500 Tom Lane <[EMAIL PROTECTED]> wrote: > Ivan Sergio Borgonovo <[EMAIL PROTECTED]> writes: > > Any suggestion about how to track down the problem? > > What you are describing sounds rather like a > use-of-uninitialized-memory problem, wherein the behavior depends >

Re: [GENERAL] still gin index creation takes forever

2008-11-11 Thread Tom Lane
Ivan Sergio Borgonovo <[EMAIL PROTECTED]> writes: > Any suggestion about how to track down the problem? What you are describing sounds rather like a use-of-uninitialized-memory problem, wherein the behavior depends on what happened to be in that memory previously. If so, using a debug/cassert-ena

[GENERAL] still gin index creation takes forever

2008-11-11 Thread Ivan Sergio Borgonovo
I'm still fighting with my very long gin index creation that happens randomly. At the beginning I had a pretty long transaction that filled several tables starting from some "temporary"[1] tables. After filling the tables I updated a tsvector column in one of them and finally in another connection