Re: [PERFORM] rtree/gist index taking enormous amount of space in 8.2.3

2007-06-29 Thread Dolafi, Tom
Thanks for looking into this and reproducing a similar result. The index took 6 hours to complete on a 1.5GB table resulting in 35GB of storage, and it took 36 hours to vacuum... I'm patient :-) In the mean time I've dropped the index which has resulted in overall performance gain on queries agai

Re: [PERFORM]

2007-06-29 Thread Ed Tyrrill
On Mon, 2007-06-25 at 21:07 -0400, Tom Lane wrote: > It's worth fooling around with work_mem just to see what happens. The > other thing that would be interesting is to force the other plan (set > enable_mergejoin = off) just to see what the planner is costing it at. > My suspicion is that the est

Re: [PERFORM] rtree/gist index taking enormous amount of space in 8.2.3

2007-06-29 Thread Dolafi, Tom
The application need is to determine genomic features present in a user-defined portion of a chromosome. My guess is that features (boxes) are overlapping along a line (chromosome), and there is a need to represent them as being stacked. Since I'm not certain of its exact use, I've emailed the ap

Re: [PERFORM] rtree/gist index taking enormous amount of space in 8.2.3

2007-06-29 Thread Dolafi, Tom
(fmax-fmin)... min | max | avg -+-+-- 1 | 2278225 | 546 I noticed 3000 occurrences where fmax is less than fmin. I excluded these values to get the min difference between the two. Also, there are 20 "invalid"/"bogus" rows with

Re: [PERFORM] rtree/gist index taking enormous amount of space in 8.2.3

2007-06-29 Thread Tom Lane
"Dolafi, Tom" <[EMAIL PROTECTED]> writes: > In the mean time I've dropped the index which has resulted in overall > performance gain on queries against the table, but we have not tested > the part of the application which would utilize this index. I noted that with the same (guessed-at) distributi

Re: [PERFORM] rtree/gist index taking enormous amount of space in 8.2.3

2007-06-29 Thread Tom Lane
"Dolafi, Tom" <[EMAIL PROTECTED]> writes: > The data is not distributed well... Can you show us min, max, and avg of fmax minus fmin? I'd like to check my guess about that being a fairly narrow range. regards, tom lane ---(end of broadcast)---

Re: [PERFORM] rtree/gist index taking enormous amount of space in 8.2.3

2007-06-29 Thread Dolafi, Tom
The data is not distributed well... Top 20 occurrences of fmin and fmax: fmin | count --+ 0 | 214476 19281576 | 2870 2490005 | 2290 1266332 | 2261 15539680 | 2086 11022233 | 2022 25559658 | 1923 3054411 | 1906 10237885 | 1890 13827272 | 18

Re: [PERFORM] rtree/gist index taking enormous amount of space in 8.2.3

2007-06-29 Thread Tom Lane
"Dolafi, Tom" <[EMAIL PROTECTED]> writes: > min(fmin) | max(fmin)|avg(fmin) >1 | 55296469 |11423945 > min(fmax) | max(fmax)|avg(fmax) > 18 | 3288 |11424491 OK, I was able to reproduce a problem after making the further guess th

Re: [PERFORM] rtree/gist index taking enormous amount of space in 8.2.3

2007-06-29 Thread Craig James
Dolafi, Tom wrote: min(fmin) | max(fmin)|avg(fmin) 1 | 55296469 |11423945 min(fmax) | max(fmax)|avg(fmax) 18 | 3288 |11424491 There are 5,704,211 rows in the table. When you're looking for weird index problems, it's more in

Re: [PERFORM] rtree/gist index taking enormous amount of space in 8.2.3

2007-06-29 Thread Dolafi, Tom
min(fmin) | max(fmin)|avg(fmin) 1 | 55296469 |11423945 min(fmax) | max(fmax)|avg(fmax) 18 | 3288 |11424491 There are 5,704,211 rows in the table. This application has been inherited by us. As far as I can tell the magic of t

Re: [PERFORM] PostgreSQL 8.0 occasionally slow down

2007-06-29 Thread Alvaro Herrera
Richard Huxton escribió: > Ho Fat Tsang wrote: > > Correct me if i am wrong. It seems postgresql 8.0 does not bundle > >auto-vacuum by default. So all vacuum and analyse are done manually ? So > >what i have tested related to vaccuum is running auto-vacuum (a executeable > >located in /bin) paral

Re: [PERFORM] PostgreSQL 8.0 occasionally slow down

2007-06-29 Thread Richard Huxton
Ho Fat Tsang wrote: Hi Richard, I've tested again according your suggestion. I noticed that for each time the pgsql slow down, there is a short period a process called "pdflush" eating up lot of I/O. I've goolgled and know it is a process for writing dirty pages back to the disk by the Linu