Re: [PERFORM] Optimising "in" queries

2007-08-22 Thread Michael Glaesemann
[Please don't top post as it makes the discussion more difficult to follow.] On Aug 22, 2007, at 18:30 , Stephen Davies wrote: I have always thought of array processing as the thing that vector processors such as Cray and ETA do/did. (I've always heard that referred to as vector processing.

Re: [PERFORM] When/if to Reindex

2007-08-22 Thread Gregory Stark
"Steven Flatt" <[EMAIL PROTECTED]> writes: > However I'm seeing that all readers of that table are blocked until the > reindex finishes, even reads that do not attempt to use the index. Is this > a problem with the docs or a bug? You'll have to describe in more detail what you're doing so we can

Re: [PERFORM] Long running transaction in pg_activity_log

2007-08-22 Thread Farhan Mughal
If I am not wrong, you must be asking about this: select * from pg_stat_activity; If you see in the query column. Turn on the stats_command_string option in postgresql.conf. Regards, Farhan - Original Message From: Sachchida Ojha <[EMAIL PROTECTED]> To: pgsql-performa

Re: [PERFORM] Long running transaction in pg_activity_log

2007-08-22 Thread Tobias Brox
[Sachchida Ojha - Wed at 04:40:09PM -0400] > I see some long running transaction in my pg_activity_log table. My app > becomes almost unusable. My question is > How can I query the database to see what sql these transactions are > running. " in transaction" means that no sql query is running at

Re: [PERFORM] Fast tsearch2, trigram matching on short phrases

2007-08-22 Thread Carlo Stonebanks
>> The problem is in idea, not in performance. Oh, I think we both agree on that! ;-D This is why I didn't post any EXPLAINs or anything like that. I thought the problem was in the entire method of how to best zero in on the set of records best suited for closer analysis by my phrase-matching f

Re: [PERFORM] Long running transaction in pg_activity_log

2007-08-22 Thread Sachchida Ojha
I can see SQL in the current SQL column of pg_activity_log when I run sql from sql window or throgh odbc/jdbc connection but when sql is embedded in java beaans i can see those sql. Is there any way to extract those sql from the database. 16385;"em_db";20220;16388;"emsowner";"select * from pg_

[PERFORM] Long running transaction in pg_activity_log

2007-08-22 Thread Sachchida Ojha
I see some long running transaction in my pg_activity_log table. My app becomes almost unusable. My question is How can I query the database to see what sql these transactions are running. 16385;"em_db";20893;16386;"em_user";" in transaction";f;"2007-08-22 20:38:06.527792+00";"2007-08-22 20:37:

Re: [PERFORM] Fast tsearch2, trigram matching on short phrases

2007-08-22 Thread Oleg Bartunov
On Wed, 22 Aug 2007, Carlo Stonebanks wrote: Hi Oleg, you didn't show us explain analyze of your select. I didn't because I didn't expect any reaction to it - my understanding is that trigram matching for phrases is not recommended because of the performance. Do you believe that I SHOULD e

Re: [PERFORM] Fast tsearch2, trigram matching on short phrases

2007-08-22 Thread Carlo Stonebanks
Hi Oleg, you didn't show us explain analyze of your select. I didn't because I didn't expect any reaction to it - my understanding is that trigram matching for phrases is not recommended because of the performance. Do you believe that I SHOULD expect good performance from trigram matching o

Re: [PERFORM] io storm on checkpoints, postgresql 8.2.4, linux

2007-08-22 Thread Greg Smith
On Wed, 22 Aug 2007, Dmitry Potapov wrote: I found this http://www.westnet.com/~gsmith/content/linux-pdflush.htm If you do end up following up with this via the Linux kernel mailing list, please pass that link along. I've been meaning to submit it to them and wait for the flood of e-mail

Re: [PERFORM] Fast tsearch2, trigram matching on short phrases

2007-08-22 Thread Carlo Stonebanks
In December I had tried this; it caused a tremendous slowdown in our system. I have avoided it since then. Do you expect pg_trgm to work with phrases? OI had read a post earlier from an earlier support question that suggested that it I SHOULD expect performance to degrade and that pg_trgrm was

Re: [PERFORM] Optimising "in" queries

2007-08-22 Thread Kevin Grittner
>>> On Tue, Aug 21, 2007 at 9:40 PM, in message <[EMAIL PROTECTED]>, Stephen Davies <[EMAIL PROTECTED]> wrote: > Is there any way to make the "larger" queries more efficient? People would be in a better position to answer that if you posted the table structure and the results of EXPLAIN ANALYZE

Re: [PERFORM] Fast tsearch2, trigram matching on short phrases

2007-08-22 Thread Oleg Bartunov
On Wed, 22 Aug 2007, Carlo Stonebanks wrote: I have read that trigram matching (similarity()) performance degrades when the matching is on longer strings such as phrases. I need to quickly match strings and rate them by similiarity. The strings are typically one to seven words in length - and

Re: [PERFORM] When/if to Reindex

2007-08-22 Thread Steven Flatt
> > It makes more sense for us to have ~1 hour's worth of reindexing > afterwards during which read performance on that partition is "compromised". > So, based on the docs, I was expecting read performance to be compromised during a reindex, specifically reads would not be allowed to use the inde

Re: [PERFORM] Fast tsearch2, trigram matching on short phrases

2007-08-22 Thread Steinar H. Gunderson
On Wed, Aug 22, 2007 at 12:02:54PM -0400, Carlo Stonebanks wrote: > Any suggestions on where to go with this project to improve performance > would be greatly appreciated. I'm a bit unsure from reading your mail -- have you tried pg_trgm with a GiST index? /* Steinar */ -- Homepage: http://www.

Re: [PERFORM] io storm on checkpoints, postgresql 8.2.4, linux

2007-08-22 Thread Dmitry Potapov
2007/8/22, Kenneth Marshall <[EMAIL PROTECTED]>: > > > You are working at the correct level. The bgwriter performs the I/O > smoothing > function at the database level. Obviously, the OS level smoothing function > needed to be tuned and you have done that within the parameters of the OS. > You may

Re: [PERFORM] io storm on checkpoints, postgresql 8.2.4, linux

2007-08-22 Thread Dmitry Potapov
2007/8/22, Joshua D. Drake <[EMAIL PROTECTED]>: > > We've run into an issue of IO storms on checkpoints. Once in 20min > > (which is checkpoint_interval) the database becomes unresponsive for > about > > 4-8 seconds. Query processing is suspended, server does nothing but > writing > > What are

Re: [PERFORM] io storm on checkpoints, postgresql 8.2.4, linux

2007-08-22 Thread Dmitry Potapov
2007/8/22, Mark Mielke <[EMAIL PROTECTED]>: > > Are you able to show that the dirty pages are all coming from postgres? > > I don't know how to prove that, but I suspect that nothing else except postgres writes to disk on that system, because it runs nothing except postgresql and syslog (which I c

[PERFORM] Fast tsearch2, trigram matching on short phrases

2007-08-22 Thread Carlo Stonebanks
I have read that trigram matching (similarity()) performance degrades when the matching is on longer strings such as phrases. I need to quickly match strings and rate them by similiarity. The strings are typically one to seven words in length - and will often include unconventional abbreviations

Re: [PERFORM] io storm on checkpoints, postgresql 8.2.4, linux

2007-08-22 Thread Joshua D. Drake
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dmitry Potapov wrote: > Hello! > > We run a large (~66Gb) web-backend database on Postgresql 8.2.4 on > Linux. The hardware is Dual Xeon 5130 with 16Gb ram, LSI Megaraid U320-2x > scsi controller w/512Mb writeback cache and a BBU. Sto

Re: [PERFORM] io storm on checkpoints, postgresql 8.2.4, linux

2007-08-22 Thread Mark Mielke
Are you able to show that the dirty pages are all coming from postgres? Cheers, mark -- Mark Mielke <[EMAIL PROTECTED]> ---(end of broadcast)--- TIP 5: don't forget to increase your free space map settings

[PERFORM] io storm on checkpoints, postgresql 8.2.4, linux

2007-08-22 Thread Dmitry Potapov
Hello! We run a large (~66Gb) web-backend database on Postgresql 8.2.4 on Linux. The hardware is Dual Xeon 5130 with 16Gb ram, LSI Megaraid U320-2x scsi controller w/512Mb writeback cache and a BBU. Storage setup contains 3 raid10 arrays (data, xlog, indexes, each on different arr

Re: [PERFORM] Optimising "in" queries

2007-08-22 Thread Michael Glaesemann
On Aug 22, 2007, at 5:58 , Russell Smith wrote: Stephen Davies wrote: select count(rdate),rdate from reading where sensor_id in (1137,1138,1139,1140) group by rdate order by rdate desc limit 1; It would have been helpful to see the table definition here. I can say up front that array pr

Re: [PERFORM] Optimising "in" queries

2007-08-22 Thread Russell Smith
Stephen Davies wrote: I have a PostgreSQL 8.2.4 table with some seven million rows. The psql query: select count(rdate),rdate from reading where sensor_id in (1137,1138,1139) group by rdate order by rdate desc limit 1; takes a few seconds but: select count(rdate),rdate from reading where se

Re: [PERFORM] Poor Performance after Upgrade

2007-08-22 Thread Russell Smith
Ben Perrault wrote: Hi, I recently inherited a very old (PostgreSQL 7.0.3) database, and have migrated it to 8.2.4 but have run into a performance issue. Basically, I did a dump and import into the new database, vacuumed and created fresh indexes and everything is work great except the foll

Re: [PERFORM] Poor Performance after Upgrade

2007-08-22 Thread vincent
> Hi, > > I recently inherited a very old (PostgreSQL 7.0.3) database, and have > migrated it to 8.2.4 but have run into a performance issue. > Did you configure the 8.2.4 server to match the memory requirements etc of the old server? PostgreSQL's default settings are usually not aimed at optimal