Re: [PERFORM] Problem with database performance, Debian 4gb ram ?

2009-11-03 Thread Kevin Grittner
Jeff Janes wrote: > On Tue, Nov 3, 2009 at 7:13 AM, Kevin Grittner > wrote: >> Grant Masan wrote: >> >> >>> cpu_tuple_cost = 0.0030 >>> cpu_index_tuple_cost = 0.0010 >>> cpu_operator_cost = 0.0005 >> >> Why did you make these adjustments? I usually have to change the >> ratio between page and c

Re: [PERFORM] maintaining a reference to a fetched row

2009-11-03 Thread Tom Lane
Brian Karlak writes: > My question is this: is there some way that I can keep a cursor / > pointer / reference / whatever to the row I fetched originally, so > that I don't have to search for it again when I'm ready to write > results? If you don't expect any updates to the row meanwhile, c

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Kevin Grittner
I wrote: > Tom Lane wrote: >> But the particular example shown here doesn't make a very good case >> for that, because it's hard to tell how much of a penalty would be >> taken in more realistic examples. > > Fair enough. We're in the early stages of moving to tsearch2 and I > haven't run acr

Re: [PERFORM] High Frequency Inserts to Postgres Database vs Writing to a File

2009-11-03 Thread David Saracini
"could be several 1000 records a second." So, are there periods when there are no/few records coming in? Do the records/data/files really need to be persisted? The following statement makes me think you should go the flat file route: "The advantage of running complex queries to mine the data

Re: [PERFORM] High Frequency Inserts to Postgres Database vs Writing to a File

2009-11-03 Thread Scott Marlowe
On Tue, Nov 3, 2009 at 8:12 PM, Jay Manni wrote: > Hi: > > > > I have an application wherein a process needs to read data from a stream and > store the records for further analysis and reporting. The data in the stream > is in the form of variable length records with clearly defined fields – so >

Re: [PERFORM] Optimizer + bind variables

2009-11-03 Thread David Kerr
On Wed, Nov 04, 2009 at 11:02:22AM +1100, Chris wrote: - David Kerr wrote: - >On Wed, Nov 04, 2009 at 07:43:16AM +0800, Craig Ringer wrote: - >- David Kerr wrote: - No. - - This is explained in the notes here: - - http://www.postgresql.org/docs/current/static/sql-prepare.html and i've read that

Re: [PERFORM] maintaining a reference to a fetched row

2009-11-03 Thread Brian Karlak
On Nov 3, 2009, at 4:03 PM, Craig Ringer wrote: I don't have a good answer for you there. Perhaps using Pg's locking to do your queueing, rather than updating a status flag, might let you use a cursor? Have a look at the list archives - there's been a fair bit of discussion of queuing mec

[PERFORM] High Frequency Inserts to Postgres Database vs Writing to a File

2009-11-03 Thread Jay Manni
Hi: I have an application wherein a process needs to read data from a stream and store the records for further analysis and reporting. The data in the stream is in the form of variable length records with clearly defined fields - so it can be stored in a database or in a file. The only caveat i

Re: [PERFORM] maintaining a reference to a fetched row

2009-11-03 Thread Craig Ringer
Brian Karlak wrote: > The setup is relatively simple: there is a central queue table in > postgres. Worker daemons do a bounded, ordered, limited SELECT to grab > a row, which they lock by setting a value in the queue.status column. You can probably do an UPDATE ... RETURNING to turn that into

Re: [PERFORM] Free memory usage Sol10, 8.2.9

2009-11-03 Thread Jeremy Harris
On 11/03/2009 07:16 PM, Subbiah Stalin-XCGF84 wrote: All, I'm trying to understand the free memory usage and why it falls below 17G sometimes and what could be causing it. Any pointers would be appreciated. r...@prod1 # prtconf System Configuration: Sun Microsystems sun4u Memory size: 32768 M

Re: [PERFORM] Optimizer + bind variables

2009-11-03 Thread Chris
David Kerr wrote: On Wed, Nov 04, 2009 at 07:43:16AM +0800, Craig Ringer wrote: - David Kerr wrote: - > Does/is it possible for the PG optimizer come up with differnet plans when - > you're using bind variables vs when you send static values? - - Yes, if the bind variable form causes your DB ac

[PERFORM] Optimizer + bind variables

2009-11-03 Thread David Kerr
Does/is it possible for the PG optimizer come up with differnet plans when you're using bind variables vs when you send static values? like if my query was select * from users (add a bunch of complex joins) where username = 'dave' vs select * from users (add a bunch of complex joins) where user

Re: [PERFORM] Optimizer + bind variables

2009-11-03 Thread Craig Ringer
David Kerr wrote: > Does/is it possible for the PG optimizer come up with differnet plans when > you're using bind variables vs when you send static values? Yes, if the bind variable form causes your DB access driver to use a server-side prepared statement. Pg can't use its statistics to improve

Re: [PERFORM] Optimizer + bind variables

2009-11-03 Thread David Kerr
On Wed, Nov 04, 2009 at 07:43:16AM +0800, Craig Ringer wrote: - David Kerr wrote: - > Does/is it possible for the PG optimizer come up with differnet plans when - > you're using bind variables vs when you send static values? - - Yes, if the bind variable form causes your DB access driver to use a

[PERFORM] maintaining a reference to a fetched row

2009-11-03 Thread Brian Karlak
Hello All -- I have a simple queuing application written on top of postgres which I'm trying to squeeze some more performance out of. The setup is relatively simple: there is a central queue table in postgres. Worker daemons do a bounded, ordered, limited SELECT to grab a row, which they

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Jesper Krogh
Tom Lane wrote: > It may well be that Jesper's identified a place where the GIN code could > be improved --- it seems like having the top-level search logic be more > aware of the AND/OR structure of queries would be useful. But the > particular example shown here doesn't make a very good case for

[PERFORM] Free memory usage Sol10, 8.2.9

2009-11-03 Thread Subbiah Stalin-XCGF84
All, I'm trying to understand the free memory usage and why it falls below 17G sometimes and what could be causing it. Any pointers would be appreciated. r...@prod1 # prtconf System Configuration: Sun Microsystems sun4u Memory size: 32768 Megabytes [postg...@prod1 ~]$ vmstat 5 10 kthr

Re: [PERFORM] Problem with database performance, Debian 4gb ram ?

2009-11-03 Thread Jeff Janes
On Tue, Nov 3, 2009 at 7:13 AM, Kevin Grittner wrote: > Grant Masan wrote: > > >> cpu_tuple_cost = 0.0030 >> cpu_index_tuple_cost = 0.0010 >> cpu_operator_cost = 0.0005 > > Why did you make these adjustments? I usually have to change the > ratio between page and cpu costs toward the other direct

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Kevin Grittner
"Kevin Grittner" wrote: > I'm wondering if anyone has ever confirmed that probing for the more > frequent term through the index is *ever* a win, versus using the > index for the most common of the top level AND conditions and doing > the rest on recheck. s/most/least/ -Kevin -- Sent via p

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Kevin Grittner
Tom Lane wrote: > The answer to that clearly is to not index common terms My understanding is that we don't currently get statistics on how common the terms in a tsvector column are until we ANALYZE the *index* created from it. Seems like sort of a Catch 22. Also, if we exclude words which a

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Tom Lane
"Kevin Grittner" writes: > Perhaps I'm missing something. My point was that there are words > which are too common to be useful for index searches, yet uncommon > enough to usefully limit the results. These words could typically > benefit from tsearch2 style parsing and dictionaries; so declarin

Re: [PERFORM] Problem with database performance, Debian 4gb ram ?

2009-11-03 Thread Kevin Grittner
Grant Masan wrote: > max_connections = 80 > shared_buffers = 512MB > temp_buffers = 8MB > work_mem = 20MB > maintenance_work_mem = 384MB > wal_buffers = 8MB > checkpoint_segments = 128MB > effective_cache_size = 2304MB > checkpoint_timeout = 1h Pending further information, these seem sane to m

Re: [PERFORM] Queryplan within FTS/GIN index -search.

2009-11-03 Thread Kevin Grittner
Tom Lane wrote: > "Kevin Grittner" writes: >> Tom Lane wrote: >>> Any sane text search application is going to try to filter out >>> common words as stopwords; it's only the failure to do that that's >>> making this run slow. > >> I'd rather have the index used for the selective test, and appl