Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Thomas Finneid
Greg Smith wrote: On Mon, 1 Sep 2008, Thomas Finneid wrote: Thanks for all the info on the disk controller, I will have to look through all that now :) I note that nobody has talked about your postgresql.conf yet. I assume you've turned autovacuum off because you're not ever deleting thing

Re: [PERFORM] too many clog files

2008-09-01 Thread Tom Lane
Greg Smith <[EMAIL PROTECTED]> writes: > On Tue, 2 Sep 2008, Duan Ligong wrote: >> - Can we controll the maximum number of the clog files? > The reference Alvaro suggested at > http://www.postgresql.org/docs/8.3/interactive/routine-vacuuming.html#VACUUM-FOR-WRAPAROUND > > goes over that. If yo

Re: [PERFORM] too many clog files

2008-09-01 Thread Greg Smith
On Tue, 2 Sep 2008, Duan Ligong wrote: - Does Vacuum delete the old clog files? Yes, if those transactions are all done. One possibility here is that you've got some really long-running transaction floating around that is keeping normal clog cleanup from happening. Take a look at the outpu

Re: [PERFORM] limit clause breaks query planner?

2008-09-01 Thread Russell Smith
Pavel Stehule wrote: > Hello > > 2008/9/1 David West <[EMAIL PROTECTED]>: > >> Thanks for your suggestion but the result is the same. >> >> Here is the explain analyse output from different queries. >> Select * from my_table where A is null and B = '21' limit 15 >> >> "Limit (cost=0.00..3.68 ro

Re: [PERFORM] too many clog files

2008-09-01 Thread Duan Ligong
Alvaro Herrera wrote: > > Duan Ligong wrote: > > I have encountered an issue that there are too many > > clog file under the .../pg_clog/ directory. Some of them > > were even produced one month ago. > > If you're going to repost a question, it is only polite that you link to > the answers alrea

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Greg Smith
On Mon, 1 Sep 2008, Thomas Finneid wrote: It does have a sata raid controller, but not have the battery pack, because its a develmachine and not a production machine, I thought it was not needed. But if you are saying the battery pack enables a cache which enables faster disk writes I will con

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Scott Carey
On Mon, Sep 1, 2008 at 12:41 PM, Gregory Stark <[EMAIL PROTECTED]>wrote: > "Scott Carey" <[EMAIL PROTECTED]> writes: > > > On Raid Controllers and Dev machines: > > > > For a dev machine the battery backup is NOT needed. > > > > Battery back up makes a _production_ system faster: In production, d

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Scott Marlowe
On Mon, Sep 1, 2008 at 2:42 PM, Scott Marlowe <[EMAIL PROTECTED]> wrote: > dialog box saying this is a bad idea. Now, if it would take you a day > of downtime to get a dev database back in place and running after a > power loss, then the bbu may be worth the $200 or so. I just wanted to comment t

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Scott Marlowe
On Mon, Sep 1, 2008 at 2:32 PM, Thomas Finneid <[EMAIL PROTECTED]> wrote: > > Scott Carey wrote: >> >> For a development box, just enable write-back caching regardless of the >> battery back up situation. As long as its not your only copy of > > Will have a look at it, the data is not important an

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Scott Carey
> > > > On the performance impact of using COPY instead of INSERT : out of >> curiosity, were you comparing COPY against raw row-by-row inserts (slow) or >> JDBC batch inserts (faster) or multi-row inserts: INSERT into X (a,b,c) >> values (1,2,3) , (4,5,6) , (7,8,9 ) , (10,11,12) ? >> > > I

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Thomas Finneid
Scott Carey wrote: For a development box, just enable write-back caching regardless of the battery back up situation. As long as its not your only copy of Will have a look at it, the data is not important and can be reproduced any time on any machine. The controller I have is a Areca ARC-12

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Gregory Stark
"Scott Carey" <[EMAIL PROTECTED]> writes: > On Raid Controllers and Dev machines: > > For a dev machine the battery backup is NOT needed. > > Battery back up makes a _production_ system faster: In production, data > integrity is everything, and write-back caching is dangerous without a > battery

Re: [PERFORM] limit clause breaks query planner?

2008-09-01 Thread Pavel Stehule
Hello 2008/9/1 David West <[EMAIL PROTECTED]>: > Thanks for your suggestion but the result is the same. > > Here is the explain analyse output from different queries. > Select * from my_table where A is null and B = '21' limit 15 > > "Limit (cost=0.00..3.68 rows=15 width=128) (actual time=85837.0

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Scott Carey
On Raid Controllers and Dev machines: For a dev machine the battery backup is NOT needed. Battery back up makes a _production_ system faster: In production, data integrity is everything, and write-back caching is dangerous without a battery back up. So: Without BBU: Write-through cache = data

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Scott Marlowe
On Mon, Sep 1, 2008 at 5:29 AM, Thomas Finneid <[EMAIL PROTECTED]> wrote: > It does have a sata raid controller, but not have the battery pack, because > its a develmachine and not a production machine, I thought it was not > needed. But if you are saying the battery pack enables a cache which ena

Re: [PERFORM] limit clause breaks query planner?

2008-09-01 Thread David West
Thanks for your suggestion but the result is the same. Here is the explain analyse output from different queries. Select * from my_table where A is null and B = '21' limit 15 "Limit (cost=0.00..3.68 rows=15 width=128) (actual time=85837.043..85896.140 rows=15 loops=1)" " -> Seq Scan on my_tab

Re: [PERFORM] too many clog files

2008-09-01 Thread Alvaro Herrera
Duan Ligong wrote: > Hi, there. > > I have encountered an issue that there are too many > clog file under the .../pg_clog/ directory. Some of them > were even produced one month ago. If you're going to repost a question, it is only polite that you link to the answers already provided. Particul

Re: [PERFORM] Best hardware/cost tradoff?

2008-09-01 Thread Fernando Hevia
> -Mensaje original- > De: [EMAIL PROTECTED] > [mailto:[EMAIL PROTECTED] En nombre de cluster > Enviado el: Sábado, 30 de Agosto de 2008 07:21 > Para: pgsql-performance@postgresql.org > Asunto: Re: [PERFORM] Best hardware/cost tradoff? > > We are now leaning towards just buying 4 SAS d

Re: [PERFORM] limit clause breaks query planner?

2008-09-01 Thread Pavel Stehule
Hello you should partial index create index foo(b) on mytable where a is null; regards Pavel Stehule 2008/9/1 David West <[EMAIL PROTECTED]>: > Hi, > > > > I have a single table with about 10 million rows, and two indexes. Index A > is on a column A with 95% null values. Index B is on a colum

[PERFORM] limit clause breaks query planner?

2008-09-01 Thread David West
Hi, I have a single table with about 10 million rows, and two indexes. Index A is on a column A with 95% null values. Index B is on a column B with about 10 values, ie. About a million rows of each value. When I do a simple query on the table (no joins) with the following condition: A is

[PERFORM] too many clog files

2008-09-01 Thread Duan Ligong
Hi, there. I have encountered an issue that there are too many clog file under the .../pg_clog/ directory. Some of them were even produced one month ago. My questions: - Does Vacuum delete the old clog files? - Can we controll the maximum number of the clog files? - When, or in what case is a

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Thomas Finneid
Craig Ringer wrote: Just on a side note, your system is pretty strangely heavy on CPU compared to its RAM and disk configuration. Unless your workload in Pg is computationally intensive or you have something else hosted on the same machine, those CPUs will probably sit mostly idle. Its a devel

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Thomas Finneid
Scott Carey wrote: Are you even getting COPY to work with JDBC? As far as I am aware, COPY doesn't work with JDBC at the moment: I used a patched jdbc driver, provided by someone on the list, dont have the reference at hand. It works perfectly and its about 5 times faster, for my job, than

Re: [PERFORM] slow update of index during insert/copy

2008-09-01 Thread Scott Carey
Are you even getting COPY to work with JDBC? As far as I am aware, COPY doesn't work with JDBC at the moment: http://jdbc.postgresql.org/todo.html Listed in the todo page, under "PG Extensions" is "Add support for COPY." I tried to use it with JDBC a while ago and gave up after a couple limit