Re: [PERFORM] Optimize update query

2012-11-29 Thread Niels Kristian Schjødt
Hmm I'm getting suspicious here. Maybe my new great setup with the SSD's is not really working as it should., and maybe new relic is not monitoring as It should. If I do a "sudo iostat -k 1" I get a lot of output like this: Device:tpskB_read/skB_wrtn/skB_readkB_wrtn s

Re: [PERFORM] Optimize update query

2012-11-29 Thread Niels Kristian Schjødt
Den 30/11/2012 kl. 02.24 skrev "Kevin Grittner" : > Niels Kristian Schjødt wrote: > >> Okay, now I'm done the updating as described above. I did the >> postgres.conf changes. I did the kernel changes, i added two >> SSD's in a software RAID1 where the pg_xlog is now located - >> unfortunately the

Re: [PERFORM] Optimize update query

2012-11-29 Thread Kevin Grittner
Niels Kristian Schjødt wrote: > Okay, now I'm done the updating as described above. I did the > postgres.conf changes. I did the kernel changes, i added two > SSD's in a software RAID1 where the pg_xlog is now located - > unfortunately the the picture is still the same :-( You said before that y

Re: [PERFORM] Optimize update query

2012-11-29 Thread Niels Kristian Schjødt
Den 28/11/2012 kl. 17.54 skrev Shaun Thomas : > On 11/28/2012 10:19 AM, Niels Kristian Schjødt wrote: > >> https://rpm.newrelic.com/public/charts/h2dtedghfsv > > Doesn't this answer your question? > > That iowait is crushing your server into the ground. It's no surprise updates > are taking s

Re: [PERFORM] Savepoints in transactions for speed?

2012-11-29 Thread Jeff Janes
On Thu, Nov 29, 2012 at 10:14 AM, Mike Blackwell wrote: > > > > On Thu, Nov 29, 2012 at 12:09 PM, Jeff Janes wrote: >> >> >> But If you do keep the drop index inside the transaction, then you >> would probably be better off using truncate rather than delete, and >> rebuild the index non-concurren

Re: [PERFORM] Savepoints in transactions for speed?

2012-11-29 Thread Claudio Freire
On Thu, Nov 29, 2012 at 3:32 PM, Jeff Davis wrote: > On Thu, 2012-11-29 at 00:48 -0300, Claudio Freire wrote: >> Not really that fast if you have indices (and who doesn't have a PK or two). >> >> I've never been able to update (update) 2M rows in one transaction in >> reasonable times (read: less

Re: [PERFORM] Savepoints in transactions for speed?

2012-11-29 Thread Jeff Davis
On Thu, 2012-11-29 at 00:48 -0300, Claudio Freire wrote: > Not really that fast if you have indices (and who doesn't have a PK or two). > > I've never been able to update (update) 2M rows in one transaction in > reasonable times (read: less than several hours) without dropping > indices. Doing it

Re: [PERFORM] Savepoints in transactions for speed?

2012-11-29 Thread Mike Blackwell
On Thu, Nov 29, 2012 at 12:09 PM, Jeff Janes wrote: > > But If you do keep the drop index inside the transaction, then you > would probably be better off using truncate rather than delete, and > rebuild the index non-concurrently and move that inside the > transaction as well. > > Hmm From

Re: [PERFORM] Savepoints in transactions for speed?

2012-11-29 Thread Jeff Janes
On Thu, Nov 29, 2012 at 8:38 AM, Mike Blackwell wrote: > > What I'm trying at this point is: > > BEGIN; > DROP INDEX -- only one unique index exists > DELETE FROM table; > COPY table FROM STDIN; > COMMIT; > CREATE INDEX CONCURRENTLY; > > Do I understand correctly that DROP/CREATE index are not tra

Re: [PERFORM] Savepoints in transactions for speed?

2012-11-29 Thread Mike Blackwell
Ah. So it does. Testing with two psql sessions locks as you said, and moving the DROP INDEX to a separate transaction give the results I was looking for. Thanks, Mike __ *Mike Blackwell | Technical Analyst, Distribu

Re: [PERFORM] Comparative tps question

2012-11-29 Thread Merlin Moncure
On Thu, Nov 29, 2012 at 10:56 AM, John Lister wrote: > I must have misread the numbers before when using bonnie++, run it again and > getting 1.3Gb/s read and 700Mb/s write which looks more promising. In terms > of vmstat: pretty nice. >> *) Very first thing we need to check is if we are storage

Re: [PERFORM] Comparative tps question

2012-11-29 Thread John Lister
On 28/11/2012 19:21, Merlin Moncure wrote: On Wed, Nov 28, 2012 at 12:37 PM, John Lister wrote: Hi, I've just been benchmarking a new box I've got and running pgbench yields what I thought was a slow tps count. It is dificult to find comparisons online of other benchmark results, I'd like to se

Re: [PERFORM] Savepoints in transactions for speed?

2012-11-29 Thread Scott Marlowe
On Thu, Nov 29, 2012 at 9:38 AM, Mike Blackwell wrote: > On Wed, Nov 28, 2012 at 9:48 PM, Claudio Freire > wrote: >> >> On Wed, Nov 28, 2012 at 8:28 PM, Jeff Davis wrote: >> > >> > The main problem with a long-running delete or update transaction is >> > that the dead tuples (deleted tuples or t

Re: [PERFORM] Savepoints in transactions for speed?

2012-11-29 Thread Mike Blackwell
On Wed, Nov 28, 2012 at 9:48 PM, Claudio Freire wrote: > On Wed, Nov 28, 2012 at 8:28 PM, Jeff Davis wrote: > > > > The main problem with a long-running delete or update transaction is > > that the dead tuples (deleted tuples or the old version of an updated > > tuple) can't be removed until the

Re: [PERFORM] 9.2.1 & index-only scans : abnormal heap fetches after VACUUM FULL

2012-11-29 Thread Pavan Deolasee
On Thu, Nov 29, 2012 at 6:06 PM, Andres Freund wrote: > On 2012-11-29 17:59:39 +0530, Pavan Deolasee wrote: > > > > > > > Yeah, that looks fairly easy to have. Thinking about it more, now that we > > have ability to skip WAL for the case when a table is created and > populated > > in the same tran

Re: [PERFORM] 9.2.1 & index-only scans : abnormal heap fetches after VACUUM FULL

2012-11-29 Thread Andres Freund
On 2012-11-29 17:59:39 +0530, Pavan Deolasee wrote: > On Thu, Nov 29, 2012 at 5:42 PM, Andres Freund wrote: > > > On 2012-11-29 17:20:01 +0530, Pavan Deolasee wrote: > > > > > Now can CLUSTER or VACUUM FULL recreate the visibility map with all bits > > > set to visible, thats an entirely different

Re: [PERFORM] 9.2.1 & index-only scans : abnormal heap fetches after VACUUM FULL

2012-11-29 Thread Pavan Deolasee
On Thu, Nov 29, 2012 at 5:42 PM, Andres Freund wrote: > On 2012-11-29 17:20:01 +0530, Pavan Deolasee wrote: > > > Now can CLUSTER or VACUUM FULL recreate the visibility map with all bits > > set to visible, thats an entirely different question. I don't think it > can, > > but then I haven't though

Re: [PERFORM] 9.2.1 & index-only scans : abnormal heap fetches after VACUUM FULL

2012-11-29 Thread Andres Freund
On 2012-11-29 17:20:01 +0530, Pavan Deolasee wrote: > On Thu, Nov 29, 2012 at 5:03 PM, Guillaume Cottenceau wrote: > > > Hello, > > > > I am toying around with 9.2.1, trying to measure/determine how > > index-only scans can improve our performance. > > > > A small script which is attached to this

Re: [PERFORM] 9.2.1 & index-only scans : abnormal heap fetches after VACUUM FULL

2012-11-29 Thread Pavan Deolasee
On Thu, Nov 29, 2012 at 5:03 PM, Guillaume Cottenceau wrote: > Hello, > > I am toying around with 9.2.1, trying to measure/determine how > index-only scans can improve our performance. > > A small script which is attached to this mail, shows that as long > as the table has been VACUUM FULL'd, the

[PERFORM] 9.2.1 & index-only scans : abnormal heap fetches after VACUUM FULL

2012-11-29 Thread Guillaume Cottenceau
Hello, I am toying around with 9.2.1, trying to measure/determine how index-only scans can improve our performance. A small script which is attached to this mail, shows that as long as the table has been VACUUM FULL'd, there is a unusual high amount of heap fetches. It is strange that the visibil