Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-12 Thread Lists
On 11/10/2012 02:21 PM, Jeff Janes wrote: On Fri, Nov 9, 2012 at 4:28 PM, Lists li...@benjamindsmith.com wrote: ... 3) For each of the tables from #2, run the commands REINDEX TABLE $table; VACUUM FULL ANALYZE $table; The end result is a squeaky-clean database server with expected disk

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-12 Thread Jeff Janes
On Mon, Nov 12, 2012 at 10:38 AM, Lists li...@benjamindsmith.com wrote: On 11/10/2012 02:21 PM, Jeff Janes wrote: On Fri, Nov 9, 2012 at 4:28 PM, Lists li...@benjamindsmith.com wrote: 2) It was sheer chance that I discovered the need to reindex prior to vacuum in order to get the disk

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-12 Thread Lists
On 11/12/2012 01:31 PM, Jeff Janes wrote: On Mon, Nov 12, 2012 at 10:38 AM, Lists li...@benjamindsmith.com wrote: On 11/10/2012 02:21 PM, Jeff Janes wrote: On Fri, Nov 9, 2012 at 4:28 PM, Lists li...@benjamindsmith.com wrote: 2) It was sheer chance that I discovered the need to reindex prior

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-11 Thread Lists
On 11/09/2012 05:26 PM, Steve Crawford wrote: Bloat in pg_attribute would correlate with A) (or any constant creation/destruction of tables). You can vacuum and/or reindex the system tables if you are connected as the superuser but you are better off preventing bloat by appropriate adjustment

Re: PG defaults and performance (was Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum))

2012-11-10 Thread Jeff Janes
On Fri, Nov 9, 2012 at 6:17 PM, Chris Angelico ros...@gmail.com wrote: On Sat, Nov 10, 2012 at 12:26 PM, Steve Crawford scrawf...@pinpointresearch.com wrote: Don't do that. Defaults are good for ensuring that PostgreSQL will start on the widest reasonable variety of systems. They are

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-10 Thread Jeff Janes
On Fri, Nov 9, 2012 at 5:56 PM, Scott Marlowe scott.marl...@gmail.com wrote: As well, since the default nap time is 1 minute, it will take at least 50 minutes to vacuum each db as nap time is how long autovac waits between databases. That isn't how it works. The naptime is per database, not

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-10 Thread Scott Marlowe
On Sat, Nov 10, 2012 at 2:12 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Fri, Nov 9, 2012 at 5:56 PM, Scott Marlowe scott.marl...@gmail.com wrote: As well, since the default nap time is 1 minute, it will take at least 50 minutes to vacuum each db as nap time is how long autovac waits

Re: PG defaults and performance (was Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum))

2012-11-10 Thread Chris Angelico
On Sun, Nov 11, 2012 at 8:05 AM, Jeff Janes jeff.ja...@gmail.com wrote: Totally not. With default settings and default pgbench, the easiest way for host B to beat host A is by lying about the durability of fsync. True. Without the ability to brutally cut the power to a cloud instance or other

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-10 Thread Adrian Klaver
On 11/10/2012 02:08 PM, Scott Marlowe wrote: On Sat, Nov 10, 2012 at 2:12 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Fri, Nov 9, 2012 at 5:56 PM, Scott Marlowe scott.marl...@gmail.com wrote: As well, since the default nap time is 1 minute, it will take at least 50 minutes to vacuum each db

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-10 Thread Jeff Janes
On Fri, Nov 9, 2012 at 4:28 PM, Lists li...@benjamindsmith.com wrote: ... 3) For each of the tables from #2, run the commands REINDEX TABLE $table; VACUUM FULL ANALYZE $table; The end result is a squeaky-clean database server with expected disk usage. NOTES: ... 2) It was sheer chance

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-10 Thread Scott Marlowe
On Sat, Nov 10, 2012 at 3:20 PM, Adrian Klaver adrian.kla...@gmail.com wrote: On 11/10/2012 02:08 PM, Scott Marlowe wrote: On Sat, Nov 10, 2012 at 2:12 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Fri, Nov 9, 2012 at 5:56 PM, Scott Marlowe scott.marl...@gmail.com wrote: As well, since

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-10 Thread Adrian Klaver
On 11/10/2012 02:23 PM, Scott Marlowe wrote: When in doubt there are the docs:) http://www.postgresql.org/docs/9.1/interactive/routine-vacuuming.html#AUTOVACUUM The autovacuum daemon actually consists of multiple processes. There is a persistent daemon process, called the autovacuum

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-09 Thread Lists
As I've spent a considerable amount of time trying to sort this out, I'm posting it for the benefit other users. I've experienced persistent, ongoing issues with autovacuum in a mixed read/write environment with midrange hardware (16 core Xeon, 128 GB RAM, 200 GB SATA3 6 Gb SSDs for disk I/O

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-09 Thread Steve Crawford
On 11/09/2012 04:28 PM, Lists wrote: As I've spent a considerable amount of time trying to sort this out, I'm posting it for the benefit other users. Having missed the earlier conversationa couple comments: I've experienced persistent, ongoing issues with autovacuum in a mixed read/write

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-09 Thread Scott Marlowe
On Fri, Nov 9, 2012 at 5:28 PM, Lists li...@benjamindsmith.com wrote: As I've spent a considerable amount of time trying to sort this out, I'm posting it for the benefit other users. SNIP D) concurrent use of pg_dump; Not usually a problem, unless it's overloading your IO subsystem.

PG defaults and performance (was Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum))

2012-11-09 Thread Chris Angelico
On Sat, Nov 10, 2012 at 12:26 PM, Steve Crawford scrawf...@pinpointresearch.com wrote: Don't do that. Defaults are good for ensuring that PostgreSQL will start on the widest reasonable variety of systems. They are *terrible* for performance and are certainly wrong for the system you describe.

Re: [GENERAL] Unexpectedly high disk space usage RESOLVED (Manual reindex/vacuum)

2012-11-09 Thread Greg Williamson
Scott --  ... To tune autovacuum with 50 databases, start by dropping nap time to something much lower, like 10s.  Then if you need to, drop cost delay until you get to 0.  If you get to 0 and it's still not hitting your IO too hard, but not keeping up, then increase cost limit.  If you get to