Re: [PERFORM] Exploring memory usage

2011-12-27 Thread Andrew Dunstan
On 12/27/2011 11:00 AM, Scott Marlowe wrote: On Tue, Dec 27, 2011 at 8:33 AM, Claudio Freire wrote: On Sat, Dec 24, 2011 at 4:22 PM, Michael Smolsky wrote: work_mem = 128MB (tried 257MB, didn't change anything) This is probably your problem. Without an EXPLAIN output, I cannot be sure, bu

Re: [PERFORM] Exploring memory usage

2011-12-27 Thread Scott Marlowe
On Tue, Dec 27, 2011 at 9:14 AM, Andrew Dunstan wrote: > It depends on the workload. Your 16M setting would make many of my clients' > systems slow to an absolute crawl for some queries, and they don't run into > swap issues, because we've made educated guesses about usage patterns. Exactly. I'v

Re: [PERFORM] Exploring memory usage

2011-12-27 Thread Scott Marlowe
On Sat, Dec 24, 2011 at 12:22 PM, Michael Smolsky wrote: > shared_buffers = 2GB (tried 8GB, didn't change anything) > work_mem = 128MB (tried 257MB, didn't change anything) As someone mentioned, lower is better here. 128M is quite high. > effective_cache_size = 12GB (tried 2GB didn't change any

Re: [PERFORM] Exploring memory usage

2011-12-27 Thread Claudio Freire
On Tue, Dec 27, 2011 at 1:00 PM, Scott Marlowe wrote: > He can lower it for just that query but honestly, even on a machine > with much more memory I'd never set it as high as he has it.  On a > busy machine with 128G RAM the max I ever had it set to was 16M, and > that was high enough I kept a cl

Re: [PERFORM] Exploring memory usage

2011-12-27 Thread Scott Marlowe
On Tue, Dec 27, 2011 at 8:33 AM, Claudio Freire wrote: > On Sat, Dec 24, 2011 at 4:22 PM, Michael Smolsky wrote: >> work_mem = 128MB (tried 257MB, didn't change anything) > > This is probably your problem. > > Without an EXPLAIN output, I cannot be sure, but 'work_mem' is not the > total amount o

Re: [PERFORM] Exploring memory usage

2011-12-27 Thread Claudio Freire
On Sat, Dec 24, 2011 at 4:22 PM, Michael Smolsky wrote: > work_mem = 128MB (tried 257MB, didn't change anything) This is probably your problem. Without an EXPLAIN output, I cannot be sure, but 'work_mem' is not the total amount of memory a query can use, it's the amount of memory it can use for

[PERFORM] Exploring memory usage

2011-12-27 Thread Michael Smolsky
Hello, I'm running a fairly complex query on my postgres-8.4.9 Ubuntu box. The box has 8-core CPU, 18G of RAM and no virtualization layer. The query takes many hours to run. The query essentially involves a join of two large tables on a common string column, but it also includes joins with oth