Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread MauMau
From: Merlin Moncure mmonc...@gmail.com can we see all of your memory settings plus physical memory? the solution is probably going to be reducing shared buffers an/or adding physical memory. Thank you for your response. The amount of physical memory is 8GB, which is enough for the workload. I

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Alvaro Herrera
Excerpts from MauMau's message of jue sep 29 09:23:48 -0300 2011: The amount of physical memory is 8GB, which is enough for the workload. I asked the customer for the output of SHOW ALL, but I haven't received it yet. However, shared_buffers should be less than 1.6GB because, as I wrote

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread MauMau
From: Alvaro Herrera alvhe...@commandprompt.com You don't really know this; some operating systems (Linux in particular) does not show shared memory as in use by a process until it is accessed. It may very well have well over 1.6 GB of shared_buffers, yet not show that in VIRT. Oh, really?

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Merlin Moncure
On Thu, Sep 29, 2011 at 7:23 AM, MauMau maumau...@gmail.com wrote: From: Merlin Moncure mmonc...@gmail.com can we see all of your memory settings plus physical memory?  the solution is probably going to be reducing shared buffers an/or adding physical memory. Thank you for your response.

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Robert Haas
On Thu, Sep 29, 2011 at 9:39 AM, Merlin Moncure mmonc...@gmail.com wrote: Like I said, this doesn't really come up this often but the 'real' solution in terms of postgrs is probably some kind of upper bound in the amount of cache memory used plus some intelligence in the cache implementation.  

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread MauMau
From: Merlin Moncure mmonc...@gmail.com -- Oh -- I missed earlier that this was 32 bit o/s. Well, I'd consider drastically reducing shared buffers, down to say 256-512mb range. Postgres function plans and various other structures, tables,

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Merlin Moncure
On Thu, Sep 29, 2011 at 8:59 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Sep 29, 2011 at 9:39 AM, Merlin Moncure mmonc...@gmail.com wrote: Like I said, this doesn't really come up this often but the 'real' solution in terms of postgrs is probably some kind of upper bound in the amount

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Tom Lane
MauMau maumau...@gmail.com writes: Anyway, I'd appreciate if anyone could tell me about RelCache/SysCache. As far as I read the code, PostgreSQL seems to use memory for RelCache/SysCache without limit until the relations are dropped. That's correct. We used to have a limit on the size of

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: ... It seems that we used to have some kind of LRU algorithm to prevent excessive memory usage, but we rippped it out because it was too expensive (see commit 8b9bc234ad43dfa788bde40ebf12e94f16556b7f). Not only was it too expensive, but performance

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread MauMau
From: Tom Lane t...@sss.pgh.pa.us That's correct. We used to have a limit on the size of catcache (if memory serves, it was something like 5000 entries). We got rid of it after observing that performance fell off a cliff as soon as you had a working set larger than the cache limit. Trust me,

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Robert Haas
On Thu, Sep 29, 2011 at 10:45 AM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... It seems that we used to have some kind of LRU algorithm to prevent excessive memory usage, but we rippped it out because it was too expensive (see commit

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-29 Thread Merlin Moncure
On Thu, Sep 29, 2011 at 10:22 AM, Robert Haas robertmh...@gmail.com wrote: I can't really explain why people seem to keep wanting to create hundreds of thousands or even millions of tables, but it's not like MauMau's customer is the first one to try to do this, and I'm sure they won't be the

[HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-28 Thread MauMau
Hello, Please let me ask you some questions about RelCache/SysCache/CatCache design. I know I should post this to pgsql-general, but I decided to post here because the content includes design questions. Background My customer is facing a out of memory problem during a batch job. I'd like to

Re: [HACKERS] Does RelCache/SysCache shrink except when relations are deleted?

2011-09-28 Thread Merlin Moncure
2011/9/28 MauMau maumau...@gmail.com: Hello, Please let me ask you some questions about RelCache/SysCache/CatCache design. I know I should post this to pgsql-general, but I decided to post here because the content includes design questions. Background My customer is facing a out of memory