Re: Creating a function for exposing memory usage of backend process

2020-08-25 Thread Fujii Masao
On 2020/08/25 11:39, Fujii Masao wrote: On 2020/08/24 21:56, torikoshia wrote: On 2020-08-24 13:13, Fujii Masao wrote: On 2020/08/24 13:01, torikoshia wrote: On 2020-08-22 21:18, Michael Paquier wrote: Thanks for reviewing! On Fri, Aug 21, 2020 at 11:27:06PM +0900, torikoshia wrote: O

Re: Creating a function for exposing memory usage of backend process

2020-08-24 Thread Fujii Masao
On 2020/08/24 21:56, torikoshia wrote: On 2020-08-24 13:13, Fujii Masao wrote: On 2020/08/24 13:01, torikoshia wrote: On 2020-08-22 21:18, Michael Paquier wrote: Thanks for reviewing! On Fri, Aug 21, 2020 at 11:27:06PM +0900, torikoshia wrote: OK. Added a regression test on sysviews.sql.

Re: Creating a function for exposing memory usage of backend process

2020-08-24 Thread Fujii Masao
On 2020/08/24 17:09, Michael Paquier wrote: On Mon, Aug 24, 2020 at 02:48:50PM +0900, Fujii Masao wrote: As far as I know, utils/adt is the directory to basically include the files for a particular type or operator. So ISTM that mcxtfuncs.c doesn't fit to this directory. Isn't it better to pu

Re: Creating a function for exposing memory usage of backend process

2020-08-24 Thread torikoshia
On 2020-08-24 13:13, Fujii Masao wrote: On 2020/08/24 13:01, torikoshia wrote: On 2020-08-22 21:18, Michael Paquier wrote: Thanks for reviewing! On Fri, Aug 21, 2020 at 11:27:06PM +0900, torikoshia wrote: OK. Added a regression test on sysviews.sql. (0001-Added-a-regression-test-for-pg_backe

Re: Creating a function for exposing memory usage of backend process

2020-08-24 Thread Michael Paquier
On Mon, Aug 24, 2020 at 02:48:50PM +0900, Fujii Masao wrote: > As far as I know, utils/adt is the directory to basically include the files > for a particular type or operator. So ISTM that mcxtfuncs.c doesn't > fit to this directory. Isn't it better to put that in utils/mmgr ? We have also stuff l

Re: Creating a function for exposing memory usage of backend process

2020-08-23 Thread Fujii Masao
On 2020/08/24 13:13, Fujii Masao wrote: On 2020/08/24 13:01, torikoshia wrote: On 2020-08-22 21:18, Michael Paquier wrote: Thanks for reviewing! On Fri, Aug 21, 2020 at 11:27:06PM +0900, torikoshia wrote: OK. Added a regression test on sysviews.sql. (0001-Added-a-regression-test-for-pg_

Re: Creating a function for exposing memory usage of backend process

2020-08-23 Thread Fujii Masao
On 2020/08/24 13:01, torikoshia wrote: On 2020-08-22 21:18, Michael Paquier wrote: Thanks for reviewing! On Fri, Aug 21, 2020 at 11:27:06PM +0900, torikoshia wrote: OK. Added a regression test on sysviews.sql. (0001-Added-a-regression-test-for-pg_backend_memory_contex.patch) Fujii-san gav

Re: Creating a function for exposing memory usage of backend process

2020-08-23 Thread torikoshia
On 2020-08-22 21:18, Michael Paquier wrote: Thanks for reviewing! On Fri, Aug 21, 2020 at 11:27:06PM +0900, torikoshia wrote: OK. Added a regression test on sysviews.sql. (0001-Added-a-regression-test-for-pg_backend_memory_contex.patch) Fujii-san gave us an example, but I added more simple on

Re: Creating a function for exposing memory usage of backend process

2020-08-22 Thread Michael Paquier
On Fri, Aug 21, 2020 at 11:27:06PM +0900, torikoshia wrote: > OK. Added a regression test on sysviews.sql. > (0001-Added-a-regression-test-for-pg_backend_memory_contex.patch) > > Fujii-san gave us an example, but I added more simple one considering > the simplicity of other tests on that. What yo

Re: Creating a function for exposing memory usage of backend process

2020-08-21 Thread torikoshia
Thanks for all your comments! Thankfully it seems that this feature is regarded as not meaningless one, I'm going to do some improvements. On Wed, Aug 19, 2020 at 10:56 PM Michael Paquier wrote: On Wed, Aug 19, 2020 at 06:12:02PM +0900, Fujii Masao wrote: On 2020/08/19 17:40, torikoshia wro

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Kasahara Tatsuhito
On 2020/08/20 0:01, Tom Lane wrote: > The only situation I could imagine where this would have any use is > where there is long-term (cross-query) bloat in, say, CacheMemoryContext Yeah, in cases where a very large number of sessions are connected to the DB for very long periods of time, the memory

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Fujii Masao
On 2020/08/20 10:43, Andres Freund wrote: Hi, On 2020-08-19 21:29:06 -0400, Tom Lane wrote: Andres Freund writes: On 2020-08-19 11:01:37 -0400, Tom Lane wrote: I agree with that, but I think this patch has a bigger problem: why bother at all? It seems like a waste of code space and futur

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Fujii Masao
On 2020/08/20 0:01, Tom Lane wrote: Hadn't been paying attention to this thread up till now, but ... Michael Paquier writes: By the way, I was looking at the code that has been committed, and I think that it is awkward to have a SQL function in mcxt.c, which is a rather low-level interface.

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Andres Freund
Hi, On 2020-08-19 21:29:06 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2020-08-19 11:01:37 -0400, Tom Lane wrote: > >> I agree with that, but I think this patch has a bigger problem: > >> why bother at all? It seems like a waste of code space and future > >> maintenance effort, because

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Tom Lane
Andres Freund writes: > On 2020-08-19 11:01:37 -0400, Tom Lane wrote: >> I agree with that, but I think this patch has a bigger problem: >> why bother at all? It seems like a waste of code space and future >> maintenance effort, because there is no use-case. > I don't agree with this at all. I t

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Andres Freund
Hi, On 2020-08-19 11:01:37 -0400, Tom Lane wrote: > Hadn't been paying attention to this thread up till now, but ... > > Michael Paquier writes: > > By the way, I was looking at the code that has been committed, and I > > think that it is awkward to have a SQL function in mcxt.c, which is a > >

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Tom Lane
Hadn't been paying attention to this thread up till now, but ... Michael Paquier writes: > By the way, I was looking at the code that has been committed, and I > think that it is awkward to have a SQL function in mcxt.c, which is a > rather low-level interface. I think that this new code should

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Michael Paquier
On Wed, Aug 19, 2020 at 06:12:02PM +0900, Fujii Masao wrote: > On 2020/08/19 17:40, torikoshia wrote: >> Yes, I didn't add regression tests because of the unstability of the output. >> I thought it would be OK since other views like pg_stat_slru and >> pg_shmem_allocations >> didn't have tests for

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread Fujii Masao
On 2020/08/19 17:40, torikoshia wrote: On 2020-08-19 15:48, Fujii Masao wrote: On 2020/08/19 9:43, torikoshia wrote: On 2020-08-18 22:54, Fujii Masao wrote: On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-0

Re: Creating a function for exposing memory usage of backend process

2020-08-19 Thread torikoshia
On 2020-08-19 15:48, Fujii Masao wrote: On 2020/08/19 9:43, torikoshia wrote: On 2020-08-18 22:54, Fujii Masao wrote: On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The foll

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread Fujii Masao
On 2020/08/19 9:43, torikoshia wrote: On 2020-08-18 22:54, Fujii Masao wrote: On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread torikoshia
On 2020-08-18 22:54, Fujii Masao wrote: On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through the commitfest application: make installc

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread Fujii Masao
On 2020/08/18 18:41, torikoshia wrote: On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through the commitfest application: make installcheck-world:  tested, passed Implements

Re: Creating a function for exposing memory usage of backend process

2020-08-18 Thread torikoshia
On 2020-08-17 21:19, Fujii Masao wrote: On 2020/08/17 21:14, Fujii Masao wrote: On 2020-08-07 16:38, Kasahara Tatsuhito wrote: The following review has been posted through the commitfest application: make installcheck-world:  tested, passed Implements feature:   tested, passed Spec complia

Re: Creating a function for exposing memory usage of backend process

2020-08-17 Thread Fujii Masao
On 2020/08/17 21:14, Fujii Masao wrote: On 2020/08/11 15:24, torikoshia wrote: On 2020-08-08 10:44, Michael Paquier wrote: On Fri, Jul 31, 2020 at 03:23:52PM -0400, Robert Haas wrote: On Fri, Jul 31, 2020 at 4:25 AM torikoshia wrote: And as Fujii-san told me in person, exposing memory a

Re: Creating a function for exposing memory usage of backend process

2020-08-17 Thread Fujii Masao
On 2020/08/11 15:24, torikoshia wrote: On 2020-08-08 10:44, Michael Paquier wrote: On Fri, Jul 31, 2020 at 03:23:52PM -0400, Robert Haas wrote: On Fri, Jul 31, 2020 at 4:25 AM torikoshia wrote: And as Fujii-san told me in person, exposing memory address seems not preferable considering the

Re: Creating a function for exposing memory usage of backend process

2020-08-10 Thread torikoshia
On 2020-08-08 10:44, Michael Paquier wrote: On Fri, Jul 31, 2020 at 03:23:52PM -0400, Robert Haas wrote: On Fri, Jul 31, 2020 at 4:25 AM torikoshia wrote: And as Fujii-san told me in person, exposing memory address seems not preferable considering there are security techniques like address spa

Re: Creating a function for exposing memory usage of backend process

2020-08-07 Thread Michael Paquier
On Fri, Jul 31, 2020 at 03:23:52PM -0400, Robert Haas wrote: > On Fri, Jul 31, 2020 at 4:25 AM torikoshia wrote: >> And as Fujii-san told me in person, exposing memory address seems >> not preferable considering there are security techniques like >> address space layout randomization. > > Yeah, e

Re: Creating a function for exposing memory usage of backend process

2020-08-07 Thread Kasahara Tatsuhito
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: not tested Documentation:tested, passed I tested the latest patch(0007-Adding-a-function-exposing-memory

Re: Creating a function for exposing memory usage of backend process

2020-07-31 Thread Robert Haas
On Fri, Jul 31, 2020 at 4:25 AM torikoshia wrote: > And as Fujii-san told me in person, exposing memory address seems > not preferable considering there are security techniques like > address space layout randomization. Yeah, exactly. ASLR wouldn't do anything to improve security if there were no

Re: Creating a function for exposing memory usage of backend process

2020-07-31 Thread torikoshia
On 2020-07-30 15:13, Kasahara Tatsuhito wrote: Hi, On Fri, Jul 10, 2020 at 5:32 PM torikoshia wrote: - whether information for identifying parent-child relation is necessary or it's an overkill I think it's important to understand the parent-child relationship of the context. Personally, I

Re: Creating a function for exposing memory usage of backend process

2020-07-29 Thread Kasahara Tatsuhito
Hi, On Fri, Jul 10, 2020 at 5:32 PM torikoshia wrote: > - whether information for identifying parent-child relation is necessary > or it's an overkill I think it's important to understand the parent-child relationship of the context. Personally, I often want to know the following two things .. -

Re: Creating a function for exposing memory usage of backend process

2020-07-12 Thread Fujii Masao
On 2020/07/10 17:32, torikoshia wrote: On 2020-07-09 02:03, Andres Freund wrote: Hi, I think this is an incredibly useful feature. Thanks for your kind comments and suggestion! On 2020-07-07 22:02:10 +0900, torikoshia wrote: > There can be multiple memory contexts with the same name. S

Re: Creating a function for exposing memory usage of backend process

2020-07-10 Thread torikoshia
On 2020-07-09 02:03, Andres Freund wrote: Hi, I think this is an incredibly useful feature. Thanks for your kind comments and suggestion! On 2020-07-07 22:02:10 +0900, torikoshia wrote: > There can be multiple memory contexts with the same name. So I'm afraid > that it's difficult to ident

Re: Creating a function for exposing memory usage of backend process

2020-07-10 Thread torikoshia
On 2020-07-08 22:12, Fujii Masao wrote: Thanks for updating the patch! It basically looks good to me. + + backend memory contexts + Do we need this indexterm? Thanks! it's not necessary. I remove this indexterm. +{ oid => '2282', descr => 'statistics: information about all memory co

Re: Creating a function for exposing memory usage of backend process

2020-07-08 Thread Andres Freund
Hi, I think this is an incredibly useful feature. On 2020-07-07 22:02:10 +0900, torikoshia wrote: > > There can be multiple memory contexts with the same name. So I'm afraid > > that it's difficult to identify the actual parent memory context from > > this > > "parent" column. This is ok when lo

Re: Creating a function for exposing memory usage of backend process

2020-07-08 Thread Fujii Masao
On 2020/07/07 22:02, torikoshia wrote: On 2020-07-06 15:16, Fujii Masao wrote: On 2020/07/06 12:12, torikoshia wrote: On Fri, Jul 3, 2020 at 7:33 PM Fujii Masao wrote: Thanks for your review! I like more specific name like pg_backend_memory_contexts. Agreed. When I was trying to add t

Re: Creating a function for exposing memory usage of backend process

2020-07-07 Thread torikoshia
On 2020-07-06 15:16, Fujii Masao wrote: On 2020/07/06 12:12, torikoshia wrote: On Fri, Jul 3, 2020 at 7:33 PM Fujii Masao wrote: Thanks for your review! I like more specific name like pg_backend_memory_contexts. Agreed. When I was trying to add this function as statistics function, I tho

Re: Creating a function for exposing memory usage of backend process

2020-07-05 Thread Fujii Masao
On 2020/07/06 12:12, torikoshia wrote: On Fri, Jul 3, 2020 at 7:33 PM Fujii Masao wrote: Thanks for your review! I like more specific name like pg_backend_memory_contexts. Agreed. When I was trying to add this function as statistics function, I thought that naming pg_stat_getbackend_mem

Re: Creating a function for exposing memory usage of backend process

2020-07-05 Thread torikoshia
On Fri, Jul 3, 2020 at 7:33 PM Fujii Masao wrote: Thanks for your review! I like more specific name like pg_backend_memory_contexts. Agreed. When I was trying to add this function as statistics function, I thought that naming pg_stat_getbackend_memory_context() might make people regarded i

Re: Creating a function for exposing memory usage of backend process

2020-07-03 Thread Fujii Masao
On 2020/07/03 11:45, torikoshia wrote: On Wed, Jul 1, 2020 at 10:15 PM torikoshia wrote: I'm going to do some renaming and transportations. - view name: pg_memory_contexts I like more specific name like pg_backend_memory_contexts. But I'd like to hear more opinions about the name from oth

Re: Creating a function for exposing memory usage of backend process

2020-07-02 Thread torikoshia
On Wed, Jul 1, 2020 at 10:15 PM torikoshia wrote: I'm going to do some renaming and transportations. - view name: pg_memory_contexts - function name: pg_get_memory_contexts() - source file: mainly src/backend/utils/mmgr/mcxt.c Attached an updated patch. On Wed, Jul 1, 2020 at 10:58 PM Fujii

Re: Creating a function for exposing memory usage of backend process

2020-07-01 Thread Fujii Masao
On 2020/07/01 22:15, torikoshia wrote: On Wed, Jul 1, 2020 at 4:43 PM Fujii Masao wrote: Thanks for reviewing! You treat pg_stat_local_memory_contexts view as a dynamic statistics view. But isn't it better to treat it as just system view like pg_shmem_allocations or pg_prepared_statements 

Re: Creating a function for exposing memory usage of backend process

2020-07-01 Thread torikoshia
On 2020-07-01 20:47, Daniel Gustafsson wrote: For the next version (if there will be one), please remove the catversion bump from the patch as it will otherwise just break patch application without constant rebasing (as it's done now). The committer will handle the catversion change if/when

Re: Creating a function for exposing memory usage of backend process

2020-07-01 Thread torikoshia
On Wed, Jul 1, 2020 at 4:43 PM Fujii Masao wrote: Thanks for reviewing! You treat pg_stat_local_memory_contexts view as a dynamic statistics view. But isn't it better to treat it as just system view like pg_shmem_allocations or pg_prepared_statements  because it's not statistics information?

Re: Creating a function for exposing memory usage of backend process

2020-07-01 Thread Daniel Gustafsson
> On 1 Jul 2020, at 07:48, torikoshia wrote: > Attached a patch for creating a view for local memory context > and its explanation on the document. For the next version (if there will be one), please remove the catversion bump from the patch as it will otherwise just break patch application with

Re: Creating a function for exposing memory usage of backend process

2020-07-01 Thread Fujii Masao
On 2020/07/01 14:48, torikoshia wrote: On Mon, Jun 29, 2020 at 3:13 PM Fujii Masao wrote: Could you add this patch to Commitfest 2020-07? Thanks for notifying, I've added it. BTW, I registered you as an author because this patch used lots of pg_cheat_funcs' codes.   https://commitfest.p

Re: Creating a function for exposing memory usage of backend process

2020-06-30 Thread torikoshia
On Mon, Jun 29, 2020 at 3:13 PM Fujii Masao wrote: Could you add this patch to Commitfest 2020-07? Thanks for notifying, I've added it. BTW, I registered you as an author because this patch used lots of pg_cheat_funcs' codes. https://commitfest.postgresql.org/28/2622/ This patch provide

Re: Creating a function for exposing memory usage of backend process

2020-06-28 Thread Bharath Rupireddy
> > If there is no such way to know the cache sizes and other info such as > > statistics, number of entries, cache misses, hits etc. can the > > approach discussed here be applied? > I think it's partially yes. > > > If the user knows the cache statistics and other information, may be > > we can

Re: Creating a function for exposing memory usage of backend process

2020-06-28 Thread Fujii Masao
On 2020/06/29 12:01, torikoshia wrote: On 2020-06-20 03:11, Robert Haas wrote: On Wed, Jun 17, 2020 at 11:56 PM Fujii Masao wrote: > As a first step, to deal with (3) I'd like to add > pg_stat_get_backend_memory_context() which target is limited to the > local backend process. +1 +1 from

Re: Creating a function for exposing memory usage of backend process

2020-06-28 Thread torikoshia
On 2020-06-20 03:11, Robert Haas wrote: On Wed, Jun 17, 2020 at 11:56 PM Fujii Masao wrote: > As a first step, to deal with (3) I'd like to add > pg_stat_get_backend_memory_context() which target is limited to the > local backend process. +1 +1 from me, too. Attached a patch that adds a fu

Re: Creating a function for exposing memory usage of backend process

2020-06-26 Thread Kasahara Tatsuhito
Hi, On Fri, Jun 26, 2020 at 3:42 PM Bharath Rupireddy wrote: > While going through the mail chain on relation, plan and catalogue > caching [1], I'm thinking on the lines that is there a way to know the > current relation, plan and catalogue cache sizes? If there is a way > already, please ignor

Re: Creating a function for exposing memory usage of backend process

2020-06-25 Thread Bharath Rupireddy
Hi, While going through the mail chain on relation, plan and catalogue caching [1], I'm thinking on the lines that is there a way to know the current relation, plan and catalogue cache sizes? If there is a way already, please ignore this and it would be grateful if someone point me to that. Post

Re: Creating a function for exposing memory usage of backend process

2020-06-25 Thread Kasahara Tatsuhito
Hi ! On Thu, Jun 18, 2020 at 12:56 PM Fujii Masao wrote: > Agreed. The feature to view how local memory contexts are used in > each process is very useful! +1 > >=# SELECT * FROM pg_stat_get_backend_memory_context(PID); > > I'm afraid that this interface is not convenient when we want to mon

Re: Creating a function for exposing memory usage of backend process

2020-06-19 Thread Robert Haas
On Wed, Jun 17, 2020 at 11:56 PM Fujii Masao wrote: > > As a first step, to deal with (3) I'd like to add > > pg_stat_get_backend_memory_context() which target is limited to the > > local backend process. > > +1 +1 from me, too. Something that exposed this via shared memory would be quite useful,

Re: Creating a function for exposing memory usage of backend process

2020-06-17 Thread Fujii Masao
On 2020/06/17 22:00, torikoshia wrote: Hi, As you may know better than I do, backend processes sometimes use a lot of memory because of the various reasons like caches, prepared statements and cursors. When supporting PostgreSQL, I face situations for investigating the reason of memory bloat.

Creating a function for exposing memory usage of backend process

2020-06-17 Thread torikoshia
Hi, As you may know better than I do, backend processes sometimes use a lot of memory because of the various reasons like caches, prepared statements and cursors. When supporting PostgreSQL, I face situations for investigating the reason of memory bloat. AFAIK, the way to examine it is attaching