Hello. At Wed, 28 Nov 2018 05:13:26 +0000, "Ideriha, Takeshi" <ideriha.take...@jp.fujitsu.com> wrote in <4E72940DA2BF16479384A86D54D0988A6F3BD73A@G01JPEXMBKW04> > Hi > > >From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com] > >Here we need something better, because once this new kind of memory fills up > >with > >leaked objects, you have to restart the whole cluster to recover. > > Thank you for the clarification. I agree that leaving memory leaking in > shared memory > without freeing it ends up cluster-wide problem.
If some of the "leaked" objects are liked each other with other "live" objects including those on local memory, we must tell whether each "maybe-leaked" object is actually a leak or not. That looks apprently stupid. Maybe we shouldn't manipulate objects on shared memory at such a low-level interface. We would need a kind of resource manager for such structures holding shared objects and should (perhaps explicitly) register the objects that shuold be free'd on error-exiting from a scope. This seems a bit different from reset callbacks of the current MemoryContext. > >If the solution involves significantly different control flows (like > >PG_TRY(), > >PG_FINALLY() manual cleanup all over the place), then lots of existing > >palloc-based > >code will need to be reviewed and rewritten very carefully for use in this > >new kind of > >memory context. If it can be done automagically, then more of our existing > >code will > >be able to participate in the construction of stuff that we want to put in > >shared > >memory. > > About resetting X->a I was thinking about how to treat dangling pointer > inside a new memory > context machinery instead of using PG_TRY() stuff. That's because putting > PG_TRY() all over the > place becomes responsible for a developer trying to shared-memory-version > MemoryContext and > it needs a lot of efforts as you noted. But in my mind even if PG_TRY() is > used, it only affects new > code uses shared-memory version MemoryContext and doesn't affect current code. > I may be missing something here... Thomas mentioned exiting upper-layer library code using palloc, like pg_list. As I wrote above, pg_list is one of such feature that would need to be carefully adjusted. > >I first thought about this in the context of shared plan caches, a topic > >that appears on > >this mailing list from time to time. There are some other fun problems, > >like cache > >invalidation for shared caches, space recycling (LRUs etc), and of course > >locking/concurrency > >(dsa_allocate() and dsa_free() are concurrency safe, but whatever data > >structures > >you build with the memory of course need their own plan for dealing with > >concurrency). > >But a strategy for clean-up on errors seems to be a major subproblem to deal > >with > >early in the project. I will be very happy if we can come up with something > >:-) > > Yeah, I hope we can do it somehow. In addition to recycling issue within a fixed area, I suppose that we will need to "expand" the fixed-address shared memory. PROT_NONE could work for that but it must break some platforms.. > >On Wed, Nov 14, 2018 at 12:40 AM Ideriha, Takeshi > ><ideriha.take...@jp.fujitsu.com> > >wrote: > >> From: Thomas Munro [mailto:thomas.mu...@enterprisedb.com] > >> >postgres=# call hoge_list(3); > >> >NOTICE: Contents of list: > >> >NOTICE: 1 > >> >NOTICE: 2 > >> >NOTICE: 3 > >> >CALL > >> > > >> >You can call that procedure from various different backends to add > >> >and remove integers from a List. The point is that it could just as > >> >easily be a query plan or any other palloc-based stuff in our tree, and > >> >the pointers > >work in any backend. > >> > >> Thanks for the patch. I know it's a rough sketch but basically that's what > >> I want to > >do. > >> I tried it and it works well. > > > >After your message I couldn't resist a small experiment. But it seems there > >are > >plenty more problems to be solved to make it useful... > > Thank you for kind remark. > By the way I just thought meta variable "hoge" is used only in Japan :) I've seen it many-many times including mine in this list, thus why don't you know it is already a world-wide jargon? :p regards. -- Kyotaro Horiguchi NTT Open Source Software Center
