Hi Thomas,
Thanks for cautioning us about possible memory leaks(during error cases)
incase of long-lived DSA segements.
Actually we are following an approach to avoid this DSA memory leaks. Let
me explain our implementation and please validate and correct us in-case we
miss anything.
Implementation:
Basically we have to put our index data into memory (Index Column Value
Vs Ctid) which we get in aminsert callback function.
Coming to the implementation, in aminsert Callback function,
We Switch to CurTransactionContext
Cache the DMLs of a transaction into dlist(global per process)
Even if different clients work parallel, it won't be a problem because every
client gets one dlist in separate process and it'll have it's own
CurTransactionContext
We have registered transaction callback (using RegisterXactCallback()
function). And during event pre-commit(XACT_EVENT_PRE_COMMIT), we populate all
the transaction specific DMLs (from dlist) into our in-memory index(DSA)
obviously inside PG_TRY/PG_CATCH block.
In case we got some errors(because of dsa_allocate() or something else) while
processing dlist(while populating in-memory index), we cleanup the DSA memory
in PG_CATCH block that is allocated/used till that point.
During other error cases, typically transactions gets aborted and PRE_COMMIT
event is not called and hence we don't touch DSA at that time. Hence no need to
bother about leaks.
Even sub transaction case is handled with sub transaction callbacks.
CurTransactionContext(dlist basically) is automatically cleared after that
particular transaction.
I want to know if this approach is good and works well in all cases. Kindly
provide your feedback on this.
Regards
G. Sai Ram
---- On Wed, 20 Sep 2017 14:25:43 +0530 Thomas Munro
<thomas.mu...@enterprisedb.com> wrote ----
On Wed, Sep 20, 2017 at 6:14 PM, Gaddam Sai Ram
<gaddamsaira...@zohocorp.com> wrote:
> Thank you very much! That fixed my issue! :)
> I was in an assumption that pinning the area will increase its lifetime
but
> yeah after taking memory context into consideration its working fine!
So far the success rate in confusing people who first try to make
long-lived DSA areas and DSM segments is 100%. Basically, this is all
designed to ensure automatic cleanup of resources in short-lived
scopes.
Good luck for your graph project. I think you're going to have to
expend a lot of energy trying to avoid memory leaks if your DSA lives
as long as the database cluster, since error paths won't automatically
free any memory you allocated in it. Right now I don't have any
particularly good ideas for mechanisms to deal with that. PostgreSQL
C has exception-like error handling, but doesn't (and probably can't)
have a language feature like scoped destructors from C++. IMHO
exceptions need either destructors or garbage collection to keep you
sane. There is a kind of garbage collection for palloc'd memory and
also for other resources like file handles, but if you're using a big
long lived DSA area you have nothing like that. You can use
PG_TRY/PG_CATCH very carefully to clean up, or (probably better) you
can try to make sure that all your interaction with shared memory is
no-throw (note that that means using dsa_allocate_extended(x,
DSA_ALLOC_NO_OOM), because dsa_allocate itself can raise errors). The
first thing I'd try would probably be to keep all shmem-allocating
code in as few routines as possible, and use only no-throw operations
in the 'critical' regions of them, and maybe look into some kind of
undo log of things to free or undo in case of error to manage
multi-allocation operations if that turned out to be necessary.
--
Thomas Munro
http://www.enterprisedb.com
