Ah. The "leak" I am thinking of is more an unreleased use of memory -
we should make sure that we activate AND deactivate indexes eventually
and don't just have more and more in-memory state over time. We should
probably do a "lifecycle review" of how we manage (i) metadata cache
entries, (ii) index metadata, and (iii) in-memory index components -
anything that has a memory footprint at runtime - and make sure we know
when each first hits memory and then UN-hits memory at some reasonable
time. We should also make sure that not all indexes have in-memory
components; if we are only reading them we shouldn't need those.
Perhaps we could do this in a "crowd-sourced" Google doc that identifies
the key lifecycle events (create index, access dataset for first time,
access index of dataset for first time, hit limit on number of active
datasets, ....).
On 7/22/15 12:58 AM, Ian Maxon wrote:
Index open/close is probably fine. I haven't traced it in a debugger
or anything, but I believe it would just look up the index in the map
rather than creating a new object. The only way to create a leak is to
create many unique indexes within an NC's lifetime.
On Wed, Jul 22, 2015 at 12:39 AM, Mike Carey <[email protected]> wrote:
But my guess is that there's similar unfortunate behavior related to index
open and close...?
(Which would happen way more regularly?) Agreed w.r.t. priority if we are
in "prototype" mode,
but if anyone wanted to go "live" and run for months at a time, this would
be a leak we'd feel,
potentially. We need to devise and do some testing around index/memory
management and
the lifecycle there. :-)
On 7/22/15 12:25 AM, abdullah alamoudi wrote:
It should be fixed but I don't think that it is a release blocker since
this rarely happen and the amount of memory per index instance object is
almost nothing.
just my 2 cents.
On Wed, Jul 22, 2015 at 10:10 AM, Michael Carey <[email protected]>
wrote:
Definitely a high priority bug - we should try to fix this pre-release!
-------- Forwarded Message --------
Subject: Issue 910 in asterixdb: Index creation/use can cause
resource leakage
Date: Wed, 22 Jul 2015 03:13:59 +0000
From: [email protected]
Reply-To: [email protected]
To: [email protected]
Status: Accepted
Owner: [email protected]
CC: [email protected]
Labels: Type-Defect Priority-High
New issue 910 by [email protected]: Index creation/use can cause resource
leakage
https://code.google.com/p/asterixdb/issues/detail?id=910
In IndexDataflowHelper, calling create() will make an index instance, and
create it. This inserts an entry into the IndexLifecycleManager's
resource
map. Entries in this map are only freed once unregister() is called on an
index. However this only happens when an index is destroyed, not when it
is
closed. There is no other way in which references to indexes are garbage
collected (even though there is a reference counter in the IndexInfo).
Therefore creating or using an index uses resources in this map even if
they are not active, and were created during the lifetime of the NC- and
there is no way to free that memory.
--
You received this message because this project is configured to send all
issue notifications to this address.
You may adjust your notification preferences at:
https://code.google.com/hosting/settings
--
You received this message because you are subscribed to the Google Groups
"asterixdb-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [email protected].
For more options, visit https://groups.google.com/d/optout.
