Hi Jianfeng, I just want to clarify that the issue causing the exception is the fact metadata datasets are being evicted and not opened again. This is caused by the special treat of metadata indexes (not going thru the same code path as other indexes). I have already filed an issue[1] for it and will submit a fix today that will fix the exception issue.
The dynamic memory allocation needs its separate discussion. Cheers, Murtadha [1] https://issues.apache.org/jira/browse/ASTERIXDB-1338 <https://issues.apache.org/jira/browse/ASTERIXDB-1338> > On Mar 10, 2016, at 2:46 PM, Jianfeng Jia <[email protected]> wrote: > > Dear Devs, > > I have some questions about the memory management of the in-memory components > for different datasets. > > The current AsterixDB backing the cloudberry demo is down every few days. It > always throws an exception like following: > Caused by: org.apache.hyracks.api.exceptions.HyracksDataException: Failed to > open index with resource ID 7 since it does not exist. > > As described in ASTERIXDB-1337, each dataset has a fixed budget no matter how > small/big it is. Then the number of datasets can be loaded at the same time > is also fixed by $number = > storage.memorycomponent.globalbudget/storage.memorycomponent.numpages. My > question is if we have more than $number of datasets, then the eviction will > happen? Will it evict a entire dataset of the victim? Base on the symptom of > above exception, it seems the metadata get evicted? Could we protect the > metadata from eviction? > > A more fundament question Is it possible that all those datasets share a > global budget in a multi-tenant way? > In my workload there are one main dataset( ~10Gb) and five tiny auxiliary > datasets (each size <20M). In addition, the client will create a bunch of > temporary datasets depends on how many concurrent users are and each > temp-dataset will be “refreshed" for a new query. (The refresh is done by > drop and create the temp-dataset). It’s hard to find one > storage.memorycomponent.numpages that make every dataset happy. > > > > Best, > > Jianfeng Jia > PhD Candidate of Computer Science > University of California, Irvine >
