Luo Chen has posted comments on this change. Change subject: [ASTERIXDB-2276][CONF] Introduce Max Active Writable Datasets ......................................................................
Patch Set 1: (1 comment) https://asterix-gerrit.ics.uci.edu/#/c/2362/1//COMMIT_MSG Commit Message: PS1, Line 19: - Introduce max active writable datasets and use : it to automatically calculate the number of pages : of a user dataset memory component. > as a user, would you rather configure a number related to datasets - someth Hard to say. For me, I would rather configure storage.memorycomponent.numpages by myself to have more control over the system performance. It's meaningless to write many datasets at the same time without a performance guarantee. This parameter is one key knob for ingestion performance. Setting max active number of datasets might seem to be "user-friendly", but it rather leaves the storage.memorycomponent.numpages as a black box, since we have to account for datasets, partitions, secondary indexes etc, to compute the actual num pages paramter. It's not quite straightforward what is the actual size of each memory component. Though the current configuration has the similar problem. Another issue with max active writable datasets is that this parameter may cause some confusion. Say I have 10GB memory for global memory components, and I set this number to 100. However, if I only have 1 active dataset for write, what the system should behave? A "smart" system might take as much memory resources as possible to improve the ingestion performance to this only active writing dataset. However, this is not the behavior of our current system. -- To view, visit https://asterix-gerrit.ics.uci.edu/2362 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: comment Gerrit-Change-Id: I9909c26b1e12b431f913e201d2c3d83769be7269 Gerrit-PatchSet: 1 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail <mhub...@apache.org> Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Jenkins <jenk...@fulliautomatix.ics.uci.edu> Gerrit-Reviewer: Luo Chen <cl...@uci.edu> Gerrit-Reviewer: Murtadha Hubail <mhub...@apache.org> Gerrit-HasComments: Yes
