Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/12683 )
Change subject: Prototype for a remote read byte cache. ...................................................................... Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/12683/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/12683/1//COMMIT_MSG@63 PS1, Line 63: you'd actually want to integrate this with the general scratch space > Currently we don't have global limits in TmpFileMgr, just per-query limits It seems like spilling is going to be pretty sequential, so spinning cares less about SSD than caching does. That's another angle to reconcile. Anyway--I agree that this is tractable. We don't want to run the user out of disk space, and evicting cache is a very reasonable thing to do. http://gerrit.cloudera.org:8080/#/c/12683/1//COMMIT_MSG@92 PS1, Line 92: useful. The buffer pool code currently provides buffers that are pinned > I think having the pages pinned by default is the right thing for this case Yep. I think the underlying question here is how much we want to "manage" this. In the traditional "local" case, we don't manage the OS buffer cache, and it definitely helps us out. I could definitely imagine a world where Impala has a default cache size of ~4GB that's always devoted to cache. The rest of Impala "managed" (as in, within memlimit) memory gets used by the cache if it's available, and evicted as necessary. The cache here is the "memory user of last resort". Your point on observability is well-taken. Remote reads incur a write-to-cache cost that's kind of unexpected. Fortunately, we presume the disk is fairly idle given that the reads are happening over the network, but it'll be an important thing to surface. -- To view, visit http://gerrit.cloudera.org:8080/12683 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic312b0f7ac7875e00a3855ef21dce5b8a9aa67c5 Gerrit-Change-Number: 12683 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger <phi...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Wed, 06 Mar 2019 20:14:26 +0000 Gerrit-HasComments: Yes
