Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13326 )
Change subject: IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load ...................................................................... IMPALA-8537: Negative values reported for tmp-file-mgr.scratch-space-bytes-used under heavy spilling load Whenever closing a FileGroup, the TmpFileMgr::scratch_bytes_used_metric_ was incorrectly being decremented by the total scratch space bytes across the entire FileGroup (i.e FileGroup::scratch_space_bytes_used_counter_), for every File in the FileGroup. This was resulting in the -ive value for the current scratch space bytes. The fix is to decrement the TmpFileMgr::scratch_bytes_used_metric_ by the FileGroup::scratch_space_bytes_used_counter_, only once when the FileGroup is closed. Testing Done: - Added checks for expected current value and HWM of the scratch space bytes in some of the existing test units in tmp-file-mgr-test.cc. - Added a new scenario in tmp-file-mgr-test.cc which mimics concurrent spilling queries and checks for propper current and HWM value for the scratch space bytes. - Ad-hoc tests forcing multiple scratch space dirs/files and running concurrent spilling queries while making sure the current value is never -ive. Change-Id: I338ecc06ddfad414091bd50f683b767b61abdcc4 Reviewed-on: http://gerrit.cloudera.org:8080/13326 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/util/metrics.h 3 files changed, 86 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13326 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I338ecc06ddfad414091bd50f683b767b61abdcc4 Gerrit-Change-Number: 13326 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat <ara...@cloudera.com> Gerrit-Reviewer: David Rorke <dro...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>
