Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/16220 )
Change subject: IMPALA-9989 Improve admission control pool stats logging ...................................................................... Patch Set 34: Code-Review+1 (8 comments) Looks good, just a few more nits http://gerrit.cloudera.org:8080/#/c/16220/34//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16220/34//COMMIT_MSG@115 PS34, Line 115: Core tests nit: Ran core tests successfully. http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/runtime/mem-tracker-test.cc File be/src/runtime/mem-tracker-test.cc: http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/runtime/mem-tracker-test.cc@389 PS34, Line 389: 100 nit: can use NUM_QUERY_MEM_TRACKERS here and the line above http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/runtime/mem-tracker.cc File be/src/runtime/mem-tracker.cc: http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/runtime/mem-tracker.cc@449 PS34, Line 449: // Collect the top N queries into a priority queue, and computes the min, the max, : // the total memory consumption, and the total number of all queries that are : // memory tracked by query memory trackers. This method should only be called for : // mem-trackers that are either query mem trackers or higher in the mem tracker : // hierarchy. nit: the comment in the header file is sufficient since this is echoing the same thing here. http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/runtime/mem-tracker.cc@461 PS34, Line 461: DCHECK(tracker->is_query_mem_tracker_); nit: can you print the mem-tracker->label here so thats its easy to debug in case it is hit http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/runtime/mem-tracker.cc@476 PS34, Line 476: // Append to ss a debug string for memory consumption part of the pool stats. : // Here is one example. : // topN_query_stats: queries=[554b016cf0f3a37f:9a1bfcfd00000000, : // 464dcd9cc47d724b:9e6a3f6400000000, 2844275a1458bf1f:0bc5887500000000, : // a449dbc7bcbd2af1:647e6ded00000000, 8c430ea5ad38e94a:3c27bf4400000000], : // total_mem_consumed=1.26 MB, fraction_of_pool_total_mem=0.61; pool_level_stats: : // num_running=10, min=0, max=257.48 KB, pool_total_mem=2.06 MB, average_per_query=210.74 : // KB : void MemTracker::AppendStatsForConsumedMemory(stringstream& ss, const TPoolStats& stats) { : ss << "topN_query_stats: "; : ss << "queries=["; : int num_ids = stats.heavy_memory_queries.size(); : int64_t total_memory_consumed_by_top_queries = 0; : for (int i = 0; i < num_ids; i++) { : auto& query = stats.heavy_memory_queries[i]; : total_memory_consumed_by_top_queries += query.memory_consumed; : ss << PrintId(query.queryId); : if (i < num_ids - 1) ss << ", "; : } : ss << "], "; : ss << "total_mem_consumed=" : << PrettyPrinter::PrintBytes(total_memory_consumed_by_top_queries); : int64_t total_memory_consumed = stats.total_memory_consumed; : if (total_memory_consumed > 0) { : ss << ", fraction_of_pool_total_mem=" << setprecision(2) : << float(total_memory_consumed_by_top_queries) / total_memory_consumed; : } : ss << "; "; : : ss << "pool_level_stats: "; : ss << "num_running=" << stats.num_running << ", "; : ss << "min=" << PrettyPrinter::PrintBytes(stats.min_memory_consumed) << ", "; : ss << "max=" << PrettyPrinter::PrintBytes(stats.max_memory_consumed) << ", "; : ss << "pool_total_mem=" << PrettyPrinter::PrintBytes(total_memory_consumed); : if (stats.num_running > 0) { : ss << ", average_per_query=" : << PrettyPrinter::PrintBytes(total_memory_consumed / stats.num_running); : } : } nit: this functionality is very admission control focused and it also only uses its related structures. It would be great if you can move this back there. http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/scheduling/admission-controller.h File be/src/scheduling/admission-controller.h: http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/scheduling/admission-controller.h@1053 PS34, Line 1053: getName nit: can you mention that this can be both the pool name and the host http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/scheduling/admission-controller.h@1054 PS34, Line 1054: getTUniqueId nit: mention here that this is the query id http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/scheduling/admission-controller.cc File be/src/scheduling/admission-controller.cc: http://gerrit.cloudera.org:8080/#/c/16220/34/be/src/scheduling/admission-controller.cc@816 PS34, Line 816: if ( not_admitted_details ) : *not_admitted_details = GetLogStringForTopNQueriesOnHost(host_id); nit: enclosing brackets for multi line if statement -- To view, visit http://gerrit.cloudera.org:8080/16220 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id995a9d044082c3b8f044e1ec25bb4c64347f781 Gerrit-Change-Number: 16220 Gerrit-PatchSet: 34 Gerrit-Owner: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Qifan Chen <qc...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Thu, 13 Aug 2020 07:13:44 +0000 Gerrit-HasComments: Yes
