Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/12262 )
Change subject: IMPALA-7941: part 2/2: use cgroups memory limit ...................................................................... Patch Set 2: (4 comments) Logic-wise, this looks good. A couple comments about logging and code comments. http://gerrit.cloudera.org:8080/#/c/12262/2/be/src/runtime/exec-env.cc File be/src/runtime/exec-env.cc: http://gerrit.cloudera.org:8080/#/c/12262/2/be/src/runtime/exec-env.cc@233 PS2, Line 233: // Depending on the system configuration, we will have to calculate the process : // memory limit either based on the available physical memory, or if overcommitting : // is turned off, we use the memory commit limit from /proc/meminfo (see : // IMPALA-1690). Can you update this comment to describe how cgroup mem limit fits in? http://gerrit.cloudera.org:8080/#/c/12262/2/be/src/runtime/exec-env.cc@358 PS2, Line 358: if (bytes_limit > MemInfo::physical_mem()) { : LOG(WARNING) << "Memory limit " : << PrettyPrinter::Print(bytes_limit, TUnit::BYTES) : << " exceeds physical memory of " : << PrettyPrinter::Print(MemInfo::physical_mem(), TUnit::BYTES); : } If there is a cgroup limit, an equivalent of this logging would be useful (i.e. bytes_limit > cgroup_mem_limit). http://gerrit.cloudera.org:8080/#/c/12262/2/be/src/runtime/exec-env.cc@364 PS2, Line 364: LOG(INFO) << "Using global memory limit: " : << PrettyPrinter::Print(bytes_limit, TUnit::BYTES); We might consider adding some information to indicate how the value was set. i.e. Using global memory limit: X (physical system memory) Using global memory limit: X (cgroup memory limit) Using global memory limit: X (constant mem_limit parameter) Using global memory limit: X (percentage mem_limit parameter, Y% of physical system memory) Using global memory limit: X (percentage mem_limit parameter, Y% of cgroup memory limit) http://gerrit.cloudera.org:8080/#/c/12262/2/docker/daemon_entrypoint.sh File docker/daemon_entrypoint.sh: http://gerrit.cloudera.org:8080/#/c/12262/2/docker/daemon_entrypoint.sh@45 PS2, Line 45: export JAVA_TOOL_OPTIONS="-Xmx2g $JAVA_TOOL_OPTIONS" You might find https://github.com/apache/impala/blob/master/docker/entrypoint.sh#L350-L368 to be interesting. We might want to set the JVM size for the normal minicluster, but that shouldn't block this change. -- To view, visit http://gerrit.cloudera.org:8080/12262 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie9fb4fb936a46fc194a204391d03c07c8c7fba21 Gerrit-Change-Number: 12262 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Reviewer: Bikramjeet Vig <bikramjeet....@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com> Gerrit-Comment-Date: Thu, 24 Jan 2019 19:13:43 +0000 Gerrit-HasComments: Yes
