[
https://issues.apache.org/jira/browse/IMPALA-15010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106969#comment-18106969
]
ASF subversion and git services commented on IMPALA-15010:
----------------------------------------------------------
Commit b21595276c6c84161eb3194228df016f120ffc17 in impala's branch
refs/heads/master from Joe McDonnell
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=b21595276 ]
IMPALA-15010: Patch gperftools to use size_t for thread cache sizes
This bumps the toolchain, which included a few different fixes.
The primary fix is that gperftools has been patched to use
size_t for thread cache sizes. This would prevent exotic
scenarios where the thread cache size integer could wrap
around to become negative. Google TCMalloc had already made
this change for their thread caching implementation.
The new toolchain also has a different structure for the
layout of the ARM hadoop-client tarball for IMPALA-12975.
When the existing logic in buildall.sh copies the binaries
to the regular HADOOP_HOME, there is a time when the
binaries are partially written that can crash the minicluster.
The original idea for a fix was to leave the ARM binaries
in a different directory and refer to them there.
It turns out to be tedious to get the minicluster to respect
libraries at a separate location from the HADOOP_HOME
location. Instead, this modifies the logic in buildall.sh to
set up a symlink to the ARM hadoop-client location and
only modify it if it is pointing to the wrong place. This
should avoid disrupting a running minicluster.
Testing:
- Ran core jobs on x86_64 and ARM
- Ran a perf-AB-test on ARM
Change-Id: I9dc5e95352f593918f9cacf26786149c8c79b4a2
Reviewed-on: http://gerrit.cloudera.org:8080/24484
Reviewed-by: Joe McDonnell <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Michael Smith <[email protected]>
> Patch gperftools to prevent integer wrap when
> tcmalloc_max_total_thread_cache_bytes > INT_MAX
> ---------------------------------------------------------------------------------------------
>
> Key: IMPALA-15010
> URL: https://issues.apache.org/jira/browse/IMPALA-15010
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Assignee: Joe McDonnell
> Priority: Major
> Fix For: Impala 5.0.0
>
>
> The size and max_size counters for a thread cache are stored as a 32-bit
> signed integers inside gperftools. With the default
> tcmalloc_max_total_thread_cache_bytes, this doesn't matter. As
> tcmalloc_max_total_thread_cache_bytes goes above INT_MAX, it becomes
> hypothetically possible for the size or max size to wrap and go negative.
> This should be hard to hit, but it would be better to patch gperftools to
> prevent it. In google tcmalloc, they use size_t for the size and max_size
> counters in the thread cache.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
