[
https://issues.apache.org/jira/browse/IMPALA-12915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830591#comment-17830591
]
ASF subversion and git services commented on IMPALA-12915:
----------------------------------------------------------
Commit 0e8a9077b10ff478e0861a1a266a38927a5ae544 in impala's branch
refs/heads/master from stiga-huang
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0e8a9077b ]
IMPALA-12915: Use libgtest.so when built with shared libs
When building Impala with shared libraries, we currently link against
the static library of GTest in both libkudu_test_util.so and
unifiedbetests. libkudu_test_util.so is also linked by unifiedbetests
dynamically. When unifiedbetests exits, both binaries try to delete the
global variables of GTest, which leads to a double-free memory issue.
To fix the double-free memory issue, instead of statically linking
libgtest.a, all binaries should dynamically link libgtest.so. So when
the process exits, only libgtest.so will delete its global variables.
Tests
- Verified the issue resolved locally
Change-Id: I27d21217db219f52b072a4e5cfa1caaace35d1a2
Reviewed-on: http://gerrit.cloudera.org:8080/21163
Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
> double-free memory issue in unifiedbetests if it's built in dynamically
> linked mode
> -----------------------------------------------------------------------------------
>
> Key: IMPALA-12915
> URL: https://issues.apache.org/jira/browse/IMPALA-12915
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Reporter: Quanlong Huang
> Assignee: Quanlong Huang
> Priority: Major
>
> I have a pretty simple change that just adds one new metric in
> common/thrift/metrics.json:
> [https://github.com/stiga-huang/impala/commit/3972aed68428b3f307e823f49a2700c1a98aa3b3]
> It breaks the clang-tidy job in the step of running "unifiedbetests
> --gtest_list_tests".
> The job builds Impala using the -so option which means dynamically link
> executables (default is static). I'm able to reproduce the issue locally.
> Using gdb, I can see the process crashed by a memory issue:
> {noformat}
> (gdb) bt
> #0 0x0000000002ff8d6b in
> tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*,
> unsigned int, int) ()
> #1 0x0000000002ff93c5 in
> tcmalloc::ThreadCache::ListTooLong(tcmalloc::ThreadCache::FreeList*, unsigned
> int) ()
> #2 0x00007fffef06db1f in
> std::_Function_base::_Base_manager<boost::algorithm::detail::is_any_ofF<char>
> >::_M_manager(std::_Any_data&, std::_Any_data const&,
> std::_Manager_operation) ()
> from
> /home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
> #3 0x00007fffef06d7e4 in std::_Function_handler<bool (char),
> boost::algorithm::detail::is_any_ofF<char> >::_M_manager(std::_Any_data&,
> std::_Any_data const&, std::_Manager_operation) ()
> from
> /home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
> #4 0x000000000109fc07 in std::_Function_base::~_Function_base() ()
> #5 0x00007fffed7a8605 in __cxa_finalize () from
> /lib/x86_64-linux-gnu/libc.so.6
> #6 0x00007fffeef995a3 in __do_global_dtors_aux () from
> /home/quanlong/workspace/Impala/be/build/release/kudu_util/libkudu_util.so
> #7 0x00007fffffff03c0 in ?? (){noformat}
> [~boroknagyz] found that upgrading gtest to 1.14 resolves the issue. The
> current gtest version is 1.6. I tried building with ASAN and also build the
> underlying gtest-1.6 with ASAN. The run then shows the memory issue:
> {noformat}
> ==1778==ERROR: AddressSanitizer: heap-use-after-free on address
> 0x603000098530 at pc 0x00000123a9a9 bp 0x7ffeeaf47d60 sp 0x7ffeeaf47d58
> READ of size 8 at 0x603000098530 thread T0
> #0 0x123a9a8 in testing::internal::String::~String()
> /home/quanlong/workspace/Impala/toolchain/toolchain-packages-gcc10.4.0/gtest-1.6.0/include/gtest/internal/gtest-string.h:218:24
> #1 0x530fc2f in void
> std::_Destroy_aux<false>::__destroy<testing::internal::String*>(testing::internal::String*,
> testing::internal::String*)
> /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_construct.h:108:6
> #2 0x52fdcdd in std::vector<testing::internal::String,
> std::allocator<testing::internal::String> >::~vector()
> /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:677:2
> #3 0x7f4ca763a604 in __cxa_finalize
> /build/glibc-CVJwZb/glibc-2.27/stdlib/cxa_finalize.c:83
> #4 0x7f4ca93a8cb2 in __do_global_dtors_aux
> (/home/quanlong/workspace/Impala/be/src/kudu/util/libkudu_test_util.so+0x6ecb2)
> 0x603000098530 is located 0 bytes inside of 32-byte region
> [0x603000098530,0x603000098550)
> freed by thread T0 here:
> #0 0x1233dcf in operator delete(void*)
> (/home/quanlong/workspace/Impala/be/build/debug/service/unifiedbetests+0x1233dcf)
> #1 0x5310da3 in std::_Vector_base<testing::internal::String,
> std::allocator<testing::internal::String> >::~_Vector_base()
> /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/stl_vector.h:332:2
> #2 0x7f4ca763a030 in __run_exit_handlers
> /build/glibc-CVJwZb/glibc-2.27/stdlib/exit.c:108
> previously allocated by thread T0 here:
> #0 0x123305f in operator new(unsigned long)
> (/home/quanlong/workspace/Impala/be/build/debug/service/unifiedbetests+0x123305f)
> #1 0x530fdd0 in std::vector<testing::internal::String,
> std::allocator<testing::internal::String>
> >::_M_realloc_insert(__gnu_cxx::__normal_iterator<testing::internal::String*,
> std::vector<testing::internal::String,
> std::allocator<testing::internal::String> > >, testing::internal::String
> const&)
> /usr/lib/gcc/x86_64-linux-gnu/9/../../../../include/c++/9/bits/vector.tcc:440:33
> #2 0x52fc053 in void testing::internal::InitGoogleTestImpl<char>(int*,
> char**)
> /media/quanlong/hdd-backup/native-toolchain/source/gtest/gtest-1.6.0/src/gtest.cc:4868:13
> #3 0x1236435 in main
> /home/quanlong/workspace/Impala/be/src/service/unified-betest-main.cc:40:3
> #4 0x7f4ca7618c86 in __libc_start_main
> /build/glibc-CVJwZb/glibc-2.27/csu/../csu/libc-start.c:310{noformat}
> Note that I have to remove the definition of "THttpTransport::CRLF_LEN" in
> be/src/transport/THttpTransport.cpp to avoid an ODR violation issue.
> libkudu_test_util.so wants to delete a vector that is already deleted by
> gtest.
> https://github.com/google/googletest/blob/release-1.6.0/include/gtest/internal/gtest-port.h#L1057
>
> In the static build, we are fine since there are no libkudu_test_util.so or
> libkudu_util.so.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
