[ https://issues.apache.org/jira/browse/IMPALA-9568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17072130#comment-17072130 ]
ASF subversion and git services commented on IMPALA-9568: --------------------------------------------------------- Commit 0038487267ccba77e6cb17a582039d6b35ff947c in impala's branch refs/heads/master from Sahil Takiar [ https://gitbox.apache.org/repos/asf?p=impala.git;h=0038487 ] IMPALA-9568: Template tuples are initialized multiple times Template tuples are initialized multiple times, concurrently, which leads to a data race reported by TSAN. The template tuples should just be initialized once. This patch removes the template tuple initialization done in the HDFS scanners and just copies them from the HdfsScanNodeBase. After this patch, data-load is now TSAN clean. I removed the TSAN flag 'halt_on_error=0' so that TSAN now crashes the process if a TSAN bug is discovered. This will allow us to setup a Jenkins job that runs dataload + be/ tests, and will crash if any new TSAN bugs are discovered. Testing: * Ran exhaustive tests Change-Id: I3bd3554b2b919b117a0c9ae86dfc0a75ae4129d2 Reviewed-on: http://gerrit.cloudera.org:8080/15604 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> > Template tuples are initialized multiple times > ---------------------------------------------- > > Key: IMPALA-9568 > URL: https://issues.apache.org/jira/browse/IMPALA-9568 > Project: IMPALA > Issue Type: Sub-task > Reporter: Sahil Takiar > Assignee: Sahil Takiar > Priority: Major > > TSAN is reporting the following data race with template tuple initialization: > {code:java} > WARNING: ThreadSanitizer: data race (pid=8124) > Write of size 4 at 0x7b30001692f4 by thread T355: > #0 impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&, > impala::TupleRow const*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:279:23 > (impalad+0x2553330) > #1 impala::ScalarExprEvaluator::GetValue(impala::TupleRow const*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:253:10 > (impalad+0x255357d) > #2 > impala::HdfsScanNodeBase::InitTemplateTuple(std::vector<impala::ScalarExprEvaluator*, > std::allocator<impala::ScalarExprEvaluator*> > const&, impala::MemPool*, > impala::RuntimeState*) const > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:900:27 > (impalad+0x237c74a) > #3 impala::HdfsScanner::Open(impala::ScannerContext*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scanner.cc:110:33 > (impalad+0x23a6437) > #4 impala::HdfsTextScanner::Open(impala::ScannerContext*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-text-scanner.cc:780:3 > (impalad+0x23e6121) > #5 > impala::HdfsScanNodeBase::CreateAndOpenScannerHelper(impala::HdfsPartitionDescriptor*, > impala::ScannerContext*, boost::scoped_ptr<impala::HdfsScanner>*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:887:3 > (impalad+0x237fc76) > #6 impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, > std::allocator<impala::FilterContext> > const&, impala::MemPool*, > impala::io::ScanRange*, long*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:479:19 > (impalad+0x24daeba) > #7 impala::HdfsScanNode::ScannerThread(bool, long) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:417:7 > (impalad+0x24da84e) > #8 > impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0::operator()() > const > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:338:13 > (impalad+0x24dbd06) > #9 > boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0, > void>::invoke(boost::detail::function::function_buffer&) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 > (impalad+0x24dbb19) > #10 boost::function0<void>::operator()() const > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 > (impalad+0x1d0af41) > #11 impala::Thread::SuperviseThread(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 > (impalad+0x22c3c66) > #12 void boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, > std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 > (impalad+0x22cbe2c) > #13 boost::_bi::bind_t<void, void (*)(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 > (impalad+0x22cbd43) > #14 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::string const&, std::string const&, boost::function<void ()>, > impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 > (impalad+0x22cba30) > #15 thread_proxy <null> (impalad+0x2fb6509) > Previous write of size 4 at 0x7b30001692f4 by thread T337: > #0 impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&, > impala::TupleRow const*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:279:23 > (impalad+0x2553330) > #1 impala::ScalarExprEvaluator::GetValue(impala::TupleRow const*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:253:10 > (impalad+0x255357d) > #2 > impala::HdfsScanNodeBase::InitTemplateTuple(std::vector<impala::ScalarExprEvaluator*, > std::allocator<impala::ScalarExprEvaluator*> > const&, impala::MemPool*, > impala::RuntimeState*) const > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:900:27 > (impalad+0x237c74a) > #3 impala::HdfsScanner::Open(impala::ScannerContext*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scanner.cc:110:33 > (impalad+0x23a6437) > #4 impala::HdfsTextScanner::Open(impala::ScannerContext*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-text-scanner.cc:780:3 > (impalad+0x23e6121) > #5 > impala::HdfsScanNodeBase::CreateAndOpenScannerHelper(impala::HdfsPartitionDescriptor*, > impala::ScannerContext*, boost::scoped_ptr<impala::HdfsScanner>*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:887:3 > (impalad+0x237fc76) > #6 impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, > std::allocator<impala::FilterContext> > const&, impala::MemPool*, > impala::io::ScanRange*, long*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:479:19 > (impalad+0x24daeba) > #7 impala::HdfsScanNode::ScannerThread(bool, long) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:417:7 > (impalad+0x24da84e) > #8 > impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0::operator()() > const > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:338:13 > (impalad+0x24dbd06) > #9 > boost::detail::function::void_function_obj_invoker0<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0, > void>::invoke(boost::detail::function::function_buffer&) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 > (impalad+0x24dbb19) > #10 boost::function0<void>::operator()() const > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 > (impalad+0x1d0af41) > #11 impala::Thread::SuperviseThread(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 > (impalad+0x22c3c66) > #12 void boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, > std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 > (impalad+0x22cbe2c) > #13 boost::_bi::bind_t<void, void (*)(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 > (impalad+0x22cbd43) > #14 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::string const&, std::string const&, boost::function<void ()>, > impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 > (impalad+0x22cba30) > #15 thread_proxy <null> (impalad+0x2fb6509) > Location is heap block of size 192 at 0x7b30001692c0 allocated by thread > T349: > #0 operator new(unsigned long) > /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_new_delete.cc:44 > (impalad+0x192f603) > #1 impala::ScalarExprEvaluator::Create(impala::ScalarExpr const&, > impala::RuntimeState*, impala::ObjectPool*, impala::MemPool*, > impala::MemPool*, impala::ScalarExprEvaluator**) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:79:21 > (impalad+0x25516b9) > #2 impala::ScalarExprEvaluator::Create(std::vector<impala::ScalarExpr*, > std::allocator<impala::ScalarExpr*> > const&, impala::RuntimeState*, > impala::ObjectPool*, impala::MemPool*, impala::MemPool*, > std::vector<impala::ScalarExprEvaluator*, > std::allocator<impala::ScalarExprEvaluator*> >*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:101:21 > (impalad+0x2551f30) > #3 impala::DescriptorTbl::CreatePartKeyExprs(impala::HdfsTableDescriptor > const&, impala::ObjectPool*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/descriptors.cc:516:5 > (impalad+0x1f2323e) > #4 > impala::DescriptorTbl::CreateTblDescriptorInternal(impala::TTableDescriptor > const&, impala::ObjectPool*, impala::TableDescriptor**) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/descriptors.cc:557:7 > (impalad+0x1f238c2) > #5 impala::DescriptorTbl::CreateInternal(impala::ObjectPool*, > impala::TDescriptorTable const&, impala::DescriptorTbl**) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/descriptors.cc:588:5 > (impalad+0x1f23bce) > #6 impala::DescriptorTbl::Create(impala::ObjectPool*, > impala::TDescriptorTableSerialized const&, impala::DescriptorTbl**) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/descriptors.cc:579:10 > (impalad+0x1f23aa5) > #7 impala::QueryState::StartFInstances() > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:570:7 > (impalad+0x1f4af0e) > #8 impala::QueryExecMgr::ExecuteQueryHelper(impala::QueryState*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-exec-mgr.cc:141:7 > (impalad+0x1f96bc9) > #9 boost::_mfi::mf1<void, impala::QueryExecMgr, > impala::QueryState*>::operator()(impala::QueryExecMgr*, impala::QueryState*) > const > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:165:29 > (impalad+0x1f97ebd) > #10 void boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, > boost::_bi::value<impala::QueryState*> >::operator()<boost::_mfi::mf1<void, > impala::QueryExecMgr, impala::QueryState*>, > boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf1<void, > impala::QueryExecMgr, impala::QueryState*>&, boost::_bi::list0&, int) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:319:9 > (impalad+0x1f97dfd) > #11 boost::_bi::bind_t<void, boost::_mfi::mf1<void, impala::QueryExecMgr, > impala::QueryState*>, > boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, > boost::_bi::value<impala::QueryState*> > >::operator()() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 > (impalad+0x1f97d63) > #12 > boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, > boost::_mfi::mf1<void, impala::QueryExecMgr, impala::QueryState*>, > boost::_bi::list2<boost::_bi::value<impala::QueryExecMgr*>, > boost::_bi::value<impala::QueryState*> > >, > void>::invoke(boost::detail::function::function_buffer&) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 > (impalad+0x1f97b01) > #13 boost::function0<void>::operator()() const > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 > (impalad+0x1d0af41) > #14 impala::Thread::SuperviseThread(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 > (impalad+0x22c3c66) > #15 void boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, > std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 > (impalad+0x22cbe2c) > #16 boost::_bi::bind_t<void, void (*)(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 > (impalad+0x22cbd43) > #17 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::string const&, std::string const&, boost::function<void ()>, > impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 > (impalad+0x22cba30) > #18 thread_proxy <null> (impalad+0x2fb6509) > Thread T355 (tid=3630, running) created by thread T364 at: > #0 pthread_create > /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 > (impalad+0x18c6feb) > #1 boost::thread::start_thread_noexcept() <null> (impalad+0x2fb5899) > #2 boost::thread::thread<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, > boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, > (impala::PromiseMode)0>*>(void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, > boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, > (impala::PromiseMode)0>*) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420:13 > (impalad+0x22c643b) > #3 impala::Thread::StartThread(std::string const&, std::string const&, > boost::function<void ()> const&, std::unique_ptr<impala::Thread, > std::default_delete<impala::Thread> >*, bool) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:317:13 > (impalad+0x22c3605) > #4 impala::Status > impala::Thread::Create<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0>(std::string > const&, std::string const&, > impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0 > const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> > >*, bool) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.h:74:12 > (impalad+0x24da3ad) > #5 > impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:341:14 > (impalad+0x24d99e9) > #6 > impala::HdfsScanNode::AddDiskIoRanges(std::vector<impala::io::ScanRange*, > std::allocator<impala::io::ScanRange*> > const&, impala::EnqueueLocation) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:213:24 > (impalad+0x24d9ffe) > #7 impala::HdfsScanNodeBase::AddDiskIoRanges(impala::HdfsFileDesc const*, > impala::EnqueueLocation) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.h:356:12 > (impalad+0x23e681b) > #8 impala::HdfsTextScanner::IssueInitialRanges(impala::HdfsScanNodeBase*, > std::vector<impala::HdfsFileDesc*, std::allocator<impala::HdfsFileDesc*> > > const&) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-text-scanner.cc:86:9 > (impalad+0x23df95b) > #9 > impala::HdfsScanNodeBase::IssueInitialScanRanges(impala::RuntimeState*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:639:9 > (impalad+0x237dce9) > #10 impala::HdfsScanNode::GetNext(impala::RuntimeState*, > impala::RowBatch*, bool*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:91:21 > (impalad+0x24d8b5c) > #11 impala::FragmentInstanceState::ExecInternal() > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:383:7 > (impalad+0x1f3b9e5) > #12 impala::FragmentInstanceState::Exec() > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:98:14 > (impalad+0x1f398a6) > #13 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:697:24 > (impalad+0x1f4c113) > #14 impala::QueryState::StartFInstances()::$_7::operator()() const > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:606:35 > (impalad+0x1f4e2d2) > #15 > boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_7, > void>::invoke(boost::detail::function::function_buffer&) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 > (impalad+0x1f4e0f9) > #16 boost::function0<void>::operator()() const > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 > (impalad+0x1d0af41) > #17 impala::Thread::SuperviseThread(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 > (impalad+0x22c3c66) > #18 void boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, > std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 > (impalad+0x22cbe2c) > #19 boost::_bi::bind_t<void, void (*)(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 > (impalad+0x22cbd43) > #20 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::string const&, std::string const&, boost::function<void ()>, > impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 > (impalad+0x22cba30) > #21 thread_proxy <null> (impalad+0x2fb6509) > Thread T337 (tid=3629, running) created by thread T364 at: > #0 pthread_create > /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 > (impalad+0x18c6feb) > #1 boost::thread::start_thread_noexcept() <null> (impalad+0x2fb5899) > #2 boost::thread::thread<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, > boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, > (impala::PromiseMode)0>*>(void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, > boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, > (impala::PromiseMode)0>*) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420:13 > (impalad+0x22c643b) > #3 impala::Thread::StartThread(std::string const&, std::string const&, > boost::function<void ()> const&, std::unique_ptr<impala::Thread, > std::default_delete<impala::Thread> >*, bool) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:317:13 > (impalad+0x22c3605) > #4 impala::Status > impala::Thread::Create<impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0>(std::string > const&, std::string const&, > impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*)::$_0 > const&, std::unique_ptr<impala::Thread, std::default_delete<impala::Thread> > >*, bool) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.h:74:12 > (impalad+0x24da3ad) > #5 > impala::HdfsScanNode::ThreadTokenAvailableCb(impala::ThreadResourcePool*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:341:14 > (impalad+0x24d99e9) > #6 > impala::HdfsScanNode::AddDiskIoRanges(std::vector<impala::io::ScanRange*, > std::allocator<impala::io::ScanRange*> > const&, impala::EnqueueLocation) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:213:24 > (impalad+0x24d9ffe) > #7 impala::HdfsScanNodeBase::AddDiskIoRanges(impala::HdfsFileDesc const*, > impala::EnqueueLocation) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.h:356:12 > (impalad+0x23e681b) > #8 impala::HdfsTextScanner::IssueInitialRanges(impala::HdfsScanNodeBase*, > std::vector<impala::HdfsFileDesc*, std::allocator<impala::HdfsFileDesc*> > > const&) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-text-scanner.cc:86:9 > (impalad+0x23df95b) > #9 > impala::HdfsScanNodeBase::IssueInitialScanRanges(impala::RuntimeState*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node-base.cc:639:9 > (impalad+0x237dce9) > #10 impala::HdfsScanNode::GetNext(impala::RuntimeState*, > impala::RowBatch*, bool*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exec/hdfs-scan-node.cc:91:21 > (impalad+0x24d8b5c) > #11 impala::FragmentInstanceState::ExecInternal() > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:383:7 > (impalad+0x1f3b9e5) > #12 impala::FragmentInstanceState::Exec() > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/fragment-instance-state.cc:98:14 > (impalad+0x1f398a6) > #13 impala::QueryState::ExecFInstance(impala::FragmentInstanceState*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:697:24 > (impalad+0x1f4c113) > #14 impala::QueryState::StartFInstances()::$_7::operator()() const > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:606:35 > (impalad+0x1f4e2d2) > #15 > boost::detail::function::void_function_obj_invoker0<impala::QueryState::StartFInstances()::$_7, > void>::invoke(boost::detail::function::function_buffer&) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 > (impalad+0x1f4e0f9) > #16 boost::function0<void>::operator()() const > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 > (impalad+0x1d0af41) > #17 impala::Thread::SuperviseThread(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 > (impalad+0x22c3c66) > #18 void boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, > std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 > (impalad+0x22cbe2c) > #19 boost::_bi::bind_t<void, void (*)(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 > (impalad+0x22cbd43) > #20 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::string const&, std::string const&, boost::function<void ()>, > impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 > (impalad+0x22cba30) > #21 thread_proxy <null> (impalad+0x2fb6509) > Thread T349 (tid=3561, running) created by thread T97 at: > #0 pthread_create > /mnt/source/llvm/llvm-5.0.1.src-p2/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors.cc:889 > (impalad+0x18c6feb) > #1 boost::thread::start_thread_noexcept() <null> (impalad+0x2fb5899) > #2 boost::thread::thread<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, > boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, > (impala::PromiseMode)0>*>(void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), std::string, std::string, > boost::function<void ()>, impala::ThreadDebugInfo*, impala::Promise<long, > (impala::PromiseMode)0>*) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:420:13 > (impalad+0x22c643b) > #3 impala::Thread::StartThread(std::string const&, std::string const&, > boost::function<void ()> const&, std::unique_ptr<impala::Thread, > std::default_delete<impala::Thread> >*, bool) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:317:13 > (impalad+0x22c3605) > #4 impala::Status impala::Thread::Create<void > (impala::QueryExecMgr::*)(impala::QueryState*), impala::QueryExecMgr*, > impala::QueryState*>(std::string const&, std::string const&, void > (impala::QueryExecMgr::* const&)(impala::QueryState*), impala::QueryExecMgr* > const&, impala::QueryState* const&, std::unique_ptr<impala::Thread, > std::default_delete<impala::Thread> >*, bool) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.h:89:12 > (impalad+0x1f971a7) > #5 impala::QueryExecMgr::StartQuery(impala::ExecQueryFInstancesRequestPB > const*, impala::TQueryCtx const&, impala::TExecPlanFragmentInfo const&) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-exec-mgr.cc:69:12 > (impalad+0x1f96092) > #6 > impala::ControlService::ExecQueryFInstances(impala::ExecQueryFInstancesRequestPB > const*, impala::ExecQueryFInstancesResponsePB*, kudu::rpc::RpcContext*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/control-service.cc:156:66 > (impalad+0x2065dee) > #7 > impala::ControlServiceIf::ControlServiceIf(scoped_refptr<kudu::MetricEntity> > const&, scoped_refptr<kudu::rpc::ResultTracker> > const&)::$_1::operator()(google::protobuf::Message const*, > google::protobuf::Message*, kudu::rpc::RpcContext*) const > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/generated-sources/gen-cpp/control_service.service.cc:70:13 > (impalad+0x2113024) > #8 std::_Function_handler<void (google::protobuf::Message const*, > google::protobuf::Message*, kudu::rpc::RpcContext*), > impala::ControlServiceIf::ControlServiceIf(scoped_refptr<kudu::MetricEntity> > const&, scoped_refptr<kudu::rpc::ResultTracker> > const&)::$_1>::_M_invoke(std::_Any_data const&, google::protobuf::Message > const*, google::protobuf::Message*, kudu::rpc::RpcContext*) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2039:2 > (impalad+0x2112e0e) > #9 std::function<void (google::protobuf::Message const*, > google::protobuf::Message*, > kudu::rpc::RpcContext*)>::operator()(google::protobuf::Message const*, > google::protobuf::Message*, kudu::rpc::RpcContext*) const > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/gcc-4.9.2/lib/gcc/x86_64-unknown-linux-gnu/4.9.2/../../../../include/c++/4.9.2/functional:2439:14 > (impalad+0x27afa68) > #10 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3 > (impalad+0x27af3ce) > #11 impala::ImpalaServicePool::RunThread() > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15 > (impalad+0x1e3260f) > #12 boost::_mfi::mf0<void, > impala::ImpalaServicePool>::operator()(impala::ImpalaServicePool*) const > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:49:29 > (impalad+0x1e38b76) > #13 void boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >::operator()<boost::_mfi::mf0<void, impala::ImpalaServicePool>, > boost::_bi::list0>(boost::_bi::type<void>, boost::_mfi::mf0<void, > impala::ImpalaServicePool>&, boost::_bi::list0&, int) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:259:9 > (impalad+0x1e38aca) > #14 boost::_bi::bind_t<void, boost::_mfi::mf0<void, > impala::ImpalaServicePool>, > boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > > >::operator()() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 > (impalad+0x1e38a53) > #15 > boost::detail::function::void_function_obj_invoker0<boost::_bi::bind_t<void, > boost::_mfi::mf0<void, impala::ImpalaServicePool>, > boost::_bi::list1<boost::_bi::value<impala::ImpalaServicePool*> > >, > void>::invoke(boost::detail::function::function_buffer&) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 > (impalad+0x1e38849) > #16 boost::function0<void>::operator()() const > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 > (impalad+0x1d0af41) > #17 impala::Thread::SuperviseThread(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*) > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 > (impalad+0x22c3c66) > #18 void boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > >::operator()<void (*)(std::string const&, std::string const&, > boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list0>(boost::_bi::type<void>, void (*&)(std::string const&, > std::string const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), boost::_bi::list0&, int) > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:531:9 > (impalad+0x22cbe2c) > #19 boost::_bi::bind_t<void, void (*)(std::string const&, std::string > const&, boost::function<void ()>, impala::ThreadDebugInfo const*, > impala::Promise<long, (impala::PromiseMode)0>*), > boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > >::operator()() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 > (impalad+0x22cbd43) > #20 boost::detail::thread_data<boost::_bi::bind_t<void, void > (*)(std::string const&, std::string const&, boost::function<void ()>, > impala::ThreadDebugInfo const*, impala::Promise<long, > (impala::PromiseMode)0>*), boost::_bi::list5<boost::_bi::value<std::string>, > boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()> >, > boost::_bi::value<impala::ThreadDebugInfo*>, > boost::_bi::value<impala::Promise<long, (impala::PromiseMode)0>*> > > > >::run() > /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/boost-1.61.0-p2/include/boost/thread/detail/thread.hpp:116:17 > (impalad+0x22cba30) > #21 thread_proxy <null> (impalad+0x2fb6509)SUMMARY: ThreadSanitizer: data > race > /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/exprs/scalar-expr-evaluator.cc:279:23 > in impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&, > impala::TupleRow const*) {code} > The odd thing is that it looks like template tuples are initialized multiple > times, once in {{HdfsScanNodeBase::Open}} --> > {{HdfsScanNodeBase::InitTemplateTuple}} and again in {{HdfsScanner::Open}}. I > think a simple fix here would be for the scanners to not initialize the > template tuples and just use the one initialized in {{HdfsScanNodeBase}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org