Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16426 )
Change subject: IMPALA-9740, IMPALA-9403: Fix remaining custom cluster TSAN errors ...................................................................... IMPALA-9740, IMPALA-9403: Fix remaining custom cluster TSAN errors This patch fixes the remaining TSAN errors reported while running custom cluster tests. After this patch, TSAN can be enabled for custom cluster tests (currently it is only run for be tests). Adds a data race suppression for HdfsColumnarScanner::ProcessScratchBatchCodegenOrInterpret, which usually calls a codegen function. TSAN currently does not support codegen functions, so this warning needs to be suppressed. The call stack of this warning is: #0 kudu::BlockBloomFilter::Find(unsigned int) const kudu/util/block_bloom_filter.cc:257:7 #1 <null> <null> (0x7f19af1c74cd) #2 impala::HdfsColumnarScanner::ProcessScratchBatchCodegenOrInterpret(impala::RowBatch*) exec/hdfs-columnar-scanner.cc:106:10 #3 impala::HdfsColumnarScanner::TransferScratchTuples(impala::RowBatch*) exec/hdfs-columnar-scanner.cc:66:34 Fixes a data race in DmlExecState::FinalizeHdfsInsert where a local HdfsFsCache::HdfsFsMap is unsafely passed between threads of a HdfsOperationSet. HdfsOperationSet instances are run in a HdfsOpThreadPool and each operation is run in one of the threads from the pool. Each operation uses HdfsFsCache::GetConnection to get a hdfsFs instance. GetConnection can take in a 'local_cache' of hdfsFs instances before using the global map. The race condition is that the same local cache is used for all operations in HdfsOperationSet. Testing: * Re-ran TSAN tests and confirmed the data races have disappeared Change-Id: If1658a9b56d220e2cfd1f8b958604edcdf7757f4 Reviewed-on: http://gerrit.cloudera.org:8080/16426 Reviewed-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenk...@cloudera.com> --- M be/src/util/hdfs-bulk-ops.cc M be/src/util/hdfs-bulk-ops.h M bin/tsan-suppressions.txt 3 files changed, 18 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16426 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: If1658a9b56d220e2cfd1f8b958604edcdf7757f4 Gerrit-Change-Number: 16426 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Joe McDonnell <joemcdonn...@cloudera.com> Gerrit-Reviewer: Sahil Takiar <stak...@cloudera.com> Gerrit-Reviewer: Tim Armstrong <tarmstr...@cloudera.com>