[jira] [Created] (IMPALA-10158) test_iceberg_query and test_iceberg_profile fail after IMPALA-9741
Fang-Yu Rao created IMPALA-10158: Summary: test_iceberg_query and test_iceberg_profile fail after IMPALA-9741 Key: IMPALA-10158 URL: https://issues.apache.org/jira/browse/IMPALA-10158 Project: IMPALA Issue Type: Bug Reporter: Fang-Yu Rao Assignee: Fang-Yu Rao We found that the tests of {{test_iceberg_query}} and {{test_iceberg_profile}} fail after the patch for IMPALA-9741 has been merged. After some investigation with the help of [~boroknagyz] and [~csringhofer], we found that it is a timezone-related issue and that we should add {{SET TIMEZONE=UTC}} in the corresponding test files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IMPALA-10157) IllegalStateException when using grouping() or grouping_id() with no GROUP BY clause
Tim Armstrong created IMPALA-10157: -- Summary: IllegalStateException when using grouping() or grouping_id() with no GROUP BY clause Key: IMPALA-10157 URL: https://issues.apache.org/jira/browse/IMPALA-10157 Project: IMPALA Issue Type: Bug Components: Frontend Reporter: Tim Armstrong Assignee: Tim Armstrong {noformat} [localhost.EXAMPLE.COM:21000] default> select int_col, tinyint_col, grouping(int_col), grouping(tinyint_col), grouping_id(int_col, tinyint_col) from functional.alltypes; Query: select int_col, tinyint_col, grouping(int_col), grouping(tinyint_col), grouping_id(int_col, tinyint_col) from functional.alltypes Query submitted at: 2020-09-08 10:25:12 (Coordinator: http://tarmstrong-box2:25000) ERROR: IllegalStateException: null {noformat} {noformat} I0908 10:25:12.834306 19139 Frontend.java:1507] a845d2d57577cc5c:b905608c] Analyzing query: select int_col, tinyint_col, grouping(int_col), grouping(tinyint_col), grouping_id(int_col, tinyint_col) from functional.alltypes db: default I0908 10:25:12.839713 19139 jni-util.cc:288] a845d2d57577cc5c:b905608c] java.lang.IllegalStateException at com.google.common.base.Preconditions.checkState(Preconditions.java:492) at org.apache.impala.analysis.MultiAggregateInfo.analyze(MultiAggregateInfo.java:274) at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.buildAggregateExprs(SelectStmt.java:845) at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.analyze(SelectStmt.java:269) at org.apache.impala.analysis.SelectStmt$SelectAnalyzer.access$100(SelectStmt.java:237) at org.apache.impala.analysis.SelectStmt.analyze(SelectStmt.java:230) at org.apache.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:472) at org.apache.impala.analysis.AnalysisContext.analyzeAndAuthorize(AnalysisContext.java:436) at org.apache.impala.service.Frontend.doCreateExecRequest(Frontend.java:1547) at org.apache.impala.service.Frontend.getTExecRequest(Frontend.java:1514) at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1484) at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:162) I0908 10:25:12.839778 19139 status.cc:129] a845d2d57577cc5c:b905608c] IllegalStateException: null @ 0x1ce443b impala::Status::Status() @ 0x2644a01 impala::JniUtil::GetJniExceptionMsg() @ 0x2446c96 impala::JniCall::Call<>() @ 0x2443e49 impala::JniUtil::CallJniMethod<>() @ 0x2442150 impala::Frontend::GetExecRequest() @ 0x2cc4d47 impala::QueryDriver::RunFrontendPlanner() @ 0x246f844 impala::ImpalaServer::ExecuteInternal() @ 0x246f202 impala::ImpalaServer::Execute() @ 0x25098c6 impala::ImpalaServer::query() @ 0x2afb7df beeswax::BeeswaxServiceProcessor::process_query() @ 0x2afb52d beeswax::BeeswaxServiceProcessor::dispatchCall() @ 0x2ac8976 impala::ImpalaServiceProcessor::dispatchCall() @ 0x1c92db3 apache::thrift::TDispatchProcessor::process() @ 0x2195f36 apache::thrift::server::TAcceptQueueServer::Task::run() @ 0x218b436 impala::ThriftThread::RunRunnable() @ 0x218ca72 boost::_mfi::mf2<>::operator()() @ 0x218c906 boost::_bi::list3<>::operator()<>() @ 0x218c64c boost::_bi::bind_t<>::operator()() @ 0x218c55e boost::detail::function::void_function_obj_invoker0<>::invoke() @ 0x2107043 boost::function0<>::operator()() @ 0x26d5ea5 impala::Thread::SuperviseThread() @ 0x26dde42 boost::_bi::list5<>::operator()<>() @ 0x26ddd66 boost::_bi::bind_t<>::operator()() @ 0x26ddd27 boost::detail::thread_data<>::run() @ 0x3ec7271 thread_proxy @ 0x7fa5b50b76da start_thread @ 0x7fa5b1aaea3e clone {noformat} -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IMPALA-10156) test_unmatched_schema flaky with wrong results
Thomas Tauber-Marshall created IMPALA-10156: --- Summary: test_unmatched_schema flaky with wrong results Key: IMPALA-10156 URL: https://issues.apache.org/jira/browse/IMPALA-10156 Project: IMPALA Issue Type: Bug Affects Versions: Impala 4.0 Reporter: Thomas Tauber-Marshall {noformat} Error Message query_test/test_scanners.py:244: in test_unmatched_schema self.run_test_case('QueryTest/test-unmatched-schema', vector) common/impala_test_suite.py:693: in run_test_case self.__verify_results_and_errors(vector, test_section, result, use_db) common/impala_test_suite.py:529: in __verify_results_and_errors replace_filenames_with_placeholder) common/test_result_verifier.py:456: in verify_raw_results VERIFIER_MAP[verifier](expected, actual) common/test_result_verifier.py:278: in verify_query_result_is_equal assert expected_results == actual_results E assert Comparing QueryTestResults (expected vs actual): E 1001,'Name1',94611,5000 == 1001,'Name1',94611,5000 E 1002,'Name2',94611,5000 != 1001,'Name1',94611,5000 E 1003,'Name3',94611,5000 != 1002,'Name2',94611,5000 E 1004,'Name4',94611,5000 != 1002,'Name2',94611,5000 E 1005,'Name5',94611,5000 != 1003,'Name3',94611,5000 E 1006,'Name16',94612,15000 != 1003,'Name3',94611,5000 E 1006,'Name16',94612,5000 != 1004,'Name4',94611,5000 E 1006,'Name16',94616,15000 != 1004,'Name4',94611,5000 E 1006,'Name16',94616,5000 != 1005,'Name5',94611,5000 E 1006,'Name6',94616,15000 != 1005,'Name5',94611,5000 E 1006,'Name6',94616,5000 != 1006,'Name16',94612,15000 E 1106,'Name16',94612,15000 != 1006,'Name16',94612,15000 E 1106,'Name16',94612,5000 != 1006,'Name16',94612,5000 E 1106,'Name16',94616,15000 != 1006,'Name16',94612,5000 E 1106,'Name16',94616,5000 != 1006,'Name16',94616,15000 E 1106,'Name6',94612,15000 != 1006,'Name16',94616,15000 E 1106,'Name6',94612,5000 != 1006,'Name16',94616,5000 E 1106,'Name6',94616,15000 != 1006,'Name16',94616,5000 E 1106,'Name6',94616,5000 != 1006,'Name6',94616,15000 E None != 1006,'Name6',94616,15000 E None != 1006,'Name6',94616,5000 E None != 1006,'Name6',94616,5000 E None != 1106,'Name16',94612,15000 E None != 1106,'Name16',94612,15000 E None != 1106,'Name16',94612,5000 E None != 1106,'Name16',94612,5000 E None != 1106,'Name16',94616,15000 E None != 1106,'Name16',94616,15000 E None != 1106,'Name16',94616,5000 E None != 1106,'Name16',94616,5000 E None != 1106,'Name6',94612,15000 E None != 1106,'Name6',94612,15000 E None != 1106,'Name6',94612,5000 E None != 1106,'Name6',94612,5000 E None != 1106,'Name6',94616,15000 E None != 1106,'Name6',94616,15000 E None != 1106,'Name6',94616,5000 E None != 1106,'Name6',94616,5000 E Number of rows returned (expected vs actual): 19 != 38 Stacktrace query_test/test_scanners.py:244: in test_unmatched_schema self.run_test_case('QueryTest/test-unmatched-schema', vector) common/impala_test_suite.py:693: in run_test_case self.__verify_results_and_errors(vector, test_section, result, use_db) common/impala_test_suite.py:529: in __verify_results_and_errors replace_filenames_with_placeholder) common/test_result_verifier.py:456: in verify_raw_results VERIFIER_MAP[verifier](expected, actual) common/test_result_verifier.py:278: in verify_query_result_is_equal assert expected_results == actual_results E assert Comparing QueryTestResults (expected vs actual): E 1001,'Name1',94611,5000 == 1001,'Name1',94611,5000 E 1002,'Name2',94611,5000 != 1001,'Name1',94611,5000 E 1003,'Name3',94611,5000 != 1002,'Name2',94611,5000 E 1004,'Name4',94611,5000 != 1002,'Name2',94611,5000 E 1005,'Name5',94611,5000 != 1003,'Name3',94611,5000 E 1006,'Name16',94612,15000 != 1003,'Name3',94611,5000 E 1006,'Name16',94612,5000 != 1004,'Name4',94611,5000 E 1006,'Name16',94616,15000 != 1004,'Name4',94611,5000 E 1006,'Name16',94616,5000 != 1005,'Name5',94611,5000 E 1006,'Name6',94616,15000 != 1005,'Name5',94611,5000 E 1006,'Name6',94616,5000 != 1006,'Name16',94612,15000 E 1106,'Name16',94612,15000 != 1006,'Name16',94612,15000 E 1106,'Name16',94612,5000 != 1006,'Name16',94612,5000 E 1106,'Name16',94616,15000 != 1006,'Name16',94612,5000 E 1106,'Name16',94616,5000 != 1006,'Name16',94616,15000 E 1106,'Name6',94612,15000 != 1006,'Name16',94616,15000 E 1106,'Name6',94612,5000 != 1006,'Name16',94616,5000 E 1106,'Name6',94616,15000 != 1006,'Name16',94616,5000 E 1106,'Name6',94616,5000 != 1006,'Name6',94616,15000 E None != 1006,'Name6',94616,15000 E None != 1006,'Name6',94616,5000 E None != 1006,'Name6',94616,5000 E None != 1106,'Name16',94612,15000 E None != 1106,'Name16',94612,15000 E None != 1106,'Name16',94612,5000
[jira] [Created] (IMPALA-10155) Apparent data race in GetTopNQueriesAndUpdatePoolStats
Thomas Tauber-Marshall created IMPALA-10155: --- Summary: Apparent data race in GetTopNQueriesAndUpdatePoolStats Key: IMPALA-10155 URL: https://issues.apache.org/jira/browse/IMPALA-10155 Project: IMPALA Issue Type: Bug Affects Versions: Impala 4.0 Reporter: Thomas Tauber-Marshall Assignee: Qifan Chen >From a tsan build: {noformat} WARNING: ThreadSanitizer: data race (pid=6487) Read of size 1 at 0x7b48001c2c28 by thread T320 (mutexes: write M866233966607478128, write M867078391537609888, write M627824798772259232, write M451058461859238408): #0 impala::MemTracker::GetTopNQueriesAndUpdatePoolStats(std::priority_queue >, std::greater >&, int, impala::TPoolStats&) /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/src/runtime/mem-tracker.cc:453:19 (impalad+0x20b2e51) #1 impala::MemTracker::UpdatePoolStatsForQueries(int, impala::TPoolStats&) /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/src/runtime/mem-tracker.cc:432:3 (impalad+0x20b2cdd) #2 impala::AdmissionController::PoolStats::UpdateMemTrackerStats() /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/src/scheduling/admission-controller.cc:1642:14 (impalad+0x21cb7b0) #3 impala::AdmissionController::AddPoolUpdates(std::vector >*) /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/src/scheduling/admission-controller.cc:1662:18 (impalad+0x21c8af3) #4 impala::AdmissionController::UpdatePoolStats(std::map, std::allocator >, impala::TTopicDelta, std::less, std::allocator > >, std::allocator, std::allocator > const, impala::TTopicDelta> > > const&, std::vector >*) /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/src/scheduling/admission-controller.cc:1355:5 (impalad+0x21c881d) #5 impala::AdmissionController::Init()::$_4::operator()(std::map, std::allocator >, impala::TTopicDelta, std::less, std::allocator > >, std::allocator, std::allocator > const, impala::TTopicDelta> > > const&, std::vector >*) const /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/src/scheduling/admission-controller.cc:643:45 (impalad+0x21cfb81) #6 boost::detail::function::void_function_obj_invoker2, std::allocator >, impala::TTopicDelta, std::less, std::allocator > >, std::allocator, std::allocator > const, impala::TTopicDelta> > > const&, std::vector >*>::invoke(boost::detail::function::function_buffer&, std::map, std::allocator >, impala::TTopicDelta, std::less, std::allocator > >, std::allocator, std::allocator > const, impala::TTopicDelta> > > const&, std::vector >*) /data/jenkins/workspace/impala-asf-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 (impalad+0x21cf9cc) #7 boost::function2, std::allocator >, impala::TTopicDelta, std::less, std::allocator > >, std::allocator, std::allocator > const, impala::TTopicDelta> > > const&, std::vector >*>::operator()(std::map, std::allocator >, impala::TTopicDelta, std::less, std::allocator > >, std::allocator, std::allocator > const, impala::TTopicDelta> > > const&, std::vector >*) const /data/jenkins/workspace/impala-asf-master-core-tsan/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 (impalad+0x23fc400) #8 impala::StatestoreSubscriber::UpdateState(std::map, std::allocator >, impala::TTopicDelta, std::less, std::allocator > >, std::allocator, std::allocator > const, impala::TTopicDelta> > > const&, impala::TUniqueId const&, std::vector >*, bool*) /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/src/statestore/statestore-subscriber.cc:471:7 (impalad+0x23f9339) #9 impala::StatestoreSubscriberThriftIf::UpdateState(impala::TUpdateStateResponse&, impala::TUpdateStateRequest const&) /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/src/statestore/statestore-subscriber.cc:110:18 (impalad+0x23fc65f) #10 impala::StatestoreSubscriberProcessor::process_UpdateState(int, apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, void*) /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/StatestoreSubscriber.cpp:543:13 (impalad+0x29b2fc4) #11 impala::StatestoreSubscriberProcessor::dispatchCall(apache::thrift::protocol::TProtocol*, apache::thrift::protocol::TProtocol*, std::__cxx11::basic_string, std::allocator > const&, int, void*) /data/jenkins/workspace/impala-asf-master-core-tsan/repos/Impala/be/generated-sources/gen-cpp/StatestoreSubscriber.cpp:516:3 (impalad+0x29b2da2) #12 apache::thrift::TDispatchProcessor::process(boost::shared_ptr, boost::shared_ptr, void*) /data/jenkins/workspace/impala-asf-master-core-tsan/Impala-Toolchain/toolchain
[jira] [Created] (IMPALA-10154) Data race on coord_backend_id
Sahil Takiar created IMPALA-10154: - Summary: Data race on coord_backend_id Key: IMPALA-10154 URL: https://issues.apache.org/jira/browse/IMPALA-10154 Project: IMPALA Issue Type: Bug Reporter: Sahil Takiar Assignee: Wenzhe Zhou TSAN is reporting a data race on {{ExecQueryFInstancesRequestPB#coord_backend_id}} {code:java} WARNING: ThreadSanitizer: data race (pid=15392) Write of size 8 at 0x7b74001104a8 by thread T83 (mutexes: write M871582266043729400): #0 impala::ExecQueryFInstancesRequestPB::mutable_coord_backend_id() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/generated-sources/gen-cpp/control_service.pb.h:6625:23 (impalad+0x20c03ed) #1 impala::QueryState::Init(impala::ExecQueryFInstancesRequestPB const*, impala::TExecPlanFragmentInfo const&) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-state.cc:216:21 (impalad+0x20b8b29) #2 impala::QueryExecMgr::StartQuery(impala::ExecQueryFInstancesRequestPB const*, impala::TQueryCtx const&, impala::TExecPlanFragmentInfo const&) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/runtime/query-exec-mgr.cc:80:23 (impalad+0x20acb59) #3 impala::ControlService::ExecQueryFInstances(impala::ExecQueryFInstancesRequestPB const*, impala::ExecQueryFInstancesResponsePB*, kudu::rpc::RpcContext*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/service/control-service.cc:157:66 (impalad+0x22a621d) #4 impala::ControlServiceIf::ControlServiceIf(scoped_refptr const&, scoped_refptr const&)::$_1::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/generated-sources/gen-cpp/control_service.service.cc:70:13 (impalad+0x23622a4) #5 std::_Function_handler const&, scoped_refptr const&)::$_1>::_M_invoke(std::_Any_data const&, google::protobuf::Message const*&&, google::protobuf::Message*&&, kudu::rpc::RpcContext*&&) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:316:2 (impalad+0x23620ed) #6 std::function::operator()(google::protobuf::Message const*, google::protobuf::Message*, kudu::rpc::RpcContext*) const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/gcc-7.5.0/lib/gcc/x86_64-pc-linux-gnu/7.5.0/../../../../include/c++/7.5.0/bits/std_function.h:706:14 (impalad+0x2a4a453) #7 kudu::rpc::GeneratedServiceIf::Handle(kudu::rpc::InboundCall*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/kudu/rpc/service_if.cc:139:3 (impalad+0x2a49efe) #8 impala::ImpalaServicePool::RunThread() /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/rpc/impala-service-pool.cc:272:15 (impalad+0x2011a12) #9 boost::_mfi::mf0::operator()(impala::ImpalaServicePool*) const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:49:29 (impalad+0x2017a16) #10 void boost::_bi::list1 >::operator(), boost::_bi::list0>(boost::_bi::type, boost::_mfi::mf0&, boost::_bi::list0&, int) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:259:9 (impalad+0x201796a) #11 boost::_bi::bind_t, boost::_bi::list1 > >::operator()() /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/bind.hpp:1222:16 (impalad+0x20178f3) #12 boost::detail::function::void_function_obj_invoker0, boost::_bi::list1 > >, void>::invoke(boost::detail::function::function_buffer&) /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:159:11 (impalad+0x20176e9) #13 boost::function0::operator()() const /data/jenkins/workspace/impala-private-parameterized/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/function/function_template.hpp:770:14 (impalad+0x1f666f1) #14 impala::Thread::SuperviseThread(std::__cxx11::basic_string, std::allocator > const&, std::__cxx11::basic_string, std::allocator > const&, boost::function, impala::ThreadDebugInfo const*, impala::Promise*) /data/jenkins/workspace/impala-private-parameterized/repos/Impala/be/src/util/thread.cc:360:3 (impalad+0x252644b) #15 void boost::_bi::list5, std::allocator > >, boost::_bi::value, std::allocator > >, boost::_bi::value >, boost::_bi::value, boost::_bi::value*> >::operator(), std::allocator > const&, std::__c
[jira] [Created] (IMPALA-10153) Support time travel for Iceberg tables
Zoltán Borók-Nagy created IMPALA-10153: -- Summary: Support time travel for Iceberg tables Key: IMPALA-10153 URL: https://issues.apache.org/jira/browse/IMPALA-10153 Project: IMPALA Issue Type: Bug Reporter: Zoltán Borók-Nagy Iceberg tables support snapshots/data versioning/time travel. It means we can query an older version of the table. Probably we'll need to extend Impala's SQL syntax to support such queries (Hive will also support such queries, so we should use the same syntax). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IMPALA-10152) Add support for Iceberg HiveCatalog
Zoltán Borók-Nagy created IMPALA-10152: -- Summary: Add support for Iceberg HiveCatalog Key: IMPALA-10152 URL: https://issues.apache.org/jira/browse/IMPALA-10152 Project: IMPALA Issue Type: Sub-task Reporter: Zoltán Borók-Nagy HadoopTables and HadoopCatalog only work on filesystems that support atomic rename. Therefore they are not safe on S3. HiveCatalog uses the Hive Metastore to keep track of Iceberg tables, so we should use this interface when we store our tables in object stores. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IMPALA-10151) Upgrade Iceberg to a version that is compatible with Hive3
Zoltán Borók-Nagy created IMPALA-10151: -- Summary: Upgrade Iceberg to a version that is compatible with Hive3 Key: IMPALA-10151 URL: https://issues.apache.org/jira/browse/IMPALA-10151 Project: IMPALA Issue Type: Sub-task Reporter: Zoltán Borók-Nagy To support Iceberg on S3, we'll need to use HiveCatalog. The current Iceberg jars only compatible with Hive2, but Impala uses Hive3. So we'll need an Iceberg version that is compatible with Hive3 as well. -- This message was sent by Atlassian Jira (v8.3.4#803005)