Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/20770 )
Change subject: IMPALA-12426: Query History Table ...................................................................... Patch Set 13: (18 comments) http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/impala-server.h File be/src/service/impala-server.h: http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/impala-server.h@1256 PS12, Line 1256: std::unique_ptr<Thread> admission_heartbeat_thr > Yes. This boolean is used to guard against spurious wakeups of the complet Ack http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/impala-server.h@1281 PS12, Line 1281: : /// Default query options in the form of TQueryOptions and b > Done Done http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/impala-server.h@1797 PS12, Line 1797: : : > Done Done http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/internal-server.cc File be/src/service/internal-server.cc: http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/internal-server.cc@117 PS12, Line 117: internal_query_id.hi != 0 && internal_query_id.lo != 0 > Done Done http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/internal-server.cc@121 PS12, Line 121: session_id > The CloseSession function does all the necessary checks on its own. Ack http://gerrit.cloudera.org:8080/#/c/20770/13/be/src/service/internal-server.cc File be/src/service/internal-server.cc: http://gerrit.cloudera.org:8080/#/c/20770/13/be/src/service/internal-server.cc@117 PS13, Line 117: internal_query_id.hi != 0 && internal_query_id.lo != 0 Use the new function in uid-util.h http://gerrit.cloudera.org:8080/#/c/20770/13/be/src/service/query-state-record.cc File be/src/service/query-state-record.cc: http://gerrit.cloudera.org:8080/#/c/20770/13/be/src/service/query-state-record.cc@216 PS13, Line 216: if (labels_iter_->find("Ready to start on ", 0) == 0) { : it.first = READY_TO_START; : } else if(labels_iter_->find("All ", 0) == 0) { : it.first = ALL_EXEC_BACKENDS; Can you make the corresponding event string in coordinator.cc into constant and then use that constant for find argument here? http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/query-state.cc File be/src/service/query-state.cc: http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/query-state.cc@95 PS12, Line 95: > I would prefer to leave this code as-is since it was directly moved from ht Ack http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/workload-management.cc File be/src/service/workload-management.cc: http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/workload-management.cc@50 PS12, Line 50: : DEFINE_bool(enable_workload_mgmt, false, : "Specifies if Impala will automatically write completed queries in the query log " : "table. If this value is set to true and then later removed, the query log table " : "will remain intact and accessible."); : > I've thought about this some, and I think we should go with a bool flag for Done http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/workload-management.cc@63 PS12, Line 63: return false; : }); : > Done Done http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/workload-management.cc@186 PS12, Line 186: /// table. : static std::list<std::tuple<std::string, std::string, WMFieldParser>> fields_; : > I added comments in impala-server.h for the functions implemented in worklo Done http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/workload-management.cc@494 PS12, Line 494: fields_.push_back(MakeTuple("network_address", _clientNetworkAddress)); : fields_.push_back(MakeTuple("start_time_utc", _queryStartTime, "TIMESTAMP")); : fields_.push_back(MakeTuple("total_time_us", _queryDuration, "BIGINT")); : fields_.push_back(MakeTuple("s > No, field order is not important except to the custom cluster python test ( Done http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/service/workload-management.cc@598 PS12, Line 598: : void ImpalaServer::ShutdownWorkloadManagement() { : unique_lock<mutex> l(completed_queries_threadstate_mu_); : if (completed_queries_thread_state_ == RUNNING) { : completed_queries_thread_state_ = SHUTDOWN_REQUESTED; : completed_queries_cv_.notify_all(); : completed_queries_shutdown_cv_.wait_for(l, : std::chrono::seconds(FLAGS_query_log_shutdown_deadline_s), : []{ return completed_queries_thread_state_ == SHUTDOWN; }); : } > Modifications to the completed queries table will be handled in the future. Ack http://gerrit.cloudera.org:8080/#/c/20770/13/be/src/util/network-util.h File be/src/util/network-util.h: http://gerrit.cloudera.org:8080/#/c/20770/13/be/src/util/network-util.h@114 PS13, Line 114: const int uds_addr_compare = first.uds_address.compare(second.uds_address); Should uds_address be the last tie breaker? http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/util/network-util.h File be/src/util/network-util.h: http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/util/network-util.h@114 PS12, Line 114: const int uds_addr_compare = first.uds_address.compare(second.uds_address); : if (uds_addr_compare < 0) { > Added. Also added ctest for this struct. Done http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/util/string-util.h File be/src/util/string-util.h: http://gerrit.cloudera.org:8080/#/c/20770/12/be/src/util/string-util.h@99 PS12, Line 99: /// is a space) or to the lower case of the character and writes the converted character : /// to a stringstream. : /// > Yes, it should. I was not aware any events have special characters in them Done http://gerrit.cloudera.org:8080/#/c/20770/12/tests/custom_cluster/test_query_log.py File tests/custom_cluster/test_query_log.py: http://gerrit.cloudera.org:8080/#/c/20770/12/tests/custom_cluster/test_query_log.py@39 PS12, Line 39: : class TestQueryLogTable(CustomClusterTestSuite): > All the clusters are already multi-coordinator. I updated a few of the tes Ok, I see test_query_log_table_flush_on_shutdown has run query using client2. http://gerrit.cloudera.org:8080/#/c/20770/12/tests/custom_cluster/test_query_log.py@633 PS12, Line 633: self.__run_sql_inject(impalad, client, sql1_str, "closing quotes", 3) : : # Try a sql inject attack with terminating quote and semicolon. : sql2_str = "select 1'); drop table {0}; select('" \ : .format(self.QUERY_TBL) : self._ > Without this sleep, the test becomes flaky. Impala does not always return Maybe it is due to stale metadata. Is it worth to run "invalidate metadata sys.impala_query_log" before querying it? Just want to be careful so this test does not become flaky due to insufficient sleep time later. -- To view, visit http://gerrit.cloudera.org:8080/20770 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2d2da9d450fba4e789400cfa62927fc25d34f844 Gerrit-Change-Number: 20770 Gerrit-PatchSet: 13 Gerrit-Owner: Jason Fehr <jf...@cloudera.com> Gerrit-Reviewer: Andrew Sherman <asher...@cloudera.com> Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com> Gerrit-Reviewer: Jason Fehr <jf...@cloudera.com> Gerrit-Reviewer: Michael Smith <michael.sm...@cloudera.com> Gerrit-Reviewer: Riza Suminto <riza.sumi...@cloudera.com> Gerrit-Comment-Date: Mon, 29 Jan 2024 22:24:31 +0000 Gerrit-HasComments: Yes
