[ https://issues.apache.org/jira/browse/IMPALA-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Qifan Chen updated IMPALA-10291: -------------------------------- Description: During /impala-asf-master-core-asan test, the following error is seen. Error Message {code:java} query_test/test_tpcds_queries.py:394: in test_tpcds_q14_1 self.run_test_case(self.get_workload() + '-decimal_v2-q14-1', vector) common/impala_test_suite.py:662: in run_test_case result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:600: in __exec_in_impala result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:920: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:205: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:187: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:365: in __execute_query self.wait_for_finished(handle) beeswax/impala_beeswax.py:386: in wait_for_finished raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted:Failed to get minimum memory reservation of 391.06 MB on daemon impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 for query 334d5c63b6767a80:432c5a9e00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation. E Query(334d5c63b6767a80:432c5a9e00000000) could not allocate 391.06 MB without exceeding limit. E Error occurred on backend impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 E Memory left in process limit: 1.25 GB E Query(334d5c63b6767a80:432c5a9e00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0 E Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. {code} Stacktrace {code:java} query_test/test_tpcds_queries.py:394: in test_tpcds_q14_1 self.run_test_case(self.get_workload() + '-decimal_v2-q14-1', vector) common/impala_test_suite.py:662: in run_test_case result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:600: in __exec_in_impala result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:920: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:205: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:187: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:365: in __execute_query self.wait_for_finished(handle) beeswax/impala_beeswax.py:386: in wait_for_finished raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted:Failed to get minimum memory reservation of 391.06 MB on daemon impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 for query 334d5c63b6767a80:432c5a9e00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation. E Query(334d5c63b6767a80:432c5a9e00000000) could not allocate 391.06 MB without exceeding limit. E Error occurred on backend impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 E Memory left in process limit: 1.25 GB E Query(334d5c63b6767a80:432c5a9e00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0 E Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. {code} There are several instances of similar errors, involving q14, q14_1, q11, q20, q23_2, q24_1 and q39_2. The test report is [here|https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-asan/1003/testReport/], was: During /impala-asf-master-core-asan test, the following error is seen. Error Message query_test/test_tpcds_queries.py:394: in test_tpcds_q14_1 self.run_test_case(self.get_workload() + '-decimal_v2-q14-1', vector) common/impala_test_suite.py:662: in run_test_case result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:600: in __exec_in_impala result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:920: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:205: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:187: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:365: in __execute_query self.wait_for_finished(handle) beeswax/impala_beeswax.py:386: in wait_for_finished raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted:Failed to get minimum memory reservation of 391.06 MB on daemon impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 for query 334d5c63b6767a80:432c5a9e00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation. E Query(334d5c63b6767a80:432c5a9e00000000) could not allocate 391.06 MB without exceeding limit. E Error occurred on backend impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 E Memory left in process limit: 1.25 GB E Query(334d5c63b6767a80:432c5a9e00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0 E Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. Stacktrace query_test/test_tpcds_queries.py:394: in test_tpcds_q14_1 self.run_test_case(self.get_workload() + '-decimal_v2-q14-1', vector) common/impala_test_suite.py:662: in run_test_case result = exec_fn(query, user=test_section.get('USER', '').strip() or None) common/impala_test_suite.py:600: in __exec_in_impala result = self.__execute_query(target_impalad_client, query, user=user) common/impala_test_suite.py:920: in __execute_query return impalad_client.execute(query, user=user) common/impala_connection.py:205: in execute return self.__beeswax_client.execute(sql_stmt, user=user) beeswax/impala_beeswax.py:187: in execute handle = self.__execute_query(query_string.strip(), user=user) beeswax/impala_beeswax.py:365: in __execute_query self.wait_for_finished(handle) beeswax/impala_beeswax.py:386: in wait_for_finished raise ImpalaBeeswaxException("Query aborted:" + error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E Query aborted:Failed to get minimum memory reservation of 391.06 MB on daemon impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 for query 334d5c63b6767a80:432c5a9e00000000 due to following error: Memory limit exceeded: Could not allocate memory while trying to increase reservation. E Query(334d5c63b6767a80:432c5a9e00000000) could not allocate 391.06 MB without exceeding limit. E Error occurred on backend impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 E Memory left in process limit: 1.25 GB E Query(334d5c63b6767a80:432c5a9e00000000): Reservation=0 ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0 E Memory is likely oversubscribed. Reducing query concurrency or configuring admission control may help avoid this error. There are several instances of similar errors. > ASAN test harness may need to be beefed up to deal with query failure in > TestTpcdsDecimalV2Query.test_tpcds_q14_1 > ----------------------------------------------------------------------------------------------------------------- > > Key: IMPALA-10291 > URL: https://issues.apache.org/jira/browse/IMPALA-10291 > Project: IMPALA > Issue Type: Bug > Reporter: Qifan Chen > Assignee: Bikramjeet Vig > Priority: Major > > During /impala-asf-master-core-asan test, the following error is seen. > Error Message > {code:java} > query_test/test_tpcds_queries.py:394: in test_tpcds_q14_1 > self.run_test_case(self.get_workload() + '-decimal_v2-q14-1', vector) > common/impala_test_suite.py:662: in run_test_case result = exec_fn(query, > user=test_section.get('USER', '').strip() or None) > common/impala_test_suite.py:600: in __exec_in_impala result = > self.__execute_query(target_impalad_client, query, user=user) > common/impala_test_suite.py:920: in __execute_query return > impalad_client.execute(query, user=user) common/impala_connection.py:205: in > execute return self.__beeswax_client.execute(sql_stmt, user=user) > beeswax/impala_beeswax.py:187: in execute handle = > self.__execute_query(query_string.strip(), user=user) > beeswax/impala_beeswax.py:365: in __execute_query > self.wait_for_finished(handle) beeswax/impala_beeswax.py:386: in > wait_for_finished raise ImpalaBeeswaxException("Query aborted:" + > error_log, None) E ImpalaBeeswaxException: ImpalaBeeswaxException: E > Query aborted:Failed to get minimum memory reservation of 391.06 MB on daemon > impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 for query > 334d5c63b6767a80:432c5a9e00000000 due to following error: Memory limit > exceeded: Could not allocate memory while trying to increase reservation. E > Query(334d5c63b6767a80:432c5a9e00000000) could not allocate 391.06 MB without > exceeding limit. E Error occurred on backend > impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 E > Memory left in process limit: 1.25 GB E > Query(334d5c63b6767a80:432c5a9e00000000): Reservation=0 ReservationLimit=9.60 > GB OtherMemory=0 Total=0 Peak=0 E Memory is likely oversubscribed. Reducing > query concurrency or configuring admission control may help avoid this error. > {code} > Stacktrace > {code:java} > query_test/test_tpcds_queries.py:394: in test_tpcds_q14_1 > self.run_test_case(self.get_workload() + '-decimal_v2-q14-1', vector) > common/impala_test_suite.py:662: in run_test_case > result = exec_fn(query, user=test_section.get('USER', '').strip() or None) > common/impala_test_suite.py:600: in __exec_in_impala > result = self.__execute_query(target_impalad_client, query, user=user) > common/impala_test_suite.py:920: in __execute_query > return impalad_client.execute(query, user=user) > common/impala_connection.py:205: in execute > return self.__beeswax_client.execute(sql_stmt, user=user) > beeswax/impala_beeswax.py:187: in execute > handle = self.__execute_query(query_string.strip(), user=user) > beeswax/impala_beeswax.py:365: in __execute_query > self.wait_for_finished(handle) > beeswax/impala_beeswax.py:386: in wait_for_finished > raise ImpalaBeeswaxException("Query aborted:" + error_log, None) > E ImpalaBeeswaxException: ImpalaBeeswaxException: > E Query aborted:Failed to get minimum memory reservation of 391.06 MB on > daemon impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 > for query 334d5c63b6767a80:432c5a9e00000000 due to following error: Memory > limit exceeded: Could not allocate memory while trying to increase > reservation. > E Query(334d5c63b6767a80:432c5a9e00000000) could not allocate 391.06 MB > without exceeding limit. > E Error occurred on backend > impala-ec2-centos74-r5-4xlarge-ondemand-1c16.vpc.cloudera.com:27002 > E Memory left in process limit: 1.25 GB > E Query(334d5c63b6767a80:432c5a9e00000000): Reservation=0 > ReservationLimit=9.60 GB OtherMemory=0 Total=0 Peak=0 > E Memory is likely oversubscribed. Reducing query concurrency or > configuring admission control may help avoid this error. > {code} > There are several instances of similar errors, involving q14, q14_1, q11, > q20, q23_2, q24_1 and q39_2. The test report is > [here|https://master-02.jenkins.cloudera.com/job/impala-asf-master-core-asan/1003/testReport/], -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org For additional commands, e-mail: issues-all-h...@impala.apache.org