[
https://issues.apache.org/jira/browse/IMPALA-14171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18016412#comment-18016412
]
Quanlong Huang commented on IMPALA-14171:
-----------------------------------------
Checked the logs (uploaded as [^query_log.txt] ), after sleeping 10s before
execution, the query still takes longer than 2s to finish then got cancelled.
{noformat}
I20250826 09:21:56.092494 214295 Frontend.java:2398]
f7448e24bd919051:0f4727a200000000] Analyzing query: SELECT * FROM tpch.lineitem
limit 1 db: default
...
I20250826 09:21:56.095382 214295 DebugUtils.java:185]
f7448e24bd919051:0f4727a200000000] Sleeping for 10000 msec to execute debug
action plan_create
...
I20250826 09:22:06.116753 216321 coordinator.cc:155]
f7448e24bd919051:0f4727a200000000] Exec()
query_id=f7448e24bd919051:0f4727a200000000 stmt=SELECT * FROM tpch.lineitem
limit 1
I20250826 09:22:06.116837 216321 coordinator.cc:531]
f7448e24bd919051:0f4727a200000000] starting execution on 1 backends for
query_id=f7448e24bd919051:0f4727a200000000
...
I20250826 09:22:08.954799 411 impala-server.cc:2969] Expiring query
f7448e24bd919051:0f4727a200000000 due to execution time limit of 2s.
...
I20250826 09:22:11.318619 216328 query-state.cc:1015]
f7448e24bd919051:0f4727a200000000] Instance completed.
instance_id=f7448e24bd919051:0f4727a200000000 #in-flight=2 status=CANCELLED:
Cancelled{noformat}
Unfortunately the profile is flushed out so we don't know why the execution is
slow. The execution might wait in increasing memory reservation since I see
lots of logs indicating the pool is almost full, i.e.
fraction_of_pool_total_mem is almost 1:
{noformat}
I20250826 09:22:06.407481 216370 admission-controller.cc:2404]
c54c0bdb72b68766:19097e2300000000] Stats: agg_num_running=9, agg_num_queued=0,
agg_mem_reserved=14.00 GB, agg_user_loads=,
local_host(local_mem_admitted=13.03 GB, local_trivial_running=0,
num_admitted_running=8, num_queued=0, backend_mem_reserved=4.83 GB,
user_loads=, topN_query_stats: queries=[cc4d4f313f9d5cb8:f1925ef300000000,
fe4fda751984f227:403a913e00000000, 194b8688d3eac53b:638eaebd00000000,
af4545a080b68e27:a4887df700000000, f7448e24bd919051:0f4727a200000000],
total_mem_consumed=708.20 MB, fraction_of_pool_total_mem=0.99;
pool_level_stats: num_running=9, min=0, max=323.25 MB, pool_total_mem=712.26
MB, average_per_query=79.14 MB) {noformat}
To deflake the test, we can mark it to run serially:
[http://gerrit.cloudera.org:8080/23351]
CC [~baggio000]
> TestQueryExecTimeLimit.test_exec_time_limit_long_plan hits expiration
> ---------------------------------------------------------------------
>
> Key: IMPALA-14171
> URL: https://issues.apache.org/jira/browse/IMPALA-14171
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 5.0.0
> Reporter: Joe McDonnell
> Priority: Critical
> Labels: broken-build, flaky
> Attachments: query_log.txt
>
>
> TestQueryExecTimeLimit.test_exec_time_limit_long_plan hit this error:
> {noformat}
> query_test/test_exec_time_limit.py:50: in test_exec_time_limit_long_plan
> result = self.execute_query(query, exec_options)
> common/impala_test_suite.py:1119: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:1161: in execute_query
> return self.__execute_query(self.client, query, query_options)
> common/impala_test_suite.py:1292: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:691: in execute
> fetch_exec_summary=fetch_exec_summary, profile_format=profile_format)
> common/impala_connection.py:704: in __fetch_results_and_profile
> profile_format=profile_format)
> common/impala_connection.py:867: in __fetch_results
> result_tuples = cursor.fetchall()
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:624:
> in fetchall
> elements = self._pop_from_buffer(self.buffersize)
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:701:
> in _pop_from_buffer
> self._ensure_buffer_is_filled()
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:683:
> in _ensure_buffer_is_filled
> convert_strings_to_unicode=self.convert_strings_to_unicode)
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:1506:
> in fetch
> resp = self._rpc('FetchResults', req, False)
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:1181:
> in _rpc
> err_if_rpc_not_ok(response)
> /data/jenkins/workspace/impala-asf-master-exhaustive-data-cache/repos/Impala/infra/python/env-gcc10.4.0/lib/python2.7/site-packages/impala/hiveserver2.py:867:
> in err_if_rpc_not_ok
> raise HiveServer2Error(resp.status.errorMessage)
> E HiveServer2Error: Query 584fd63b893c49c4:90cdfbce00000000 failed:
> E Query 584fd63b893c49c4:90cdfbce00000000 expired due to execution time
> limit of 2s000ms{noformat}
> This is intermittent, only seen once so far.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
