[jira] [Commented] (IMPALA-11072) TestSpillingDebugActionDimensions.test_spilling is flaky
[
https://issues.apache.org/jira/browse/IMPALA-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17490588#comment-17490588
]
Riza Suminto commented on IMPALA-11072:
---
Hi [~stigahuang] ,
I've seen some flakiness in downstream build for this exact testcase.
There seems to be inconsistent number of fragments assigned to each impalad due
to different parquet file count/size being created on each run.
I think it is better to investigate in separate JIRA.
> TestSpillingDebugActionDimensions.test_spilling is flaky
>
>
> Key: IMPALA-11072
> URL: https://issues.apache.org/jira/browse/IMPALA-11072
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.0.0
>Reporter: Riza Suminto
>Assignee: Riza Suminto
>Priority: Major
> Fix For: Impala 4.1.0
>
>
> We have seen some failure of TestSpillingDebugActionDimensions.test_spilling
> in GVO jenkins job and downstream nightly tests. Latest one happen in
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/15503/]
>
>
> {code:java}
> query_test/test_spilling.py:75: in test_spilling
> self.run_test_case('QueryTest/spilling', vector)
> common/impala_test_suite.py:743: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:636: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*SpilledPartitions: .* \([1-9][0-9]*\)
> E
> E ACTUAL PROFILE:
> E Query (id=8b433ac02c4d3fd2:3c50b7c4):
> E - InactiveTotalTime: 0.000ns
> E - TotalTime: 0.000ns
> E Summary:
> E Session ID: 9448bded8acf05c6:428a7a797f6b9483
> E Session Type: BEESWAX
> E Start Time: 2022-01-08 09:37:07.647285000
> E End Time: 2022-01-08 09:37:15.514936000
> E Query Type: QUERY
> E Query State: FINISHED
> E Impala Query State: FINISHED
> E Query Status: OK
> E Impala Version: impalad version 4.1.0-SNAPSHOT RELEASE (build
> 560ff976d3a08920a08b4ce3325a1dd9dbe81765)
> E User: ubuntu
> E Connected User: ubuntu
> E Delegated User:
> E Network Address: :::127.0.0.1:44648
> E Default Db: tpch_parquet
> E Sql Statement: select count(l1.l_tax)
> E from
> E lineitem l1,
> E lineitem l2,
> E lineitem l3
> E where
> E l1.l_tax < 0.01 and
> E l2.l_tax < 0.04 and
> E l1.l_orderkey = l2.l_orderkey and
> E l1.l_orderkey = l3.l_orderkey and
> E l1.l_comment = l3.l_comment and
> E l1.l_shipdate = l3.l_shipdate
> E Coordinator: ip-172-31-21-231:27000
> E Query Options (set by configuration):
> BUFFER_POOL_LIMIT=225443840,MT_DOP=0,DEFAULT_SPILLABLE_BUFFER_SIZE=262144,TIMEZONE=Universal,CLIENT_IDENTIFIER=query_test/test_spilling.py::TestSpillingDebugActionDimensions::()::test_spilling[protocol:beeswax|exec_option:{'mt_dop':0;'debug_action':None;'default_spillable_buffer_size':'256k'}|table_format:parquet/none]
> E Query Options (set by configuration and planner):
> BUFFER_POOL_LIMIT=225443840,MT_DOP=0,DEFAULT_SPILLABLE_BUFFER_SIZE=262144,TIMEZONE=Universal,CLIENT_IDENTIFIER=query_test/test_spilling.py::TestSpillingDebugActionDimensions::()::test_spilling[protocol:beeswax|exec_option:{'mt_dop':0;'debug_action':None;'default_spillable_buffer_size':'256k'}|table_format:parquet/none],MINMAX_FILTER_THRESHOLD=0.5,MINMAX_FILTERING_LEVEL=PAGE
>
> ...{code}
>
> We should lower the configured BUFFER_POOL_LIMIT for this test to less than
> 215MB so that it spill more consistently.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
[jira] [Commented] (IMPALA-11072) TestSpillingDebugActionDimensions.test_spilling is flaky
[
https://issues.apache.org/jira/browse/IMPALA-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17490585#comment-17490585
]
Quanlong Huang commented on IMPALA-11072:
-
Saw this again in an unrelated change:
[https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/5248/]
{code:java}
query_test/test_spilling.py:75: in test_spilling
self.run_test_case('QueryTest/spilling', vector)
common/impala_test_suite.py:743: in run_test_case
update_section=pytest.config.option.update_results)
common/test_result_verifier.py:636: in verify_runtime_profile
actual))
E AssertionError: Did not find matches for lines in runtime profile:
E EXPECTED LINES:
E row_regex: .*SpilledPartitions: .* \([1-9][0-9]*\)
E
E ACTUAL PROFILE:
E Query (id=6d47a6323a1d674b:133714c1):
E DEBUG MODE WARNING: Query profile created while running a DEBUG build of
Impala. Use RELEASE builds to measure query performance.
E - InactiveTotalTime: 0.000ns
E - TotalTime: 0.000ns
E Summary:
E Session ID: d446ae1fd4c1316d:f637b568190aa0ba
E Session Type: BEESWAX
E Start Time: 2022-02-10 14:08:19.382722000
E End Time: 2022-02-10 14:08:39.128335000
E Query Type: QUERY
E Query State: FINISHED
E Impala Query State: FINISHED
E Query Status: OK
E Impala Version: impalad version 4.1.0-SNAPSHOT DEBUG (build
4e3271faf44433c5d3f847a0f965ab4ef1b1a48d)
E User: ubuntu
E Connected User: ubuntu
E Delegated User:
E Network Address: 172.18.0.1:41204
E Default Db: tpch_parquet
E Sql Statement: SELECT straight_join o_orderkey
E FROM (
E SELECT *
E FROM orders
E JOIN customer ON o_custkey = c_custkey
E JOIN nation ON c_nationkey = n_nationkey
E JOIN region ON n_regionkey = r_regionkey
E WHERE o_orderkey < 50) o1
E LEFT ANTI JOIN /*+broadcast*/ (
E SELECT *
E FROM orders
E JOIN customer ON o_custkey = c_custkey
E JOIN nation ON c_nationkey = n_nationkey
E JOIN region ON n_regionkey = r_regionkey
E WHERE o_orderkey < 50) o2 ON o1.o_orderkey = o2.o_orderkey
E AND o1.o_custkey = o2.o_custkey
E AND o1.o_orderstatus = o2.o_orderstatus
E AND o1.o_totalprice = o2.o_totalprice
E AND o1.o_orderdate = o2.o_orderdate
E AND o1.o_orderpriority = o2.o_orderpriority
E AND o1.o_clerk = o2.o_clerk
E AND o1.o_shippriority = o2.o_shippriority
E AND o1.o_comment = o2.o_comment
E AND o1.c_custkey = o2.c_custkey
E AND o1.c_name = o2.c_name
E AND o1.c_address = o2.c_address
E AND o1.c_nationkey = o2.c_nationkey
E AND o1.c_phone = o2.c_phone
E AND o1.c_acctbal = o2.c_acctbal
E AND o1.c_mktsegment = o2.c_mktsegment
E AND o1.n_nationkey = o2.n_nationkey
E AND o1.n_name = o2.n_name
E AND o1.n_regionkey = o2.n_regionkey
E AND o1.n_comment = o2.n_comment
E AND o1.r_name = o2.r_name
E AND o1.r_comment = o2.r_comment
E AND fnv_hash(o1.n_name) = fnv_hash(o2.n_name)
E AND fnv_hash(o1.r_name) = fnv_hash(o2.r_name)
E AND fnv_hash(o1.o_orderstatus) = fnv_hash(o2.o_orderstatus)
E AND fnv_hash(o1.o_shippriority) = fnv_hash(o2.o_shippriority)
E AND fnv_hash(o1.o_orderdate) = fnv_hash(o2.o_orderdate)
E AND fnv_hash(o1.o_orderpriority) = fnv_hash(o2.o_orderpriority)
E AND fnv_hash(o1.o_clerk) = fnv_hash(o2.o_clerk)
E ORDER BY o_orderkey
E Coordinator: 172.18.0.4:27000
E Query Options (set by configuration):
BUFFER_POOL_LIMIT=115343360,RUNTIME_FILTER_MODE=OFF,MT_DOP=0,DEFAULT_SPILLABLE_BUFFER_SIZE=262144,TIMEZONE=UTC,CLIENT_IDENTIFIER=query_test/test_spilling.py::TestSpillingDebugActionDimensions::()::test_spilling[protocol:beeswax|exec_option:{'mt_dop':0;'debug_action':None;'default_spillable_buffer_size':'256k'}|table_format:parquet/none]
E Query Options (set by configuration and planner):
BUFFER_POOL_LIMIT=115343360,RUNTIME_FILTER_MODE=OFF,MT_DOP=0,DEFAULT_SPILLABLE_BUFFER_SIZE=262144,TIMEZONE=UTC,CLIENT_IDENTIFIER=query_test/test_spilling.py::TestSpillingDebugActionDimensions::()::test_spilling[protocol:beeswax|exec_option:{'mt_dop':0;'debug_action':None;'default_spillable_buffer_size':'256k'}|table_format:parquet/none]
E Plan:
...{code}
This is another query. We probably need to set another BUFFER_POOL_LIMIT for
it. Should we reopen this Jira or create another one?
> TestSpillingDebugActionDimensions.test_spilling is flaky
>
>
> Key: IMPALA-11072
> URL: https://issues.apache.org/jira/browse/IMPALA-11072
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.0.0
>Reporter: Riza Suminto
[jira] [Commented] (IMPALA-11072) TestSpillingDebugActionDimensions.test_spilling is flaky
[
https://issues.apache.org/jira/browse/IMPALA-11072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17477499#comment-17477499
]
ASF subversion and git services commented on IMPALA-11072:
--
Commit 577fc2ee21a8cb6fc49f9df8367d63ebccc13e61 in impala's branch
refs/heads/master from Riza Suminto
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=577fc2e ]
IMPALA-11072: Deflake TestSpillingDebugActionDimensions.test_spilling
The first test case in TestSpillingDebugActionDimensions.test_spilling
has been flaky for not spilling any partitions in its hash join node.
This patch fixes the flakiness by reducing the buffer_pool_limit from
215 MB to 110 MB, which is around double of the query Per Host Min
Memory Reservation.
Testing:
- Manually run the first test case of
TestSpillingDebugActionDimensions.test_spilling. Verify that both of
the hash joins are spilling and the test pass.
Change-Id: Ie8802505e0dcae1be5e855107436805bd10e0077
Reviewed-on: http://gerrit.cloudera.org:8080/18138
Reviewed-by: Impala Public Jenkins
Tested-by: Impala Public Jenkins
> TestSpillingDebugActionDimensions.test_spilling is flaky
>
>
> Key: IMPALA-11072
> URL: https://issues.apache.org/jira/browse/IMPALA-11072
> Project: IMPALA
> Issue Type: Bug
> Components: Infrastructure
>Affects Versions: Impala 4.0.0
>Reporter: Riza Suminto
>Assignee: Riza Suminto
>Priority: Major
> Fix For: Impala 4.1.0
>
>
> We have seen some failure of TestSpillingDebugActionDimensions.test_spilling
> in GVO jenkins job and downstream nightly tests. Latest one happen in
> [https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/15503/]
>
>
> {code:java}
> query_test/test_spilling.py:75: in test_spilling
> self.run_test_case('QueryTest/spilling', vector)
> common/impala_test_suite.py:743: in run_test_case
> update_section=pytest.config.option.update_results)
> common/test_result_verifier.py:636: in verify_runtime_profile
> actual))
> E AssertionError: Did not find matches for lines in runtime profile:
> E EXPECTED LINES:
> E row_regex: .*SpilledPartitions: .* \([1-9][0-9]*\)
> E
> E ACTUAL PROFILE:
> E Query (id=8b433ac02c4d3fd2:3c50b7c4):
> E - InactiveTotalTime: 0.000ns
> E - TotalTime: 0.000ns
> E Summary:
> E Session ID: 9448bded8acf05c6:428a7a797f6b9483
> E Session Type: BEESWAX
> E Start Time: 2022-01-08 09:37:07.647285000
> E End Time: 2022-01-08 09:37:15.514936000
> E Query Type: QUERY
> E Query State: FINISHED
> E Impala Query State: FINISHED
> E Query Status: OK
> E Impala Version: impalad version 4.1.0-SNAPSHOT RELEASE (build
> 560ff976d3a08920a08b4ce3325a1dd9dbe81765)
> E User: ubuntu
> E Connected User: ubuntu
> E Delegated User:
> E Network Address: :::127.0.0.1:44648
> E Default Db: tpch_parquet
> E Sql Statement: select count(l1.l_tax)
> E from
> E lineitem l1,
> E lineitem l2,
> E lineitem l3
> E where
> E l1.l_tax < 0.01 and
> E l2.l_tax < 0.04 and
> E l1.l_orderkey = l2.l_orderkey and
> E l1.l_orderkey = l3.l_orderkey and
> E l1.l_comment = l3.l_comment and
> E l1.l_shipdate = l3.l_shipdate
> E Coordinator: ip-172-31-21-231:27000
> E Query Options (set by configuration):
> BUFFER_POOL_LIMIT=225443840,MT_DOP=0,DEFAULT_SPILLABLE_BUFFER_SIZE=262144,TIMEZONE=Universal,CLIENT_IDENTIFIER=query_test/test_spilling.py::TestSpillingDebugActionDimensions::()::test_spilling[protocol:beeswax|exec_option:{'mt_dop':0;'debug_action':None;'default_spillable_buffer_size':'256k'}|table_format:parquet/none]
> E Query Options (set by configuration and planner):
> BUFFER_POOL_LIMIT=225443840,MT_DOP=0,DEFAULT_SPILLABLE_BUFFER_SIZE=262144,TIMEZONE=Universal,CLIENT_IDENTIFIER=query_test/test_spilling.py::TestSpillingDebugActionDimensions::()::test_spilling[protocol:beeswax|exec_option:{'mt_dop':0;'debug_action':None;'default_spillable_buffer_size':'256k'}|table_format:parquet/none],MINMAX_FILTER_THRESHOLD=0.5,MINMAX_FILTERING_LEVEL=PAGE
>
> ...{code}
>
> We should lower the configured BUFFER_POOL_LIMIT for this test to less than
> 215MB so that it spill more consistently.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
