[
https://issues.apache.org/jira/browse/IMPALA-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
LiFu He reassigned IMPALA-10559:
--------------------------------
Assignee: Riza Suminto (was: LiFu He)
> TestScratchLimit seems flaky
> ----------------------------
>
> Key: IMPALA-10559
> URL: https://issues.apache.org/jira/browse/IMPALA-10559
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.0
> Reporter: Yida Wu
> Assignee: Riza Suminto
> Priority: Blocker
> Labels: broken-build
> Fix For: Impala 4.0
>
>
>
> The TestScratchLimit testcase runs fine alone. But if runs TestScratchDir
> first, then runs TestScratchLimit, it will always fail as follow shows unless
> restarts the services (e.g. using "./buildall.sh -noclean -notests -format
> -start_minicluster -start_impala_cluster").
> {code:java}
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_low_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> query_test/test_scratch_limit.py::TestScratchLimit::test_without_specifying_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_high_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_unlimited_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw1] PASSED
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw5] FAILED
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_low_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw4] FAILED
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_high_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw2] FAILED
> query_test/test_scratch_limit.py::TestScratchLimit::test_without_specifying_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw3] FAILED
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_unlimited_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw0] PASSED
> query_test/test_scratch_limit.py::TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> ------------------------------------------- generated xml file:
> /home/yida/Impala/logs/ee_tests/results/TEST-impala-parallel.xml
> --------------------------------------------
> ==========================================================================
> short test summary info
> ==========================================================================
> FAIL
> query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_low_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> FAIL
> query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_high_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> FAIL
> query_test/test_scratch_limit.py::TestScratchLimit::()::test_without_specifying_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> FAIL
> query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_unlimited_scratch_limit[protocol:
> beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> =================================================================================
> FAILURES
> ==================================================================================
> TestScratchLimit.test_with_low_scratch_limit[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw5] linux2 -- Python 2.7.16
> /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
> query_test/test_scratch_limit.py:90: in test_with_low_scratch_limit
> assert expected_error % scratch_limit_in_bytes in str(e)
> E assert ('Scratch space limit of %s bytes exceeded' % 25165824) in
> 'ImpalaBeeswaxException:\n Query aborted:Could not create files in any
> configured scratch directories (--scratch_dirs=...ous errors that may have
> prevented creating or writing scratch files. The following directories were
> at capacity: \n\n'
> E + where 'ImpalaBeeswaxException:\n Query aborted:Could not create files in
> any configured scratch directories (--scratch_dirs=...ous errors that may
> have prevented creating or writing scratch files. The following directories
> were at capacity: \n\n' = str(ImpalaBeeswaxException())
> ---------------------------------------------------------------------------
> Captured stderr setup
> ---------------------------------------------------------------------------
> SET
> client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_low_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshol;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2021-03-03 07:12:08,115 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2021-03-03 07:12:08,140 INFO MainThread: Closing active operation
> ---------------------------------------------------------------------------
> Captured stderr call
> ----------------------------------------------------------------------------
> SET
> client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_low_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_threshol;
> SET scratch_limit=24m;
> SET batch_size=0;
> SET num_nodes=0;
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET buffer_pool_limit=32m;
> -- executing against localhost:21000
> select o_orderdate, o_custkey, o_comment
> from tpch.orders
> order by o_orderdate
> ;
> -- 2021-03-03 07:12:08,329 INFO MainThread: Started query
> f44b4a50bec6b10d:e45a053d00000000
> TestScratchLimit.test_with_high_scratch_limit[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw4] linux2 -- Python 2.7.16
> /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
> query_test/test_scratch_limit.py:74: in test_with_high_scratch_limit
> self.execute_query_expect_success(self.client, self.spilling_sort_query,
> exec_option)
> common/impala_test_suite.py:809: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:817: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:918: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Could not create files in any configured scratch directories
> (--scratch_dirs=) on backend 'yida-OptiPlex-7060:27001'. 0 of scratch is
> currently in use by this Impala Daemon (0 by this query). See logs for
> previous errors that may have prevented creating or writing scratch files.
> The following directories were at capacity:
> ---------------------------------------------------------------------------
> Captured stderr setup
> ---------------------------------------------------------------------------
> SET
> client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_high_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_thresho;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2021-03-03 07:12:08,114 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2021-03-03 07:12:08,136 INFO MainThread: Closing active operation
> ---------------------------------------------------------------------------
> Captured stderr call
> ----------------------------------------------------------------------------
> SET
> client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_high_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_thresho;
> SET scratch_limit=500m;
> SET batch_size=0;
> SET num_nodes=0;
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET buffer_pool_limit=32m;
> -- executing against localhost:21000
> select o_orderdate, o_custkey, o_comment
> from tpch.orders
> order by o_orderdate
> ;
> -- 2021-03-03 07:12:08,329 INFO MainThread: Started query
> 814fccc9c1242331:83738b0800000000
> TestScratchLimit.test_without_specifying_scratch_limit[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw2] linux2 -- Python 2.7.16
> /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
> query_test/test_scratch_limit.py:118: in test_without_specifying_scratch_limit
> self.execute_query_expect_success(self.client, self.spilling_sort_query,
> exec_option)
> common/impala_test_suite.py:809: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:817: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:918: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Could not create files in any configured scratch directories
> (--scratch_dirs=) on backend 'yida-OptiPlex-7060:27002'. 0 of scratch is
> currently in use by this Impala Daemon (0 by this query). See logs for
> previous errors that may have prevented creating or writing scratch files.
> The following directories were at capacity:
> ---------------------------------------------------------------------------
> Captured stderr setup
> ---------------------------------------------------------------------------
> SET
> client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_without_specifying_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2021-03-03 07:12:08,114 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2021-03-03 07:12:08,137 INFO MainThread: Closing active operation
> ---------------------------------------------------------------------------
> Captured stderr call
> ----------------------------------------------------------------------------
> SET
> client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_without_specifying_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_row;
> SET batch_size=0;
> SET num_nodes=0;
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET buffer_pool_limit=32m;
> -- executing against localhost:21000
> select o_orderdate, o_custkey, o_comment
> from tpch.orders
> order by o_orderdate
> ;
> -- 2021-03-03 07:12:08,329 INFO MainThread: Started query
> 944af93b81ee62cf:ab33f08800000000
> TestScratchLimit.test_with_unlimited_scratch_limit[protocol: beeswax |
> exec_option: {'batch_size': 0, 'num_nodes': 0,
> 'disable_codegen_rows_threshold': 5000, 'disable_codegen': False,
> 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format:
> text/none]
> [gw3] linux2 -- Python 2.7.16
> /home/yida/Impala/bin/../infra/python/env-gcc7.5.0/bin/python
> query_test/test_scratch_limit.py:110: in test_with_unlimited_scratch_limit
> self.execute_query_expect_success(self.client, self.spilling_sort_query,
> exec_option)
> common/impala_test_suite.py:809: in wrapper
> return function(*args, **kwargs)
> common/impala_test_suite.py:817: in execute_query_expect_success
> result = cls.__execute_query(impalad_client, query, query_options, user)
> common/impala_test_suite.py:918: in __execute_query
> return impalad_client.execute(query, user=user)
> common/impala_connection.py:205: in execute
> return self.__beeswax_client.execute(sql_stmt, user=user)
> beeswax/impala_beeswax.py:187: in execute
> handle = self.__execute_query(query_string.strip(), user=user)
> beeswax/impala_beeswax.py:365: in __execute_query
> self.wait_for_finished(handle)
> beeswax/impala_beeswax.py:386: in wait_for_finished
> raise ImpalaBeeswaxException("Query aborted:" + error_log, None)
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E Query aborted:Could not create files in any configured scratch directories
> (--scratch_dirs=) on backend 'yida-OptiPlex-7060:27001'. 0 of scratch is
> currently in use by this Impala Daemon (0 by this query). See logs for
> previous errors that may have prevented creating or writing scratch files.
> The following directories were at capacity:
> ---------------------------------------------------------------------------
> Captured stderr setup
> ---------------------------------------------------------------------------
> SET
> client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_unlimited_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_th;
> -- connecting to: localhost:21000
> -- connecting to localhost:21050 with impyla
> -- 2021-03-03 07:12:08,114 INFO MainThread: Closing active operation
> -- connecting to localhost:28000 with impyla
> -- 2021-03-03 07:12:08,146 INFO MainThread: Closing active operation
> ---------------------------------------------------------------------------
> Captured stderr call
> ----------------------------------------------------------------------------
> SET
> client_identifier=query_test/test_scratch_limit.py::TestScratchLimit::()::test_with_unlimited_scratch_limit[protocol:beeswax|exec_option:{'batch_size':0;'num_nodes':0;'disable_codegen_rows_threshold':5000;'disable_codegen':False;'abort_on_error':1;'exec_single_node_rows_th;
> SET scratch_limit=-1;
> SET batch_size=0;
> SET num_nodes=0;
> SET disable_codegen_rows_threshold=5000;
> SET disable_codegen=False;
> SET abort_on_error=1;
> SET exec_single_node_rows_threshold=0;
> SET buffer_pool_limit=32m;
> -- executing against localhost:21000
> select o_orderdate, o_custkey, o_comment
> from tpch.orders
> order by o_orderdate
> ;
> -- 2021-03-03 07:12:08,329 INFO MainThread: Started query
> 5c492cfd62287b03:f9c3370b00000000
> ==================================================================== 4
> failed, 2 passed in 14.11 seconds
> ====================================================================
> Running TestExecutor with args: ['--ignore', 'query_test', '--ignore',
> 'catalog_service', '--ignore', 'hs2', '--ignore', 'experiments', '--ignore',
> 'data_errors', '--ignore', 'performance', '--ignore', 'beeswax', '--ignore',
> 'authorization', '--ignore', 'metadata', '--ignore', 'shell', '--ignore',
> 'benchmark', '--ignore', 'infra', '--ignore', 'statestore', '--ignore',
> 'failure', '--ignore', 'webserver', '--ignore', 'comparison', '--ignore',
> 'stress', '--ignore', 'util', '--ignore', 'custom_cluster', '--ignore',
> 'common', '--ignore', 'observability', '--ignore', 'unittests',
> '--resultlog',
> '/home/yida/Impala/logs/ee_tests/results/TEST-impala-verify-metrics.log',
> '--junitxml',
> '/home/yida/Impala/logs/ee_tests/results/TEST-impala-verify-metrics.xml',
> 'verifiers/test_verify_metrics.py']
> rootLoggerLevel = INFO
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscr...@impala.apache.org
For additional commands, e-mail: issues-all-h...@impala.apache.org
