Hello Jason Fehr, Csaba Ringhofer, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24026
to look at the new patch set (#5).
Change subject: IMPALA-14776 (part 1): Auto-close manually created impala
clients
......................................................................
IMPALA-14776 (part 1): Auto-close manually created impala clients
There are several APIs in ImpalaTestSuite that tests can use to
create a new client:
- create_impala_client()
- create_impala_client_from_vector()
- create_client_for_nth_impalad()
If the tests don't close the client properly, pytest will report
a leak of a socket. It seems possible that this type of leak can
also result in leaving an unclosed session on the impalad. This can
trigger TestValidateMetrics.test_metrics_are_zero(), which has
been an issue since switching to pytest 6.2.5.
This modifies those APIs to register the clients in a list and
close them during test method teardown. That should prevent
resource leaks (sockets, sessions) associated with these clients.
The clients may have been cleaned up by other means (with-as or
try-finally or a manual close), so it is heavily relying on
close() being idempotent.
The list is maintained at the instance level, and it is set in
setup_method() and freed in teardown_method(). This means that
test classes need to call ImpalaTestSuite::setup_method() and
teardown_method(). This modifies several test classes to call
these methods appropriately.
This also means that the APIs are no longer class methods, so callers
need to call them on their instance rather than on the class
itself. This required changing a handful of locations.
ImpalaTestSuite's built-in clients use an internal API so that
they can be reused across test methods. They instead are scoped
at the class level.
Testing:
- Ran an exhaustive job
Change-Id: Ib550527838a81cd2aaf69bb715080f6ac6da3786
---
M tests/common/custom_cluster_test_suite.py
M tests/common/impala_test_suite.py
M tests/conftest.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_breakpad.py
M tests/custom_cluster/test_parquet_max_page_header.py
M tests/custom_cluster/test_permanent_udfs.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_redaction.py
M tests/custom_cluster/test_scratch_disk.py
M tests/metadata/test_hdfs_encryption.py
M tests/metadata/test_hdfs_permissions.py
M tests/metadata/test_load.py
M tests/query_test/test_compressed_formats.py
M tests/query_test/test_insert_permutation.py
M tests/stress/test_acid_stress.py
M tests/stress/test_update_stress.py
17 files changed, 162 insertions(+), 55 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/24026/5
--
To view, visit http://gerrit.cloudera.org:8080/24026
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib550527838a81cd2aaf69bb715080f6ac6da3786
Gerrit-Change-Number: 24026
Gerrit-PatchSet: 5
Gerrit-Owner: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Jason Fehr <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
