Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/23976 )
Change subject: IMPALA-14770: Upgrade impala-python3 to 3.11.14 ...................................................................... IMPALA-14770: Upgrade impala-python3 to 3.11.14 This upgrades the impala-python3 virtualenv to use Python 3.11.14. Upgrading to Python 3.11 allows us to use the most recent version of most packages. Python 3.11 delivered performance improvements over previous versions. It seems to reduce the runtime of precommit by about 15-20 minutes (~4:40 -> ~4:25). This fixes code using removed library functions: - XML's getiterator() was removed and code needs to use iter() - thread's isAlive() was removed and code needs to use is_alive() This fixes some pieces of code to avoid warnings: - The "pipes" package is deprecated, so this changes code using pipes.quote() to use shlex.quote() instead. - This changes some regex patterns to use raw strings to avoid warnings about invalid escape sequences. This upgrades a variety of packages (some of them required, many of them optional): 1. Upgrades setuptools to a version from about a month ago 2. Upgrades requests to the latest version - Addresses some warnings that show for pytest - Requests got stricter about certificates and server-cert.pem is no longer considered valid. - This regenerates the server-key.pem/server-key-password.pem/server-cert.pem to add a subject alternative name of localhost. 3. Upgrades python-magic to the latest and fixes dump_breakpad_symbols.py 4. Upgrades psutil to the latest and fixes tests/common/impala_cluster.py to handle the changes in how it represents zombie processes 5. Upgrades several other Python packages to more recent versions without needing code changes (allpairspy, flake8, hdfs, kazoo, prometheus-client, six, sasl, avro, distro, pytz, Cython). Removes unused pg8000 package. There were issues upgrading some other packages, so this does not upgrade them: - Newer versions of sqlparse have a regression in split.split() that makes it inoperable for us. - Newer versions of pexpect would require adjustment in the interactive shell tests. - New versions of pytest are not difficult but require additional changes that would be better suited to a separate commit Testing: - Ran precommit - Ran exhaustive release jobs Change-Id: I9383ebfb310a4ee30134e02b195c332ffbb6a7ec Reviewed-on: http://gerrit.cloudera.org:8080/23976 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Michael Smith <[email protected]> --- M be/src/testutil/certificates-info.txt M be/src/testutil/server-cert.pem M be/src/testutil/server-key-password.pem M be/src/testutil/server-key.pem M bin/dump_breakpad_symbols.py M bin/impala-config.sh M infra/python/deps/requirements.txt M infra/python/deps/setuptools-requirements.txt M tests/common/custom_cluster_test_suite.py M tests/common/impala_cluster.py M tests/common/resource_pool_config.py M tests/common/test_result_verifier.py M tests/comparison/cluster.py M tests/comparison/leopard/controller.py M tests/custom_cluster/test_admission_controller.py M tests/stress/query_retries_stress_runner.py M tests/util/auto_scaler.py 17 files changed, 164 insertions(+), 137 deletions(-) Approvals: Impala Public Jenkins: Verified Michael Smith: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/23976 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9383ebfb310a4ee30134e02b195c332ffbb6a7ec Gerrit-Change-Number: 23976 Gerrit-PatchSet: 20 Gerrit-Owner: Joe McDonnell <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
