Joe McDonnell has uploaded this change for review. (
http://gerrit.cloudera.org:8080/19588
Change subject: IMPALA-11973: Add absolute_import, division to all eligible
Python files
......................................................................
IMPALA-11973: Add absolute_import, division to all eligible Python files
This takes steps to make Python 2 behave like Python 3 as
a way to flush out issues with running on Python 3. Specifically,
it handles two main differences:
1. Python 3 requires absolute imports within packages. This
can be emulated via "from __future__ import absolute_import"
2. Python 3 changed division to "true" division that doesn't
round to an integer. This can be emulated via
"from __future__ import division"
This changes all Python files to add imports for absolute_import
and division. For completeness, this also includes print_function in the
import.
I scrutinized each old-division location and converted some locations
to use the integer division '//' operator if it needed an integer
result (e.g. for indices, counts of records, etc). Some code was also using
relative imports and needed to be adjusted to handle absolute_import.
This fixes all Pylint warnings about no-absolute-import and old-division,
and these warnings are now banned.
Testing:
- Ran core tests
Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
---
M bin/banned_py3k_warnings.txt
M bin/dump_breakpad_symbols.py
M bin/generate_minidump_collection_testdata.py
M bin/get_code_size.py
M bin/inline_pom.py
M bin/load-data.py
M bin/parse-thrift-profile.py
M bin/run-workload.py
M bin/single_node_perf_run.py
M bin/start-impala-cluster.py
M docker/setup_build_context.py
M infra/python/bootstrap_virtualenv.py
M lib/python/impala_py_lib/gdb/impala-gdb.py
M lib/python/impala_py_lib/helpers.py
M lib/python/impala_py_lib/profiles.py
M testdata/bin/check-hbase-nodes.py
M testdata/bin/generate-schema-statements.py
M testdata/bin/generate-test-vectors.py
M testdata/bin/load-tpc-kudu.py
M testdata/bin/load_nested.py
M testdata/bin/random_avro_schema.py
M testdata/bin/rewrite-iceberg-metadata.py
M testdata/bin/wait-for-hiveserver2.py
M testdata/bin/wait-for-metastore.py
M testdata/common/cgroups.py
M testdata/common/text_delims_table.py
M testdata/common/widetable.py
M tests/authorization/test_authorization.py
M tests/authorization/test_authorized_proxy.py
M tests/authorization/test_provider.py
M tests/authorization/test_ranger.py
M tests/beeswax/impala_beeswax.py
M tests/benchmark/plugins/clear_buffer_cache.py
M tests/benchmark/plugins/vtune_plugin.py
M tests/benchmark/report_benchmark_results.py
M tests/catalog_service/test_catalog_service_client.py
M tests/catalog_service/test_large_num_partitions.py
M tests/common/base_test_suite.py
M tests/common/custom_cluster_test_suite.py
M tests/common/environ.py
M tests/common/file_utils.py
M tests/common/iceberg_test_suite.py
M tests/common/impala_cluster.py
M tests/common/impala_connection.py
M tests/common/impala_service.py
M tests/common/impala_test_suite.py
M tests/common/kudu_test_suite.py
M tests/common/network.py
M tests/common/parametrize.py
M tests/common/patterns.py
M tests/common/resource_pool_config.py
M tests/common/skip.py
M tests/common/test_dimensions.py
M tests/common/test_result_verifier.py
M tests/common/test_vector.py
M tests/comparison/cli_options.py
M tests/comparison/cluster.py
M tests/comparison/common.py
M tests/comparison/compat.py
M tests/comparison/data_generator.py
M tests/comparison/data_generator_mapred_common.py
M tests/comparison/db_connection.py
M tests/comparison/db_types.py
M tests/comparison/discrepancy_searcher.py
M tests/comparison/funcs.py
M tests/comparison/leopard/controller.py
M tests/comparison/leopard/front_end.py
M tests/comparison/leopard/impala_docker_env.py
M tests/comparison/leopard/job.py
M tests/comparison/leopard/report.py
M tests/comparison/leopard/schedule_item.py
M tests/comparison/model_translator.py
M tests/comparison/query.py
M tests/comparison/query_flattener.py
M tests/comparison/query_generator.py
M tests/comparison/query_profile.py
M tests/comparison/random_val_generator.py
M tests/comparison/statement_generator.py
M tests/comparison/tests/conftest.py
M tests/comparison/tests/fake_query.py
M tests/comparison/tests/hive/test_hive_create_agg_or_analytic_tree.py
M tests/comparison/tests/hive/test_hive_create_relational_join_condition.py
M tests/comparison/tests/query_object_testdata.py
M tests/comparison/tests/test_cluster.py
M tests/comparison/tests/test_cursor.py
M tests/comparison/tests/test_query_generator.py
M tests/comparison/tests/test_query_objects.py
M tests/comparison/tests/test_use_nested_with.py
M tests/comparison/util/verify-oracle-connection.py
M tests/conftest.py
M tests/custom_cluster/test_admission_controller.py
M tests/custom_cluster/test_alloc_fail.py
M tests/custom_cluster/test_always_false_filter.py
M tests/custom_cluster/test_auto_scaling.py
M tests/custom_cluster/test_automatic_invalidation.py
M tests/custom_cluster/test_blacklist.py
M tests/custom_cluster/test_blacklisted_dbs_and_tables.py
M tests/custom_cluster/test_breakpad.py
M tests/custom_cluster/test_catalog_hms_failures.py
M tests/custom_cluster/test_catalog_wait.py
M tests/custom_cluster/test_client_ssl.py
M tests/custom_cluster/test_codegen_cache.py
M tests/custom_cluster/test_compact_catalog_updates.py
M tests/custom_cluster/test_concurrent_ddls.py
M tests/custom_cluster/test_concurrent_kudu_create.py
M tests/custom_cluster/test_coordinators.py
M tests/custom_cluster/test_custom_hive_configs.py
M tests/custom_cluster/test_custom_statestore.py
M tests/custom_cluster/test_data_cache.py
M tests/custom_cluster/test_delegation.py
M tests/custom_cluster/test_disable_catalog_data_ops.py
M tests/custom_cluster/test_disable_features.py
M tests/custom_cluster/test_disk_spill_configurations.py
M tests/custom_cluster/test_events_custom_configs.py
M tests/custom_cluster/test_exchange_deferred_batches.py
M tests/custom_cluster/test_exchange_delays.py
M tests/custom_cluster/test_exchange_eos.py
M tests/custom_cluster/test_executor_groups.py
M tests/custom_cluster/test_frontend_connection_limit.py
M tests/custom_cluster/test_geospatial_library.py
M tests/custom_cluster/test_hbase_hms_column_order.py
M tests/custom_cluster/test_hdfs_fd_caching.py
M tests/custom_cluster/test_hdfs_timeout.py
M tests/custom_cluster/test_hedged_reads.py
M tests/custom_cluster/test_hive_parquet_codec_interop.py
M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py
M tests/custom_cluster/test_hive_text_codec_interop.py
M tests/custom_cluster/test_hs2.py
M tests/custom_cluster/test_hs2_fault_injection.py
M tests/custom_cluster/test_incremental_metadata_updates.py
M tests/custom_cluster/test_insert_behaviour.py
M tests/custom_cluster/test_jvm_mem_tracking.py
M tests/custom_cluster/test_krpc_mem_usage.py
M tests/custom_cluster/test_krpc_metrics.py
M tests/custom_cluster/test_krpc_options.py
M tests/custom_cluster/test_krpc_socket.py
M tests/custom_cluster/test_kudu.py
M tests/custom_cluster/test_kudu_not_available.py
M tests/custom_cluster/test_kudu_table_create_without_hms.py
M tests/custom_cluster/test_lineage.py
M tests/custom_cluster/test_local_catalog.py
M tests/custom_cluster/test_local_tz_conversion.py
M tests/custom_cluster/test_logging.py
M tests/custom_cluster/test_mem_reservations.py
M tests/custom_cluster/test_metadata_no_events_processing.py
M tests/custom_cluster/test_metadata_replicas.py
M tests/custom_cluster/test_metastore_events_cleanup.py
M tests/custom_cluster/test_metastore_service.py
M tests/custom_cluster/test_mt_dop.py
M tests/custom_cluster/test_observability.py
M tests/custom_cluster/test_parquet_max_page_header.py
M tests/custom_cluster/test_partition.py
M tests/custom_cluster/test_pause_monitor.py
M tests/custom_cluster/test_permanent_udfs.py
M tests/custom_cluster/test_preload_table_types.py
M tests/custom_cluster/test_process_failures.py
M tests/custom_cluster/test_query_concurrency.py
M tests/custom_cluster/test_query_event_hooks.py
M tests/custom_cluster/test_query_expiration.py
M tests/custom_cluster/test_query_retries.py
M tests/custom_cluster/test_re2_max_mem.py
M tests/custom_cluster/test_redaction.py
M tests/custom_cluster/test_reserved_words_version.py
M tests/custom_cluster/test_restart_services.py
M tests/custom_cluster/test_result_spooling.py
M tests/custom_cluster/test_rpc_exception.py
M tests/custom_cluster/test_rpc_timeout.py
M tests/custom_cluster/test_runtime_profile.py
M tests/custom_cluster/test_s3a_access.py
M tests/custom_cluster/test_saml2_sso.py
M tests/custom_cluster/test_scheduler_locality.py
M tests/custom_cluster/test_scratch_disk.py
M tests/custom_cluster/test_seq_file_filtering.py
M tests/custom_cluster/test_services_rpc_errors.py
M tests/custom_cluster/test_session_expiration.py
M tests/custom_cluster/test_set_and_unset.py
M tests/custom_cluster/test_shared_tzdb.py
M tests/custom_cluster/test_shell_commandline.py
M tests/custom_cluster/test_shell_interactive.py
M tests/custom_cluster/test_shell_interactive_reconnect.py
M tests/custom_cluster/test_startup_filesystem_checks.py
M tests/custom_cluster/test_stats_extrapolation.py
M tests/custom_cluster/test_thrift_debug_string_exception.py
M tests/custom_cluster/test_thrift_socket.py
M tests/custom_cluster/test_topic_update_frequency.py
M tests/custom_cluster/test_udf_concurrency.py
M tests/custom_cluster/test_web_pages.py
M tests/custom_cluster/test_wide_table_operations.py
M tests/data_errors/test_data_errors.py
M tests/experiments/test_targeted_perf.py
M tests/failure/test_failpoints.py
M tests/hs2/hs2_test_suite.py
M tests/hs2/test_fetch.py
M tests/hs2/test_fetch_first.py
M tests/hs2/test_fetch_timeout.py
M tests/hs2/test_hs2.py
M tests/hs2/test_json_endpoints.py
M tests/infra/test_perf_infra.py
M tests/infra/test_stress_infra.py
M tests/infra/test_utils.py
M tests/metadata/test_catalogd_debug_actions.py
M tests/metadata/test_compute_stats.py
M tests/metadata/test_ddl.py
M tests/metadata/test_ddl_base.py
M tests/metadata/test_event_processing.py
M tests/metadata/test_explain.py
M tests/metadata/test_hdfs_encryption.py
M tests/metadata/test_hdfs_permissions.py
M tests/metadata/test_hidden_files.py
M tests/metadata/test_hms_integration.py
M tests/metadata/test_last_ddl_time_update.py
M tests/metadata/test_load.py
M tests/metadata/test_metadata_query_statements.py
M tests/metadata/test_partition_metadata.py
M tests/metadata/test_recover_partitions.py
M tests/metadata/test_recursive_listing.py
M tests/metadata/test_refresh_partition.py
M tests/metadata/test_reset_metadata.py
M tests/metadata/test_reuse_partitions.py
M tests/metadata/test_set.py
M tests/metadata/test_show_create_table.py
M tests/metadata/test_stale_metadata.py
M tests/metadata/test_stats_extrapolation.py
M tests/metadata/test_testcase_builder.py
M tests/metadata/test_views_compatibility.py
M tests/observability/test_jvm_metrics.py
M tests/observability/test_log_fragments.py
M tests/observability/test_profile_tool.py
M tests/performance/query.py
M tests/performance/query_exec_functions.py
M tests/performance/query_executor.py
M tests/performance/scheduler.py
M tests/performance/workload.py
M tests/performance/workload_runner.py
M tests/query_test/test_acid.py
M tests/query_test/test_acid_row_validation.py
M tests/query_test/test_aggregation.py
M tests/query_test/test_analytic_tpcds.py
M tests/query_test/test_async_codegen.py
M tests/query_test/test_avro_schema_resolution.py
M tests/query_test/test_beeswax.py
M tests/query_test/test_cancellation.py
M tests/query_test/test_cast_with_format.py
M tests/query_test/test_chars.py
M tests/query_test/test_codegen.py
M tests/query_test/test_compressed_formats.py
M tests/query_test/test_datasketches.py
M tests/query_test/test_datastream_sender.py
M tests/query_test/test_date_queries.py
M tests/query_test/test_decimal_casting.py
M tests/query_test/test_decimal_fuzz.py
M tests/query_test/test_decimal_queries.py
M tests/query_test/test_delimited_text.py
M tests/query_test/test_errorlog.py
M tests/query_test/test_exprs.py
M tests/query_test/test_fetch.py
M tests/query_test/test_geospatial_functions.py
M tests/query_test/test_hash_join_timer.py
M tests/query_test/test_hbase_queries.py
M tests/query_test/test_hdfs_caching.py
M tests/query_test/test_hdfs_file_mods.py
M tests/query_test/test_iceberg.py
M tests/query_test/test_insert.py
M tests/query_test/test_insert_behaviour.py
M tests/query_test/test_insert_parquet.py
M tests/query_test/test_insert_permutation.py
M tests/query_test/test_invalid_test_header.py
M tests/query_test/test_io_metrics.py
M tests/query_test/test_join_queries.py
M tests/query_test/test_kudu.py
M tests/query_test/test_lifecycle.py
M tests/query_test/test_limit.py
M tests/query_test/test_limit_pushdown_analytic.py
M tests/query_test/test_local_fs.py
M tests/query_test/test_mem_usage_scaling.py
M tests/query_test/test_mt_dop.py
M tests/query_test/test_multiple_filesystems.py
M tests/query_test/test_nested_types.py
M tests/query_test/test_observability.py
M tests/query_test/test_orc_stats.py
M tests/query_test/test_parquet_bloom_filter.py
M tests/query_test/test_parquet_late_materialization.py
M tests/query_test/test_parquet_page_index.py
M tests/query_test/test_parquet_stats.py
M tests/query_test/test_partitioning.py
M tests/query_test/test_queries.py
M tests/query_test/test_query_compilation.py
M tests/query_test/test_query_mem_limit.py
M tests/query_test/test_query_opts.py
M tests/query_test/test_resource_limits.py
M tests/query_test/test_result_spooling.py
M tests/query_test/test_rows_availability.py
M tests/query_test/test_runtime_filters.py
M tests/query_test/test_scanners.py
M tests/query_test/test_scanners_fuzz.py
M tests/query_test/test_scratch_limit.py
M tests/query_test/test_sfs.py
M tests/query_test/test_sort.py
M tests/query_test/test_spilling.py
M tests/query_test/test_tablesample.py
M tests/query_test/test_tpcds_queries.py
M tests/query_test/test_tpch_nested_queries.py
M tests/query_test/test_tpch_queries.py
M tests/query_test/test_udfs.py
M tests/query_test/test_utf8_strings.py
M tests/run-tests.py
M tests/shell/test_cookie_util.py
M tests/shell/test_shell_client.py
M tests/shell/test_shell_commandline.py
M tests/shell/test_shell_interactive.py
M tests/shell/util.py
M tests/statestore/test_statestore.py
M tests/stress/concurrent_select.py
M tests/stress/extract_min_mem.py
M tests/stress/mem_broker.py
M tests/stress/queries.py
M tests/stress/query_retries_stress_runner.py
M tests/stress/query_runner.py
M tests/stress/runtime_info.py
M tests/stress/stress_util.py
M tests/stress/test_acid_stress.py
M tests/stress/test_ddl_stress.py
M tests/stress/test_insert_stress.py
M tests/stress/util.py
M tests/unittests/test_command.py
M tests/unittests/test_file_parser.py
M tests/unittests/test_result_verifier.py
M tests/util/acid_txn.py
M tests/util/adls_util.py
M tests/util/auto_scaler.py
M tests/util/calculation_util.py
M tests/util/cancel_util.py
M tests/util/cluster_controller.py
M tests/util/compute_table_stats.py
M tests/util/concurrent_workload.py
M tests/util/event_processor_utils.py
M tests/util/failpoints_util.py
M tests/util/filesystem_base.py
M tests/util/filesystem_utils.py
M tests/util/get_parquet_metadata.py
M tests/util/hdfs_util.py
M tests/util/iceberg_util.py
M tests/util/parse_util.py
M tests/util/plugin_runner.py
M tests/util/run_impyla_http_query.py
M tests/util/shell_util.py
M tests/util/ssh_util.py
M tests/util/test_file_parser.py
M tests/util/thrift_util.py
M tests/util/web_pages_util.py
M tests/verifiers/mem_usage_verifier.py
M tests/verifiers/metric_verifier.py
M tests/verifiers/test_verify_metrics.py
M tests/webserver/test_web_pages.py
354 files changed, 408 insertions(+), 113 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/19588/1
--
To view, visit http://gerrit.cloudera.org:8080/19588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Idb0fcbd11f3e8791f5951c4944be44fb580e576b
Gerrit-Change-Number: 19588
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell <joemcdonn...@cloudera.com>
