----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61716/ -----------------------------------------------------------
Review request for hive, Rui Li and Xuefu Zhang. Bugs: HIVE-17292 https://issues.apache.org/jira/browse/HIVE-17292 Repository: hive-git Description ------- Currently MiniSparkShim will be able to provide only 1 executor, since the FairScheduler is configured, and the allocation increment is not set so the default 1024 is used. So only 1 executor could fit into the memory restrictions. Changing Hadoop23Shims.java, so the MiniSparkShim will able to provide the requested 2 executors. Changing QTestUtil.setSparkSession, so we will wait until every executor is available, not only the 1st. Changing SparkSessionImpl.getMemoryAndCores, so we use the client provided paralellism in case of local spark.master too. Regenerating golden files (numReducers, and number of files changed in the explain plans) The change contains 2 golden file changes (spark_dynamic_partition_pruning_mapjoin_only.q.out, spark_dynamic_partition_pruning.q.out) which are containing other neccessary changes for a green run, so this patch should be regenerated after their corresponding jiras are solved (HIVE-17347, HIVE-17346) Diffs ----- data/conf/spark/standalone/hive-site.xml 989e65e itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 825f826 ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java 8224ef9 ql/src/test/results/clientpositive/spark/constprog_semijoin.q.out be33189 ql/src/test/results/clientpositive/spark/index_bitmap3.q.out 31c921c ql/src/test/results/clientpositive/spark/index_bitmap_auto.q.out e3b205c ql/src/test/results/clientpositive/spark/infer_bucket_sort_map_operators.q.out 356f625 ql/src/test/results/clientpositive/spark/infer_bucket_sort_reducers_power_two.q.out bf010e7 ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning.q.out d9da762 ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_2.q.out 8a794e7 ql/src/test/results/clientpositive/spark/spark_dynamic_partition_pruning_mapjoin_only.q.out 564a6a2 ql/src/test/results/clientpositive/spark/spark_explain_groupbyshuffle.q.out 9d162c9 ql/src/test/results/clientpositive/spark/spark_use_ts_stats_for_mapjoin.q.out f92356e shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java ae6f542 Diff: https://reviews.apache.org/r/61716/diff/1/ Testing ------- Thanks, Peter Vary