> On 一月 26, 2015, 10:30 p.m., Xuefu Zhang wrote: > > data/conf/spark/yarn-client/hive-site.xml, line 225 > > <https://reviews.apache.org/r/30264/diff/1/?file=834064#file834064line225> > > > > Only one executor? Maybe 2 will make it more general.
Yes, that make sense. On 一月 26, 2015, 10:30 p.m., chengxiang li wrote: > > I'm wondering why we have a new set of .out files? Every Test*CliDriver has its own output directory, i didn't think much about this previously. With your remind, i think, yes, we could share the golden files with TestSparkCliDriver, as it's golden file should be the same as TestMiniSparkOnYarnCliDriver for each qtest. One thing more to be note here is that, as spark.query.files contains more than 500 qtests, and it take long enough time to run a full unit test for hive now, i didn't enable all spark.query.files qtest for TestMiniSparkOnYarnCliDriver, instead, i enable qtests from minimr,query.files for it, which contains about 50 qtests, it takes about 10 minutes in my own desktop. - chengxiang ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30264/#review69685 ----------------------------------------------------------- On 一月 26, 2015, 6:37 a.m., chengxiang li wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30264/ > ----------------------------------------------------------- > > (Updated 一月 26, 2015, 6:37 a.m.) > > > Review request for hive, Szehon Ho and Xuefu Zhang. > > > Bugs: HIVE-9211 > https://issues.apache.org/jira/browse/HIVE-9211 > > > Repository: hive-git > > > Description > ------- > > MiniSparkOnYarnCluster is enabled for unit test, Spark is deployed on > miniYarnCluster on yarn-client mode, all qfiles in minimr.query.files are > enabled in this unit test except 3 qfile: bucket_num_reducers.q, > bucket_num_reducers2.q, udf_using.q, which is not supported in HoS. > > > Diffs > ----- > > data/conf/spark/hive-site.xml 016f568 > data/conf/spark/standalone/hive-site.xml PRE-CREATION > data/conf/spark/yarn-client/hive-site.xml PRE-CREATION > itests/pom.xml e1e88f6 > itests/qtest-spark/pom.xml d12fad5 > itests/src/test/resources/testconfiguration.properties f583aaf > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 095b9bd > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java > 41a2ab7 > > ql/src/test/results/clientpositive/miniSparkOnYarn/auto_sortmerge_join_16.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucket4.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucket5.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucket6.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/bucketizedhiveinputformat.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin6.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/bucketmapjoin7.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/constprog_partitioner.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/disable_merge_for_bucketing.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/empty_dir_in_table.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/external_table_with_space_in_location_path.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/file_with_header_footer.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/groupby1.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/groupby2.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/import_exported_table.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap3.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/index_bitmap_auto.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_bucketed_table.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_dyn_part.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_map_operators.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_merge.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_num_buckets.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/infer_bucket_sort_reducers_power_two.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/input16_cc.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/join1.q.out PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/leftsemijoin_mr.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/list_bucket_dml_10.q.java1.7.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/load_fs2.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/load_hdfs_file_with_space_in_the_name.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/parallel_orderby.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_1.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/ql_rewrite_gbtoidx_cbo_2.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/quotedid_smb.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/reduce_deduplicate.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/remote_script.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/root_dir_external_table.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/schemeAuthority2.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/scriptfile1.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/smb_mapjoin_8.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/stats_counter_partitioned.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/temp_table_external.q.out > PRE-CREATION > > ql/src/test/results/clientpositive/miniSparkOnYarn/truncate_column_buckets.q.out > PRE-CREATION > ql/src/test/results/clientpositive/miniSparkOnYarn/uber_reduce.q.out > PRE-CREATION > shims/0.20S/src/main/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java > b17f465 > shims/0.23/src/main/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java > a61c3ac > > shims/0.23/src/main/java/org/apache/hadoop/hive/shims/MiniSparkOnYARNCluster.java > PRE-CREATION > shims/common/src/main/java/org/apache/hadoop/hive/shims/HadoopShims.java > 064304c > > spark-client/src/main/java/org/apache/hive/spark/client/SparkClientImpl.java > aea90db > > Diff: https://reviews.apache.org/r/30264/diff/ > > > Testing > ------- > > > Thanks, > > chengxiang li > >
