Re: Review Request 28017: HIVE-8776 - Generate MapredLocalWork in SparkMapJoinResolver [Spark Brach]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28017/#review61373 --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java https://reviews.apache.org/r/28017/#comment102946 currentTask seems to be the container for sparkWork. Do we need to pass in both of them? BTW, currentTask seems to be a misleading varaible name. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java https://reviews.apache.org/r/28017/#comment102947 Can we have a local variable for sparkWork.getAllWorkUnsorted()? It's called again later in the code. - Xuefu Zhang On Nov. 14, 2014, 12:03 a.m., Chao Sun wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28017/ --- (Updated Nov. 14, 2014, 12:03 a.m.) Review request for hive, Jimmy Xiang, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-8776 https://issues.apache.org/jira/browse/HIVE-8776 Repository: hive-git Description --- In SparkMapJoinResolver, we need to populate MapredLocalWork for all MapWorks with MapJoinOperator. It is needed later in HashTableLoader, for example, to retrieve small hash tables and direct fetch tables. We need to set up information, such as aliasToWork, aliasToFetchWork, directFetchOp, inputFileChangeSensitive, tmpPath, etc., for the new local works. Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java d30ae51 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java 4b9a6cb ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 785e4a0 Diff: https://reviews.apache.org/r/28017/diff/ Testing --- Thanks, Chao Sun
Re: Review Request 28017: HIVE-8776 - Generate MapredLocalWork in SparkMapJoinResolver [Spark Brach]
On Nov. 14, 2014, 1:53 a.m., Xuefu Zhang wrote: ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java, line 138 https://reviews.apache.org/r/28017/diff/1/?file=763012#file763012line138 currentTask seems to be the container for sparkWork. Do we need to pass in both of them? BTW, currentTask seems to be a misleading varaible name. Good point. I always forget this.. Changed the name to originalTask. - Chao --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28017/#review61373 --- On Nov. 14, 2014, 12:03 a.m., Chao Sun wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28017/ --- (Updated Nov. 14, 2014, 12:03 a.m.) Review request for hive, Jimmy Xiang, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-8776 https://issues.apache.org/jira/browse/HIVE-8776 Repository: hive-git Description --- In SparkMapJoinResolver, we need to populate MapredLocalWork for all MapWorks with MapJoinOperator. It is needed later in HashTableLoader, for example, to retrieve small hash tables and direct fetch tables. We need to set up information, such as aliasToWork, aliasToFetchWork, directFetchOp, inputFileChangeSensitive, tmpPath, etc., for the new local works. Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java d30ae51 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java 4b9a6cb ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 785e4a0 Diff: https://reviews.apache.org/r/28017/diff/ Testing --- Thanks, Chao Sun
Re: Review Request 28017: HIVE-8776 - Generate MapredLocalWork in SparkMapJoinResolver [Spark Brach]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28017/ --- (Updated Nov. 14, 2014, 2:43 a.m.) Review request for hive, Jimmy Xiang, Szehon Ho, and Xuefu Zhang. Changes --- Thanks Xuefu for the comments! Bugs: HIVE-8776 https://issues.apache.org/jira/browse/HIVE-8776 Repository: hive-git Description --- In SparkMapJoinResolver, we need to populate MapredLocalWork for all MapWorks with MapJoinOperator. It is needed later in HashTableLoader, for example, to retrieve small hash tables and direct fetch tables. We need to set up information, such as aliasToWork, aliasToFetchWork, directFetchOp, inputFileChangeSensitive, tmpPath, etc., for the new local works. Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java d30ae51 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java 9ce1a18 ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 785e4a0 Diff: https://reviews.apache.org/r/28017/diff/ Testing --- Thanks, Chao Sun
Re: Review Request 28017: HIVE-8776 - Generate MapredLocalWork in SparkMapJoinResolver [Spark Brach]
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28017/#review61393 --- Ship it! Ship It! - Xuefu Zhang On Nov. 14, 2014, 2:43 a.m., Chao Sun wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28017/ --- (Updated Nov. 14, 2014, 2:43 a.m.) Review request for hive, Jimmy Xiang, Szehon Ho, and Xuefu Zhang. Bugs: HIVE-8776 https://issues.apache.org/jira/browse/HIVE-8776 Repository: hive-git Description --- In SparkMapJoinResolver, we need to populate MapredLocalWork for all MapWorks with MapJoinOperator. It is needed later in HashTableLoader, for example, to retrieve small hash tables and direct fetch tables. We need to set up information, such as aliasToWork, aliasToFetchWork, directFetchOp, inputFileChangeSensitive, tmpPath, etc., for the new local works. Diffs - ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HashTableLoader.java d30ae51 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java 9ce1a18 ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 785e4a0 Diff: https://reviews.apache.org/r/28017/diff/ Testing --- Thanks, Chao Sun
