> On Nov. 8, 2014, 3:15 p.m., Xuefu Zhang wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java, > > line 214 > > <https://reviews.apache.org/r/27627/diff/3/?file=754597#file754597line214> > > > > This assumes that result SparkWorks will be linearly dependent on each > > other, which isn't true in general.Let's say the are two works (w1 and w2), > > each having a map join operator. w1 and w2 are connected to w3 via HTS. w3 > > also contains map join operator. Dependency in this scenario will be > > graphic rather than linear. > > Chao Sun wrote: > I was thinking, in this case, if there's no dependency between w1 and w2, > they can be put in the same SparkWork, right? > Otherwise, they will form a linear dependency too.
w1 and w2 are fine. they will be in the same SparkWork. This SparkWork will depends on both the SparkWork generated at w1 and SparkWork generated at w2. This dependency is not linear. To put more details, for each work that has map join op, we need to create a SparkWork to handle its small tables. So, both w1 and w2 will need to create such SparkWork. While w1 and w2 are in the same SparkWork, this SparkWork depends on the two SparkWorks created. - Xuefu ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27627/#review60482 ----------------------------------------------------------- On Nov. 7, 2014, 6:07 p.m., Chao Sun wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/27627/ > ----------------------------------------------------------- > > (Updated Nov. 7, 2014, 6:07 p.m.) > > > Review request for hive. > > > Bugs: HIVE-8622 > https://issues.apache.org/jira/browse/HIVE-8622 > > > Repository: hive-git > > > Description > ------- > > This is a sub-task of map-join for spark > https://issues.apache.org/jira/browse/HIVE-7613 > This can use the baseline patch for map-join > https://issues.apache.org/jira/browse/HIVE-8616 > > > Diffs > ----- > > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/plan/SparkWork.java 66fd6b6 > > Diff: https://reviews.apache.org/r/27627/diff/ > > > Testing > ------- > > > Thanks, > > Chao Sun > >
