[jira] [Updated] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works
[ https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17178: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks [~stakiar] for the review. > Spark Partition Pruning Sink Operator can't target multiple Works > - > > Key: HIVE-17178 > URL: https://issues.apache.org/jira/browse/HIVE-17178 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Rui Li >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-17178.1.patch, HIVE-17178.2.patch, > HIVE-17178.3.patch, HIVE-17178.4.patch, HIVE-17178.5.patch, HIVE-17178.6.patch > > > A Spark Partition Pruning Sink Operator cannot be used to target multiple Map > Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated > if a single table needs to be used to target multiple Map Works. > The following query shows the issue: > {code} > set hive.spark.dynamic.partition.pruning=true; > set hive.auto.convert.join=true; > create table part_table_1 (col int) partitioned by (part_col int); > create table part_table_2 (col int) partitioned by (part_col int); > create table regular_table (col int); > insert into table regular_table values (1); > alter table part_table_1 add partition (part_col=1); > insert into table part_table_1 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_1 add partition (part_col=2); > insert into table part_table_1 partition (part_col=2) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=1); > insert into table part_table_2 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=2); > insert into table part_table_2 partition (part_col=2) values (1), (2), (3), > (4); > explain select * from regular_table, part_table_1, part_table_2 where > regular_table.col = part_table_1.part_col and regular_table.col = > part_table_2.part_col; > {code} > The explain plan is > {code} > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > A masked pattern was here > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: regular_table > Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE > Column stats: NONE > Filter Operator > predicate: col is not null (type: boolean) > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: col (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col0 (type: int) > 1 _col1 (type: int) > 2 _col1 (type: int) > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 2 > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part
[jira] [Updated] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works
[ https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17178: -- Attachment: HIVE-17178.6.patch > Spark Partition Pruning Sink Operator can't target multiple Works > - > > Key: HIVE-17178 > URL: https://issues.apache.org/jira/browse/HIVE-17178 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Rui Li >Priority: Major > Attachments: HIVE-17178.1.patch, HIVE-17178.2.patch, > HIVE-17178.3.patch, HIVE-17178.4.patch, HIVE-17178.5.patch, HIVE-17178.6.patch > > > A Spark Partition Pruning Sink Operator cannot be used to target multiple Map > Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated > if a single table needs to be used to target multiple Map Works. > The following query shows the issue: > {code} > set hive.spark.dynamic.partition.pruning=true; > set hive.auto.convert.join=true; > create table part_table_1 (col int) partitioned by (part_col int); > create table part_table_2 (col int) partitioned by (part_col int); > create table regular_table (col int); > insert into table regular_table values (1); > alter table part_table_1 add partition (part_col=1); > insert into table part_table_1 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_1 add partition (part_col=2); > insert into table part_table_1 partition (part_col=2) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=1); > insert into table part_table_2 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=2); > insert into table part_table_2 partition (part_col=2) values (1), (2), (3), > (4); > explain select * from regular_table, part_table_1, part_table_2 where > regular_table.col = part_table_1.part_col and regular_table.col = > part_table_2.part_col; > {code} > The explain plan is > {code} > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > A masked pattern was here > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: regular_table > Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE > Column stats: NONE > Filter Operator > predicate: col is not null (type: boolean) > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: col (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col0 (type: int) > 1 _col1 (type: int) > 2 _col1 (type: int) > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 2 > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > targe
[jira] [Updated] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works
[ https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17178: -- Attachment: HIVE-17178.5.patch > Spark Partition Pruning Sink Operator can't target multiple Works > - > > Key: HIVE-17178 > URL: https://issues.apache.org/jira/browse/HIVE-17178 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Rui Li >Priority: Major > Attachments: HIVE-17178.1.patch, HIVE-17178.2.patch, > HIVE-17178.3.patch, HIVE-17178.4.patch, HIVE-17178.5.patch > > > A Spark Partition Pruning Sink Operator cannot be used to target multiple Map > Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated > if a single table needs to be used to target multiple Map Works. > The following query shows the issue: > {code} > set hive.spark.dynamic.partition.pruning=true; > set hive.auto.convert.join=true; > create table part_table_1 (col int) partitioned by (part_col int); > create table part_table_2 (col int) partitioned by (part_col int); > create table regular_table (col int); > insert into table regular_table values (1); > alter table part_table_1 add partition (part_col=1); > insert into table part_table_1 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_1 add partition (part_col=2); > insert into table part_table_1 partition (part_col=2) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=1); > insert into table part_table_2 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=2); > insert into table part_table_2 partition (part_col=2) values (1), (2), (3), > (4); > explain select * from regular_table, part_table_1, part_table_2 where > regular_table.col = part_table_1.part_col and regular_table.col = > part_table_2.part_col; > {code} > The explain plan is > {code} > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > A masked pattern was here > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: regular_table > Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE > Column stats: NONE > Filter Operator > predicate: col is not null (type: boolean) > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: col (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col0 (type: int) > 1 _col1 (type: int) > 2 _col1 (type: int) > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 2 > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_
[jira] [Updated] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works
[ https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17178: -- Attachment: HIVE-17178.4.patch > Spark Partition Pruning Sink Operator can't target multiple Works > - > > Key: HIVE-17178 > URL: https://issues.apache.org/jira/browse/HIVE-17178 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Rui Li >Priority: Major > Attachments: HIVE-17178.1.patch, HIVE-17178.2.patch, > HIVE-17178.3.patch, HIVE-17178.4.patch > > > A Spark Partition Pruning Sink Operator cannot be used to target multiple Map > Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated > if a single table needs to be used to target multiple Map Works. > The following query shows the issue: > {code} > set hive.spark.dynamic.partition.pruning=true; > set hive.auto.convert.join=true; > create table part_table_1 (col int) partitioned by (part_col int); > create table part_table_2 (col int) partitioned by (part_col int); > create table regular_table (col int); > insert into table regular_table values (1); > alter table part_table_1 add partition (part_col=1); > insert into table part_table_1 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_1 add partition (part_col=2); > insert into table part_table_1 partition (part_col=2) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=1); > insert into table part_table_2 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=2); > insert into table part_table_2 partition (part_col=2) values (1), (2), (3), > (4); > explain select * from regular_table, part_table_1, part_table_2 where > regular_table.col = part_table_1.part_col and regular_table.col = > part_table_2.part_col; > {code} > The explain plan is > {code} > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > A masked pattern was here > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: regular_table > Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE > Column stats: NONE > Filter Operator > predicate: col is not null (type: boolean) > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: col (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col0 (type: int) > 1 _col1 (type: int) > 2 _col1 (type: int) > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 2 > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col >
[jira] [Updated] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works
[ https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17178: -- Attachment: HIVE-17178.3.patch > Spark Partition Pruning Sink Operator can't target multiple Works > - > > Key: HIVE-17178 > URL: https://issues.apache.org/jira/browse/HIVE-17178 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Rui Li >Priority: Major > Attachments: HIVE-17178.1.patch, HIVE-17178.2.patch, > HIVE-17178.3.patch > > > A Spark Partition Pruning Sink Operator cannot be used to target multiple Map > Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated > if a single table needs to be used to target multiple Map Works. > The following query shows the issue: > {code} > set hive.spark.dynamic.partition.pruning=true; > set hive.auto.convert.join=true; > create table part_table_1 (col int) partitioned by (part_col int); > create table part_table_2 (col int) partitioned by (part_col int); > create table regular_table (col int); > insert into table regular_table values (1); > alter table part_table_1 add partition (part_col=1); > insert into table part_table_1 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_1 add partition (part_col=2); > insert into table part_table_1 partition (part_col=2) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=1); > insert into table part_table_2 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=2); > insert into table part_table_2 partition (part_col=2) values (1), (2), (3), > (4); > explain select * from regular_table, part_table_1, part_table_2 where > regular_table.col = part_table_1.part_col and regular_table.col = > part_table_2.part_col; > {code} > The explain plan is > {code} > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > A masked pattern was here > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: regular_table > Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE > Column stats: NONE > Filter Operator > predicate: col is not null (type: boolean) > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: col (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col0 (type: int) > 1 _col1 (type: int) > 2 _col1 (type: int) > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 2 > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target
[jira] [Updated] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works
[ https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17178: -- Attachment: HIVE-17178.2.patch > Spark Partition Pruning Sink Operator can't target multiple Works > - > > Key: HIVE-17178 > URL: https://issues.apache.org/jira/browse/HIVE-17178 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Rui Li >Priority: Major > Attachments: HIVE-17178.1.patch, HIVE-17178.2.patch > > > A Spark Partition Pruning Sink Operator cannot be used to target multiple Map > Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated > if a single table needs to be used to target multiple Map Works. > The following query shows the issue: > {code} > set hive.spark.dynamic.partition.pruning=true; > set hive.auto.convert.join=true; > create table part_table_1 (col int) partitioned by (part_col int); > create table part_table_2 (col int) partitioned by (part_col int); > create table regular_table (col int); > insert into table regular_table values (1); > alter table part_table_1 add partition (part_col=1); > insert into table part_table_1 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_1 add partition (part_col=2); > insert into table part_table_1 partition (part_col=2) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=1); > insert into table part_table_2 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=2); > insert into table part_table_2 partition (part_col=2) values (1), (2), (3), > (4); > explain select * from regular_table, part_table_1, part_table_2 where > regular_table.col = part_table_1.part_col and regular_table.col = > part_table_2.part_col; > {code} > The explain plan is > {code} > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > A masked pattern was here > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: regular_table > Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE > Column stats: NONE > Filter Operator > predicate: col is not null (type: boolean) > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: col (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col0 (type: int) > 1 _col1 (type: int) > 2 _col1 (type: int) > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 2 > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 3 >
[jira] [Updated] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works
[ https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17178: -- Status: Patch Available (was: Open) > Spark Partition Pruning Sink Operator can't target multiple Works > - > > Key: HIVE-17178 > URL: https://issues.apache.org/jira/browse/HIVE-17178 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Rui Li >Priority: Major > Attachments: HIVE-17178.1.patch > > > A Spark Partition Pruning Sink Operator cannot be used to target multiple Map > Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated > if a single table needs to be used to target multiple Map Works. > The following query shows the issue: > {code} > set hive.spark.dynamic.partition.pruning=true; > set hive.auto.convert.join=true; > create table part_table_1 (col int) partitioned by (part_col int); > create table part_table_2 (col int) partitioned by (part_col int); > create table regular_table (col int); > insert into table regular_table values (1); > alter table part_table_1 add partition (part_col=1); > insert into table part_table_1 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_1 add partition (part_col=2); > insert into table part_table_1 partition (part_col=2) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=1); > insert into table part_table_2 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=2); > insert into table part_table_2 partition (part_col=2) values (1), (2), (3), > (4); > explain select * from regular_table, part_table_1, part_table_2 where > regular_table.col = part_table_1.part_col and regular_table.col = > part_table_2.part_col; > {code} > The explain plan is > {code} > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > A masked pattern was here > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: regular_table > Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE > Column stats: NONE > Filter Operator > predicate: col is not null (type: boolean) > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: col (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col0 (type: int) > 1 _col1 (type: int) > 2 _col1 (type: int) > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 2 > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 3 > Local Work
[jira] [Updated] (HIVE-17178) Spark Partition Pruning Sink Operator can't target multiple Works
[ https://issues.apache.org/jira/browse/HIVE-17178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17178: -- Attachment: HIVE-17178.1.patch > Spark Partition Pruning Sink Operator can't target multiple Works > - > > Key: HIVE-17178 > URL: https://issues.apache.org/jira/browse/HIVE-17178 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Sahil Takiar >Assignee: Rui Li >Priority: Major > Attachments: HIVE-17178.1.patch > > > A Spark Partition Pruning Sink Operator cannot be used to target multiple Map > Work objects. The entire DPP subtree (SEL-GBY-SPARKPRUNINGSINK) is duplicated > if a single table needs to be used to target multiple Map Works. > The following query shows the issue: > {code} > set hive.spark.dynamic.partition.pruning=true; > set hive.auto.convert.join=true; > create table part_table_1 (col int) partitioned by (part_col int); > create table part_table_2 (col int) partitioned by (part_col int); > create table regular_table (col int); > insert into table regular_table values (1); > alter table part_table_1 add partition (part_col=1); > insert into table part_table_1 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_1 add partition (part_col=2); > insert into table part_table_1 partition (part_col=2) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=1); > insert into table part_table_2 partition (part_col=1) values (1), (2), (3), > (4); > alter table part_table_2 add partition (part_col=2); > insert into table part_table_2 partition (part_col=2) values (1), (2), (3), > (4); > explain select * from regular_table, part_table_1, part_table_2 where > regular_table.col = part_table_1.part_col and regular_table.col = > part_table_2.part_col; > {code} > The explain plan is > {code} > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > A masked pattern was here > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: regular_table > Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE > Column stats: NONE > Filter Operator > predicate: col is not null (type: boolean) > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Select Operator > expressions: col (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col0 (type: int) > 1 _col1 (type: int) > 2 _col1 (type: int) > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 2 > Select Operator > expressions: _col0 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > partition key expr: part_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target column name: part_col > target work: Map 3 > Local Work: >