[
https://issues.apache.org/jira/browse/SPARK-12978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16610729#comment-16610729
]
Wenchen Fan commented on SPARK-12978:
-------------------------------------
Sorry we missed this issue. I think this is a very complicated problem now and
we need to write a design doc for it. Possible approaches:
1. introduce logical partial aggregate
2. treat partial aggregate as a physical optimization. We need to think about
how it interacts with EnsureRequirements
> Skip unnecessary final group-by when input data already clustered with
> group-by keys
> ------------------------------------------------------------------------------------
>
> Key: SPARK-12978
> URL: https://issues.apache.org/jira/browse/SPARK-12978
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Takeshi Yamamuro
> Assignee: Takeshi Yamamuro
> Priority: Major
>
> This ticket targets the optimization to skip an unnecessary group-by
> operation below;
> Without opt.:
> {code}
> == Physical Plan ==
> TungstenAggregate(key=[col0#159],
> functions=[(sum(col1#160),mode=Final,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)],
> output=[col0#159,sum(col1)#177,avg(col2)#178])
> +- TungstenAggregate(key=[col0#159],
> functions=[(sum(col1#160),mode=Partial,isDistinct=false),(avg(col2#161),mode=Partial,isDistinct=false)],
> output=[col0#159,sum#200,sum#201,count#202L])
> +- TungstenExchange hashpartitioning(col0#159,200), None
> +- InMemoryColumnarTableScan [col0#159,col1#160,col2#161],
> InMemoryRelation [col0#159,col1#160,col2#161], true, 10000,
> StorageLevel(true, true, false, true, 1), ConvertToUnsafe, None
> {code}
> With opt.:
> {code}
> == Physical Plan ==
> TungstenAggregate(key=[col0#159],
> functions=[(sum(col1#160),mode=Complete,isDistinct=false),(avg(col2#161),mode=Final,isDistinct=false)],
> output=[col0#159,sum(col1)#177,avg(col2)#178])
> +- TungstenExchange hashpartitioning(col0#159,200), None
> +- InMemoryColumnarTableScan [col0#159,col1#160,col2#161], InMemoryRelation
> [col0#159,col1#160,col2#161], true, 10000, StorageLevel(true, true, false,
> true, 1), ConvertToUnsafe, None
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
