[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API
[
https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110877#comment-16110877
]
Liang-Chi Hsieh commented on SPARK-21591:
-
[~yanboliang] Thanks for linking to the related JIRA. Yeah, I was first
thinking about {{TypedImperativeAggregate}} for the initial implementation for
such aggregation. I basically have the same concern brought by others there.
But I agree this still a good direction to follow in the future.
> Implement treeAggregate on Dataset API
> --
>
> Key: SPARK-21591
> URL: https://issues.apache.org/jira/browse/SPARK-21591
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory
> and CPU for Spark application. However, in MLlib we still not migrate the
> internal computing workload from {{RDD}} to {{DataFrame}}.
> There are lots of blocking issues for the migration, lack of
> {{treeAggregate}} on {{DataFrame}} is one of them. {{treeAggregate}} is very
> important for MLlib algorithms, since they do aggregate on {{Vector}} which
> may has millions of elements. As we all know, {{RDD}} based {{treeAggregate}}
> reduces the aggregation time by an order of magnitude for lots of MLlib
> algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}}
> API and do the performance benchmark related issues. And I think other
> scenarios except for MLlib will also benefit from this improvement if we get
> it done.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API
[
https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110869#comment-16110869
]
Yanbo Liang commented on SPARK-21591:
-
[~viirya] I agree there are lots of performance bottlenecks, such as
serialization/deserialization cost between {{UnsafeRow}} and JVM object, reduce
data copy between different format if applicable, etc. There are discussion
about the bottlenecks at SPARK-19634 and corresponding PR. This JIRA is just
used to track ```treeAggregate``` related issue, and it only has a significant
impact when we handle vector of large dimension. Thanks.
> Implement treeAggregate on Dataset API
> --
>
> Key: SPARK-21591
> URL: https://issues.apache.org/jira/browse/SPARK-21591
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory
> and CPU for Spark application. However, in MLlib we still not migrate the
> internal computing workload from {{RDD}} to {{DataFrame}}.
> There are lots of blocking issues for the migration, lack of
> {{treeAggregate}} on {{DataFrame}} is one of them. {{treeAggregate}} is very
> important for MLlib algorithms, since they do aggregate on {{Vector}} which
> may has millions of elements. As we all know, {{RDD}} based {{treeAggregate}}
> reduces the aggregation time by an order of magnitude for lots of MLlib
> algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}}
> API and do the performance benchmark related issues. And I think other
> scenarios except for MLlib will also benefit from this improvement if we get
> it done.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API
[
https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110226#comment-16110226
]
Liang-Chi Hsieh commented on SPARK-21591:
-
IIUC, basically the aggregation in SparkSQL doesn't follow the tree pattern of
{{treeAggregate}}.
Btw, because we will use arbitrary user object, e.g.,
{{MultivariateOnlineSummarizer}} as aggregate buffer, the extra serialization
cost might be a burden on the {{treeAggregate}} of {{Dataset}}.
Maybe others can have more insights to share.
> Implement treeAggregate on Dataset API
> --
>
> Key: SPARK-21591
> URL: https://issues.apache.org/jira/browse/SPARK-21591
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory
> and CPU for Spark application. However, in MLlib we still not migrate the
> internal computing workload from {{RDD}} to {{DataFrame}}.
> There are lots of blocking issues for the migration, lack of
> {{treeAggregate}} on {{DataFrame}} is one of them. {{treeAggregate}} is very
> important for MLlib algorithms, since they do aggregate on {{Vector}} which
> may has millions of elements. As we all know, {{RDD}} based {{treeAggregate}}
> reduces the aggregation time by an order of magnitude for lots of MLlib
> algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}}
> API and do the performance benchmark related issues. And I think other
> scenarios except for MLlib will also benefit from this improvement if we get
> it done.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API
[
https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110196#comment-16110196
]
Yanbo Liang commented on SPARK-21591:
-
[~viirya] Yep, this is the way we are using, but we want to enjoy Tungsten
execution engine. :)
> Implement treeAggregate on Dataset API
> --
>
> Key: SPARK-21591
> URL: https://issues.apache.org/jira/browse/SPARK-21591
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory
> and CPU for Spark application. However, in MLlib we still not migrate the
> internal computing workload from {{RDD}} to {{DataFrame}}.
> There are lots of blocking issues for the migration, lack of
> {{treeAggregate}} on {{DataFrame}} is one of them. {{treeAggregate}} is very
> important for MLlib algorithms, since they do aggregate on {{Vector}} which
> may has millions of elements. As we all know, {{RDD}} based {{treeAggregate}}
> reduces the aggregation time by an order of magnitude for lots of MLlib
> algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}}
> API and do the performance benchmark related issues. And I think other
> scenarios except for MLlib will also benefit from this improvement if we get
> it done.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API
[
https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110169#comment-16110169
]
Liang-Chi Hsieh commented on SPARK-21591:
-
The most straightforward way is similar to Dataset.foreach, i.e., do
treeAggregate on the rdd of Dataset.
> Implement treeAggregate on Dataset API
> --
>
> Key: SPARK-21591
> URL: https://issues.apache.org/jira/browse/SPARK-21591
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory
> and CPU for Spark application. However, in MLlib we still not migrate the
> internal computing workload from {{RDD}} to {{DataFrame}}.
> There are lots of blocking issues for the migration, lack of
> {{treeAggregate}} on {{DataFrame}} is one of them. {{treeAggregate}} is very
> important for MLlib algorithms, since they do aggregate on {{Vector}} which
> may has millions of elements. As we all know, {{RDD}} based {{treeAggregate}}
> reduces the aggregation time by an order of magnitude for lots of MLlib
> algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}}
> API and do the performance benchmark related issues. And I think other
> scenarios except for MLlib will also benefit from this improvement if we get
> it done.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API
[
https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108644#comment-16108644
]
Kazuaki Ishizaki commented on SPARK-21591:
--
I like this idea
> Implement treeAggregate on Dataset API
> --
>
> Key: SPARK-21591
> URL: https://issues.apache.org/jira/browse/SPARK-21591
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory
> and CPU for Spark application. However, in MLlib we still not migrate the
> internal computing workload from {{RDD}} to {{DataFrame}}.
> One of the block issue is there is no {{treeAggregate}} on {{DataFrame}}.
> It's very important for MLlib algorithms, since they do aggregate on
> {{Vector}} which may has millions of elements. As we all know, {{RDD}} based
> {{treeAggregate}} reduces the aggregation time by an order of magnitude for
> lots of MLlib
> algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}}
> API and do the performance benchmark related issues. And I think other
> scenarios except for MLlib will also benefit from this improvement if we get
> it done.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21591) Implement treeAggregate on Dataset API
[
https://issues.apache.org/jira/browse/SPARK-21591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16108476#comment-16108476
]
Yanbo Liang commented on SPARK-21591:
-
cc [~cloud_fan]
> Implement treeAggregate on Dataset API
> --
>
> Key: SPARK-21591
> URL: https://issues.apache.org/jira/browse/SPARK-21591
> Project: Spark
> Issue Type: Brainstorming
> Components: SQL
>Affects Versions: 2.2.0
>Reporter: Yanbo Liang
>
> The Tungsten execution engine substantially improved the efficiency of memory
> and CPU for Spark application. However, in MLlib we still not migrate the
> internal computing workload from {{RDD}} to {{DataFrame}}.
> The main block issue is there is no {{treeAggregate}} on {{DataFrame}}. As we
> all know, {{RDD}} based {{treeAggregate}} reduces the aggregation time by an
> order of magnitude for lots of MLlib
> algorithms(https://databricks.com/blog/2014/09/22/spark-1-1-mllib-performance-improvements.html).
> I open this JIRA to discuss to implement {{treeAggregate}} on {{DataFrame}}
> API and do the performance benchmark related issues.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
