[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-08-31 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13802 yea, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if th

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-08-31 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/13802 @maropu could you close this one? It is not that relevant anymore. Thanks for working on it though! --- If your project is set up for it, you can reply to this email and have your reply appear o

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-22 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13802 okay --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featur

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-22 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/13802 Could you have a go? Would be great! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature ena

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-22 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13802 No, I'd like to fix incorrect comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enab

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-22 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/13802 What do you want to fix? WindowAggregateFunctions? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have th

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-22 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13802 Is it okay to make a new pr to fix these? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature en

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-22 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13802 Thanks for your explanation! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wis

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-22 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/13802 I think we should rename and document `supportsPartial` to reflect what it actually does. `Rank` and `RowNumer` are window functions. They both rely on ordered evaluation, and they shoul

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-22 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13802 As for `supportPartial`, I could understand that `collect` and `hive_udaf` has such a limitation though, how about `AggregateWindowFunction`? It seems these functions `RowNumber` and `Rank` work

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-22 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13802 @hvanhovell As for `UnsafeMapData`, could you check #13847? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not h

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/13802 @maropu all aggregates that current set `supportsPartial = false` cannot be partially aggregated and require that the entire group is processed in one step. So the name is a bit misleading. I sup

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/13802 @hvanhovell oh, I see. okay, I'll check we can implement mutable `ArrayData` and `MapData`. btw, I have some question; 1. Any reason to use `SortAggregateExec` for all the non-partial aggreg

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/13802 @maropu this won't work for other `hive_udf`s since these also maintain internal state, and currently require per group processing. This also has a greater potential of creating out-of-memory err

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13802 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60933/ Test PASSed. ---

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13802 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13802 **[Test build #60933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60933/consoleFull)** for PR 13802 at commit [`88ba697`](https://github.com/apache/spark/commit/

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13802 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13802 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60928/ Test PASSed. ---

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13802 **[Test build #60928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60928/consoleFull)** for PR 13802 at commit [`0506453`](https://github.com/apache/spark/commit/

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13802 **[Test build #60933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60933/consoleFull)** for PR 13802 at commit [`88ba697`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13802 **[Test build #60928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60928/consoleFull)** for PR 13802 at commit [`0506453`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13802 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13802 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/60919/ Test FAILed. ---

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13802 **[Test build #60919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60919/consoleFull)** for PR 13802 at commit [`517d7ea`](https://github.com/apache/spark/commit/

[GitHub] spark issue #13802: [SPARK-16094][SQL] Support HashAggregateExec for non-par...

2016-06-21 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13802 **[Test build #60919 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/60919/consoleFull)** for PR 13802 at commit [`517d7ea`](https://github.com/apache/spark/commit/5